Re: [PATCH] Bad sed expression in bootstrap.bat

[Giangiacomo Zaffini]

Updated bootstrap.bat runs well on my Windows 11 laptop. Thank You.

Il giorno sab 31 dic 2022 alle ore 22:15 Paul Smith  ha
scritto:

> On Mon, 2022-12-26 at 09:34 -0500, Paul Smith wrote:
> Anyway I pushed a fix to bootstrap.bat that hopefully avoids the issue.
> I was still not ever able to reproduce the original failure on the
> Windows systems I have access to so I can't test it directly.
>


-- 
il riposo dopo il lavoro

Re: [PATCH] Bad sed expression in bootstrap.bat

[Paul Smith]

On Mon, 2022-12-26 at 09:34 -0500, Paul Smith wrote:
> > If you do this with "echo", you don't need to worry about backslash
> > munging, I think, since cmd.exe's "echo" is very simple-minded, and
> > just blindly copies every character, including the quotes.
> 
> Oh... OK I will give this a try :).

I realized I must have already known this at some point and forgotten
it, since I used this method for other sed scripts in bootstrap.bat.

Anyway I pushed a fix to bootstrap.bat that hopefully avoids the issue.
I was still not ever able to reproduce the original failure on the
Windows systems I have access to so I can't test it directly.

Re: [PATCH] Bad sed expression in bootstrap.bat

[Paul Smith]

On Mon, 2022-12-26 at 16:30 +0200, Eli Zaretskii wrote:
> Well, if that is a problem, you could have bootstrap.bat write that
> file for you.

I thought of that but how can I write a file containing backslashes?

> If you do this with "echo", you don't need to worry about backslash
> munging, I think, since cmd.exe's "echo" is very simple-minded, and
> just blindly copies every character, including the quotes.

Oh... OK I will give this a try :).

Re: [PATCH] Bad sed expression in bootstrap.bat

[Eli Zaretskii]

> From: Paul Smith 
> Cc: gzaf...@gmail.com, bug-make@gnu.org
> Date: Mon, 26 Dec 2022 09:15:56 -0500
> 
> On Mon, 2022-12-26 at 05:29 +0200, Eli Zaretskii wrote:
> > How about reading the Sed script from a file, with the -f command-
> > line argument?
> 
> That would work.  It's annoying to have to add an entirely new file in
> source control just to work around this though.

Well, if that is a problem, you could have bootstrap.bat write that
file for you.  If you do this with "echo", you don't need to worry
about backslash munging, I think, since cmd.exe's "echo" is very
simple-minded, and just blindly copies every character, including the
quotes.

Re: [PATCH] Bad sed expression in bootstrap.bat

[Paul Smith]

On Mon, 2022-12-26 at 05:29 +0200, Eli Zaretskii wrote:
> How about reading the Sed script from a file, with the -f command-
> line argument?

That would work.  It's annoying to have to add an entirely new file in
source control just to work around this though.

> > bootstrap.bat is the counterpart to the bootstrap shell script, for
> > when you are running on Windows.  It allows someone to check out
> > the Git repository and build it natively on Windows without needing
> > all the autotools, etc.  It needs no POSIX tools other than a
> > version of sed.
> 
> So it should basically only work for you and no one else?

No; actually it's ONLY for other people: I don't use it myself.  When I
do testing on Windows I always build a dist file on my GNU/Linux system
and send it over to Windows and unpack and test it there.

That script is for other people who want to build GNU make checked out
directly from Git on their Windows system and don't want to have to
install, or have access to, a fully decked-out POSIX environment where
they can run all the auto tools like autoconf, automake, gettext, etc.
that are normally needed to build from Git.

Re: [PATCH] Bad sed expression in bootstrap.bat

[Eli Zaretskii]

> From: Paul Smith 
> Cc: gzaf...@gmail.com, bug-make@gnu.org
> Date: Sun, 25 Dec 2022 21:02:25 -0500
> 
> On Sun, 2022-12-25 at 19:20 +0200, Eli Zaretskii wrote:
> > So this command will work differently depending on which version of
> > Windows it runs, and therefore I would suggest to replace it with
> > something that doesn't need to double or triple backslashes, in order
> > for it to be more portable.
> 
> I would love to do that but I have no idea how.
> 
> The goal is to add a backslash to the end of every line in the file.  I
> don't see a way to do it, without having a backslash on the command
> line.  I'm certainly not even close to an expert in anything Windows
> however.

How about reading the Sed script from a file, with the -f command-line
argument?  In the file you can have whatever you need, and the
contents will not be subject to command-line expansion that creates
this problem.

> > Btw, what is the purpose of the bootstrap.bat file?  It is not part
> > of the Make source tarball, so who and when uses it?
> 
> bootstrap.bat is the counterpart to the bootstrap shell script, for
> when you are running on Windows.  It allows someone to check out the
> Git repository and build it natively on Windows without needing all the
> autotools, etc.  It needs no POSIX tools other than a version of sed.

So it should basically only work for you and no one else?

Re: [PATCH] Bad sed expression in bootstrap.bat

[Paul Smith]

On Sun, 2022-12-25 at 19:20 +0200, Eli Zaretskii wrote:
> So this command will work differently depending on which version of
> Windows it runs, and therefore I would suggest to replace it with
> something that doesn't need to double or triple backslashes, in order
> for it to be more portable.

I would love to do that but I have no idea how.

The goal is to add a backslash to the end of every line in the file.  I
don't see a way to do it, without having a backslash on the command
line.  I'm certainly not even close to an expert in anything Windows
however.

> Btw, what is the purpose of the bootstrap.bat file?  It is not part
> of the Make source tarball, so who and when uses it?

bootstrap.bat is the counterpart to the bootstrap shell script, for
when you are running on Windows.  It allows someone to check out the
Git repository and build it natively on Windows without needing all the
autotools, etc.  It needs no POSIX tools other than a version of sed.

Re: [PATCH] Bad sed expression in bootstrap.bat

[Eli Zaretskii]

> From: Paul Smith 
> Cc: bug-make@gnu.org
> Date: Sun, 25 Dec 2022 09:06:53 -0500
> 
> On Sun, 2022-12-25 at 02:29 +0100, Giangiacomo Zaffini wrote:
> > I cannot see where
> > sed -e "s/$/ \\/"
> > can be wrong on Windows command line or Windows PowerShell.
> 
> It's because, apparently, cmd.exe sees the "\\" and turns it into a
> single "\" before it invokes the command.  Then the syntax is:
> 
>   sed -e 's/$/ \/'
> 
> which is not legal because the backslash escapes the trailing slash,
> and now there's no trailing slash for the s/// operator.  So you get an
> error.
> 
> If I'm in cmd.exe and I use Perl to show how cmd.exe will interpret the
> command line (using Perl because I know it will be installed if someone
> is running the GNU make test suite; you could use something else if you
> like):
> 
>   C:\Users\user> perl -e "print $ARGV[0];" "s/$/ \\/"
>   s/$/ \/
> 
> note how there's only one "\" here which is wrong and gives an error

Not that it matters much for the purposes of this discussion, but it
is not cmd.exe that does this, it is the startup code of the program
which processes the command line and breaks it into argv[] elements.
The specific function which does that is part of the Microsoft runtime
library.

And one other nit: the working of that function in the Microsoft
runtime has changed between XP and newer systems.  On XP, the original
command, viz.:

  sed -e "s/$/ \\/"

yields the expected "s/$/ \\/" argv element; the backslash is not
eaten.  I see this both in Perl and in a simple C program I have which
just prints its argv[].  So this command will work differently
depending on which version of Windows it runs, and therefore I would
suggest to replace it with something that doesn't need to double or
triple backslashes, in order for it to be more portable.

Btw, what is the purpose of the bootstrap.bat file?  It is not part of
the Make source tarball, so who and when uses it?

Re: [PATCH] Bad sed expression in bootstrap.bat

[Paul Smith]

On Sun, 2022-12-25 at 02:29 +0100, Giangiacomo Zaffini wrote:
> I cannot see where
> sed -e "s/$/ \\/"
> can be wrong on Windows command line or Windows PowerShell.

It's because, apparently, cmd.exe sees the "\\" and turns it into a
single "\" before it invokes the command.  Then the syntax is:

  sed -e 's/$/ \/'

which is not legal because the backslash escapes the trailing slash,
and now there's no trailing slash for the s/// operator.  So you get an
error.

If I'm in cmd.exe and I use Perl to show how cmd.exe will interpret the
command line (using Perl because I know it will be installed if someone
is running the GNU make test suite; you could use something else if you
like):

  C:\Users\user> perl -e "print $ARGV[0];" "s/$/ \\/"
  s/$/ \/

note how there's only one "\" here which is wrong and gives an error
when passed to sed (at least, my version of sed).  On the other hand if
I run this:

  C:\Users\user> perl -e "print $ARGV[0];" "s/$/ \\\/"
  s/$/ \\/

now see how the "\" is escaped, as it should be, and now sed would be
happy with this.

> On the other hand, as reported, it is true that original sed command
> is correct on MSYS/mingw64/mingw32

I don't understand what you mean here.  MSYS/mingw64/mingw32 is not a
command interpreter, it's a subsystem description.

> gzaff@LAPTOP-6HAJ0RG0 MSYS /c/Users/gzaff/Devs/make
> $ echo static const char *const GUILE_module_defn = \" \\> src/gmk-
> default.h

This is a POSIX shell, it's irrelevant what the behavior is in a POSIX
shell since we're working with a cmd.exe batch file (bootstrap.bat).

> I don't know, maybe understanding if bootstrap.bat is running under
> MSYS2/mingw64/mingw32 bash
> or 
> Windows command line/Windows PowerShell 
> and use correct sed syntax for the case ?

Again I didn't understand this, sorry.  How can we run a .bat file
"under bash"?  Bash is a POSIX shell: it can't interpret a .bat file.

If you have a POSIX shell, you should be running "bootstrap" which is a
shell script, not "bootstrap.bat" which is a batch file.

Re: [PATCH] Bad sed expression in bootstrap.bat

[Giangiacomo Zaffini]

I successfully tested modified bootstrap.bat on both Windows command line
and Windows PowerShell (PSVersion 7.2.6)
with sed.exe (GNU sed) 4.8 that is the latest on Windows according to the
used installer scoop.

```
PS C:\Users\gzaff\Devs\make> .\bootstrap.bat
-- Downloading Gnulib modules
-- Configuring the workspace
- Creating Basic.mk
- Creating src\mkconfig.h
- Creating src\gmk-default.h

Done.  Run build_w32.bat to build GNU make.
PS C:\Users\gzaff\Devs\make> .\build_w32.bat --without-guile tcc

Creating GNU Make for Windows 9X/NT/2K/XP/Vista/7/8/10

- Building without Guile
- Building with TinyC
- Enabling maintainer mode

tcc version 0.9.27 (x86_64 Windows)

Compiling .\TccRel version
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
- Compiling src/ar.c
- Compiling src/arscan.c
- Compiling src/commands.c
- Compiling src/default.c
- Compiling src/dir.c
- Compiling src/expand.c
- Compiling src/file.c
- Compiling src/function.c
- Compiling src/getopt.c
- Compiling src/getopt1.c
- Compiling src/guile.c
- Compiling src/hash.c
- Compiling src/implicit.c
- Compiling src/job.c
- Compiling src/load.c
- Compiling src/loadapi.c
- Compiling src/main.c
- Compiling src/misc.c
- Compiling src/output.c
- Compiling src/read.c
- Compiling src/remake.c
- Compiling src/remote-stub.c
- Compiling src/rule.c
- Compiling src/shuffle.c
- Compiling src/signame.c
- Compiling src/strcache.c
- Compiling src/variable.c
- Compiling src/version.c
- Compiling src/vpath.c
- Compiling src/w32/pathstuff.c
- Compiling src/w32/w32os.c
- Compiling src/w32/compat/posixfcn.c
- Compiling src/w32/subproc/misc.c
- Compiling src/w32/subproc/sub_proc.c
- Compiling src/w32/subproc/w32err.c
In file included from src/w32/subproc/w32err.c:19:
./src/makeint.h:409: warning: WIN32_LEAN_AND_MEAN redefined
- Compiling lib/fnmatch.c
- Compiling lib/glob.c
- Compiling lib/getloadavg.c
- Compiling src\w32\compat\dirent.c

- Linking ./TccRel/gnumake.exe

.\TccRel build succeeded.
1 file(s) copied.
PS C:\Users\gzaff\Devs\make> sed --version
C:\Users\gzaff\scoop\apps\sed\current\sed.exe (GNU sed) 4.8
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
Paolo Bonzini, Jim Meyering, and Assaf Gordon.

This sed program was built without SELinux support.

GNU sed home page: .
General help using GNU software: .
E-mail bug reports to: .
Patched (v2) by: Michael M. Builov .
PS C:\Users\gzaff\Devs\make> $PsVersionTable

Name   Value
   -
PSVersion  7.2.6
PSEdition  Core
GitCommitId7.2.6
OS Microsoft Windows 10.0.22621
Platform   Win32NT
PSCompatibleVersions   {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion  2.3
SerializationVersion   1.1.0.1
WSManStackVersion  3.0
```

I cannot see where
sed -e "s/$/ \\/"
can be wrong on Windows command line or Windows PowerShell.

On the other hand, as reported, it is true that original sed command is
correct on MSYS/mingw64/mingw32

```
gzaff@LAPTOP-6HAJ0RG0 MSYS /c/Users/gzaff/Devs/make
$ echo static const char *const GUILE_module_defn = \" \\> src/gmk-default.h

gzaff@LAPTOP-6HAJ0RG0 MSYS /c/Users/gzaff/Devs/make
$ cat src/gmk-default.h
static const char *const GUILE_module_defn = " \

gzaff@LAPTOP-6HAJ0RG0 MSYS /c/Users/gzaff/Devs/make
$ sed -e "s/;.*//" -e "/^[ \t]*$/d" -e "s/\"/\"/g" -e "s/$/ \\/"
src/gmk-default.scm >> src/gmk-default.h
sed: -e expression #4, char 7: unterminated `s' command

gzaff@LAPTOP-6HAJ0RG0 MSYS /c/Users/gzaff/Devs/make
$ sed -e "s/;.*//" -e "/^[ \t]*$/d" -e "s/\"/\"/g" -e "s/$/ \\\/"
src/gmk-default.scm >> src/gmk-default.h

gzaff@LAPTOP-6HAJ0RG0 MSYS /c/Users/gzaff/Devs/make
$ sed --version
sed (GNU sed) 4.7
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
Paolo Bonzini, Jim Meyering, and Assaf Gordon.
GNU sed home page: .
General help using GNU software: .
E-mail bug reports to: .

gzaff@LAPTOP-6HAJ0RG0 MSYS /c/Users/gzaff/Devs/make
$
```

I don't know, maybe understanding if bootstrap.bat is running under
MSYS2/mingw64/mingw32 bash
or
Windows command line/Windows PowerShell
and use correct sed syntax for the case ?

Giangiacomo

Il giorno sab 24 dic 2022 alle ore 17:32 Paul Smith  ha
scritto:

> On Thu, 2022-12-22 at 18:17 +0100, Giangiacomo 

Re: [PATCH] Bad sed expression in bootstrap.bat

[Paul Smith]

On Sat, 2022-12-24 at 11:32 -0500, Paul Smith wrote:
> On Thu, 2022-12-22 at 18:17 +0100, Giangiacomo Zaffini wrote:
> > C:\Users\gzaff\scoop\apps\sed\current\sed.exe: -e expression #4,
> > char 9: unterminated `s' command
> 
> Interestingly, this works fine [for me] and the replacement version
> you suggest shows an error:
> 
>   sed: -e expression #4, char 7: unterminated `s' command
> 
> Note I am running this in cmd.exe.  Maybe you are running in
> PowerShell and the quoting is different, or something?

Nope, I get the same behavior with PowerShell.

Maybe it's something to do with the versions of sed we are using?

I don't know how to fix this problem but since your change makes the
script fail in my environment, and I can't make your change work in my
environment, I'll have to wait for someone to provide more information
on what the real issue is before I modify anything.

Re: [PATCH] Bad sed expression in bootstrap.bat

[Paul Smith]

On Thu, 2022-12-22 at 18:17 +0100, Giangiacomo Zaffini wrote:
> C:\Users\gzaff\scoop\apps\sed\current\sed.exe: -e expression #4, char
> 9: unterminated `s' command

Someone else mentioned this a week or so ago.

Interestingly, this works fine with the version of sed I have installed
(the version that comes with Git for Windows):

  sed (GNU sed) 4.9

And the replacement version you suggest shows an error:

  sed: -e expression #4, char 7: unterminated `s' command

Note I am running this in cmd.exe.  Maybe you are running in PowerShell
and the quoting is different, or something?

Very confusing.

[PATCH] Bad sed expression in bootstrap.bat

[Giangiacomo Zaffini]

Lately I read that make can be built on source with TCC Tiny C Compiler on
Windows.
So as an exercise I tried to do so.

While bootstrapping on Windows 11 machine I noticed bootstrap/bat
complaining about a sed expression
```
**
** Visual Studio 2022 Developer Command Prompt v17.4.3
** Copyright (c) 2022 Microsoft Corporation
**
[vcvarsall.bat] Environment initialized for: 'x64'

C:\Program Files\Microsoft Visual Studio\2022\Community>cd
C:\Users\gzaff\Devs\make

C:\Users\gzaff\Devs\make>bootstrap.bat
-- Downloading Gnulib modules
-- Configuring the workspace
- Creating Basic.mk
- Creating src\mkconfig.h
- Creating src\gmk-default.h
C:\Users\gzaff\scoop\apps\sed\current\sed.exe: -e expression #4, char 9:
unterminated `s' command
*** Bootstrap failed.
Resolve the issue, or use the configured source in the release tarball

C:\Users\gzaff\Devs\make>
```
It can be fixed as this patch shows
```
diff --git a/bootstrap.bat b/bootstrap.bat
index c1f6819a..71ca9721 100644
--- a/bootstrap.bat
+++ b/bootstrap.bat
@@ -70,7 +70,7 @@ if ERRORLEVEL 1 goto Failed

 echo - Creating src\gmk-default.h
 echo static const char *const GUILE_module_defn = ^" \ > src\gmk-default.h
-sed -e "s/;.*//" -e "/^[ \t]*$/d" -e "s/\"/\"/g" -e "s/$/ \\\/"
src\gmk-default.scm >> src\gmk-default.h
+sed -e "s/;.*//" -e "/^[ \t]*$/d" -e "s/\"/\"/g" -e "s/$/ \\/"
src\gmk-default.scm >> src\gmk-default.h
 if ERRORLEVEL 1 goto Failed
 echo ^";>> src\gmk-default.h


```

I could build make without problem with MSVC (later on I will test TCC) with
```
C:\Users\gzaff\Devs\make>build_w32.bat --without-guile
```

Regards,
Giangiacomo Zaffini

-- 
il riposo dopo il lavoro


win11_bootstrap_bat_changes.patch
Description: Binary data