Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Thu, 2014-02-27 at 10:53 +0900, Mike Hommey wrote: I have no problems with your original patch. I think I said that right there and then. If Paul agrees, I will commit it. Paul? I'll reiterate my position that (a) I've seen nothing showing that it's inherently impossible for make to figure out for itself what the right thing to do is, instead requiring the user to specify it (if there is such a case it should be simple to show an example and explain why it's impossible for make to decide), and (b) given that (a) is true, I'm not excited about adding a makefile flag to force the user to make that decision... what happens, for example, if someone does decide to fix this the right way in the future so the user option is not needed... now we have this make setting which doesn't make any sense but which we have to maintain for backward-compatibility. HOWEVER. I don't know nearly enough about all the variations and different possibilities on Windows to make sound judgements on the specific situation, so if Eli is OK with it then I'm OK with it. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
From: Paul Smith psm...@gnu.org Cc: Eli Zaretskii e...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net Date: Sun, 02 Mar 2014 10:52:33 -0500 I'll reiterate my position that (a) I've seen nothing showing that it's inherently impossible for make to figure out for itself what the right thing to do is, instead requiring the user to specify it (if there is such a case it should be simple to show an example and explain why it's impossible for make to decide), and (b) given that (a) is true, I'm not excited about adding a makefile flag to force the user to make that decision... what happens, for example, if someone does decide to fix this the right way in the future so the user option is not needed... now we have this make setting which doesn't make any sense but which we have to maintain for backward-compatibility. I don't think (a) is possible, unless we decide that the option is only for when MSYS Bash is being used as the shell, and we teach Make to detect that flavor of Bash. If we do that, other users who just might need this mode of operation will not have any fire escapes. And if we don't want this only for MSYS, then I don't even know how to define the conditions under which this option will be needed. HOWEVER. I don't know nearly enough about all the variations and different possibilities on Windows to make sound judgements on the specific situation, so if Eli is OK with it then I'm OK with it. OK. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Thu, 6 Feb 2014 16:01:06 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net I gave multiple examples already. It doesn't require backslashes to be a problem. Here's another one: $ cat EOF foo.mk default: c:/path/to/python.exe -c 'import sys; sys.argv[1:]' a b c EOF $ make -f foo.mk c:/path/to/python.exe -c 'import sys; sys.argv[1:]' a b c ['a b', 'c'] And that (obviously) doesn't need make to fail: (in a cmd.exe shell): c:/path/to/sh.exe -c c:/path/to/python.exe -c 'import sys; sys.argv[1:]' \a\ b c ['a b', 'c'] (yes, this is how make quotes it, and the quoting is correct) If you get the same result from the command line, then how is this a problem with what Make does? What am I missing? $ mkdir foo $ echo foo foo/bar $ cat EOF foo.mk default: grep foo foo\\bar grep foo foo\\bar $ make -f foo.mk grep foo foo\\bar foo grep foo foo\\bar /usr/bin/sh: foobar: No such file or directory If you use forward slashes, do you see any problems in this case? Obviously, no. But again, this is not a path problem: $ cat EOF foo.mk default: echo foo foo\\bar echo foo foo\\bar EOF $ make -f foo.mk echo foo foo\\bar foo foo\bar echo foo foo\\bar foo foobar $ msysmake -f foo.mk echo foo foo\\bar foo foo\bar echo foo foo\\bar foo foo\bar That's what I said earlier: if you want Posix compatibility with backslash characters when your shell is MSYS Bash, you need to use the MSYS Make. The native build of Make doesn't treat backslashes as merely escape characters, it also treats them as directory separators. Obviously, that heuristic fails in this case, but the native Make is not supposed to deal with such cases. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Thu, Feb 06, 2014 at 10:45:39AM +0200, Eli Zaretskii wrote: Date: Thu, 6 Feb 2014 16:01:06 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net I gave multiple examples already. It doesn't require backslashes to be a problem. Here's another one: $ cat EOF foo.mk default: c:/path/to/python.exe -c 'import sys; sys.argv[1:]' a b c EOF $ make -f foo.mk c:/path/to/python.exe -c 'import sys; sys.argv[1:]' a b c ['a b', 'c'] And that (obviously) doesn't need make to fail: (in a cmd.exe shell): c:/path/to/sh.exe -c c:/path/to/python.exe -c 'import sys; sys.argv[1:]' \a\ b c ['a b', 'c'] (yes, this is how make quotes it, and the quoting is correct) If you get the same result from the command line, then how is this a problem with what Make does? What am I missing? The point is, batch shell mode only flag is advertized in config.h as being the workaround for the broken sh -c. I'd like that workaround to be available at runtime instead of build time. $ mkdir foo $ echo foo foo/bar $ cat EOF foo.mk default: grep foo foo\\bar grep foo foo\\bar $ make -f foo.mk grep foo foo\\bar foo grep foo foo\\bar /usr/bin/sh: foobar: No such file or directory If you use forward slashes, do you see any problems in this case? Obviously, no. But again, this is not a path problem: $ cat EOF foo.mk default: echo foo foo\\bar echo foo foo\\bar EOF $ make -f foo.mk echo foo foo\\bar foo foo\bar echo foo foo\\bar foo foobar $ msysmake -f foo.mk echo foo foo\\bar foo foo\bar echo foo foo\\bar foo foo\bar That's what I said earlier: if you want Posix compatibility with backslash characters when your shell is MSYS Bash, you need to use the MSYS Make. The native build of Make doesn't treat backslashes as merely escape characters, it also treats them as directory separators. Obviously, that heuristic fails in this case, but the native Make is not supposed to deal with such cases. The native build of make doesn't end up treating backslashes equally depending on whether the command line happens to be nice enough for it to call CreateProcess directly or not. While it makes /some/ technical sense that it is this way, it makes *no* sense from a user perspective. Add to that the fact that both direct CreateProcess and batch shell mode have the same behaviour while sh -c doesn't. And now that i look at it, it *is* another sh -c bug. sigh. I'd be all for detecting if the shell used is win32 native or not, but at least, a user-runtime-togglable workaround for that would be better. So we're back to square one. How about that first patch of mine? Mike ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Thu, 6 Feb 2014 18:54:33 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net The point is, batch shell mode only flag is advertized in config.h as being the workaround for the broken sh -c. I'd like that workaround to be available at runtime instead of build time. I don't have a problem with that, never did. And now that i look at it, it *is* another sh -c bug. sigh. MSYS Bash _is_ weird, it tries to solve a hard problem, and sometimes fails. That's why using it with a native Make is not such a great idea. I'd be all for detecting if the shell used is win32 native or not Make already does that, but only by the executable's file name. I don't see any good reasons to try detecting MSYS Bash as opposed to native Windows ports of shells, as there aren't good native ports. at least, a user-runtime-togglable workaround for that would be better. So we're back to square one. How about that first patch of mine? I have no problems with your original patch. I think I said that right there and then. If Paul agrees, I will commit it. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Wed, 5 Feb 2014 16:30:25 +0900 From: Mike Hommey m...@glandium.org Cc: Eli Zaretskii e...@gnu.org, bug-make@gnu.org I agree about using backslashes as directory separators, that obviously cannot work in /bin/sh, even on Windows. Actually, with msys sh, it works. But Make doesn't know about that. When it works with a Unixy shell, it expects Posix file-name semantics. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Wed, 5 Feb 2014 16:33:47 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bo...@kolpackov.net, bug-make@gnu.org On Tue, Feb 04, 2014 at 06:54:36PM +0200, Eli Zaretskii wrote: Why are you using backslashes in file names when your shell is a Unixy shell? That makes little sense to me, and I don't see why Make on Windows should support such use. Unixy shells are supposed to get Unixy file names with forward slashes, including in redirection. I didn't write that makefile, and that makefile is just a pile of awful things, not to say worse, and it does rely on the paths using backslashes for many different reasons. I tried switching to forward slashes, but many pieces broke as a result. The fact that I can get that makefile to work with batch shell mode is kind of a deliverance: i don't have to touch that pile of garbage. I see, thanks for explaining this. However, now I wonder why would it make sense to have this solved in upstream Make. Your use case sounds rather singular, so it could be solved by modifying your own copy of Make, right? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Wed, Feb 05, 2014 at 05:42:17PM +0200, Eli Zaretskii wrote: Date: Wed, 5 Feb 2014 16:30:25 +0900 From: Mike Hommey m...@glandium.org Cc: Eli Zaretskii e...@gnu.org, bug-make@gnu.org I agree about using backslashes as directory separators, that obviously cannot work in /bin/sh, even on Windows. Actually, with msys sh, it works. But Make doesn't know about that. When it works with a Unixy shell, it expects Posix file-name semantics. But the thing is there is still inconsistency in how things end up being invoked whether - make calls them directly - make uses sh -c - make uses sh script.sh That's barely a posix file-name semantics issue. In fact, for all purposes, make doesn't even need to know they are file names. The most bothering part is the inconsistency, not the fact that it considers them as file names or not. On Wed, Feb 05, 2014 at 05:44:18PM +0200, Eli Zaretskii wrote: Date: Wed, 5 Feb 2014 16:33:47 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bo...@kolpackov.net, bug-make@gnu.org On Tue, Feb 04, 2014 at 06:54:36PM +0200, Eli Zaretskii wrote: Why are you using backslashes in file names when your shell is a Unixy shell? That makes little sense to me, and I don't see why Make on Windows should support such use. Unixy shells are supposed to get Unixy file names with forward slashes, including in redirection. I didn't write that makefile, and that makefile is just a pile of awful things, not to say worse, and it does rely on the paths using backslashes for many different reasons. I tried switching to forward slashes, but many pieces broke as a result. The fact that I can get that makefile to work with batch shell mode is kind of a deliverance: i don't have to touch that pile of garbage. I see, thanks for explaining this. However, now I wonder why would it make sense to have this solved in upstream Make. Your use case sounds rather singular, so it could be solved by modifying your own copy of Make, right? I can see my unusual case as being a problem for other people too. It's always better if you can rely on unmodified tools to get things working. That's why I'd be totally fine with the first patch I attached, which adds the fake target to enable the workaround at runtime instead of buildtime. (And I wrote that patch because I don't want the horror show to propagate unwillingly and have other makefiles silently depend on paths with backslashes) Mike ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Thu, 6 Feb 2014 05:58:08 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net But the thing is there is still inconsistency in how things end up being invoked whether - make calls them directly - make uses sh -c - make uses sh script.sh When the shell is a Unixy one, only the first 2 methods are used, and they are used exactly like on Posix platforms. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Wed, Feb 05, 2014 at 11:10:11PM +0200, Eli Zaretskii wrote: Date: Thu, 6 Feb 2014 05:58:08 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net But the thing is there is still inconsistency in how things end up being invoked whether - make calls them directly - make uses sh -c - make uses sh script.sh When the shell is a Unixy one, only the first 2 methods are used, and they are used exactly like on Posix platforms. Except when make is built with BATCH_MODE_SHELL_ONLY. Which is needed when sh -c fucks up with double quotes. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Thu, Feb 06, 2014 at 06:23:46AM +0900, Mike Hommey wrote: On Wed, Feb 05, 2014 at 11:10:11PM +0200, Eli Zaretskii wrote: Date: Thu, 6 Feb 2014 05:58:08 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net But the thing is there is still inconsistency in how things end up being invoked whether - make calls them directly - make uses sh -c - make uses sh script.sh When the shell is a Unixy one, only the first 2 methods are used, and they are used exactly like on Posix platforms. Except when make is built with BATCH_MODE_SHELL_ONLY. Which is needed when sh -c fucks up with double quotes. And, more importantly, it's the first two that don't work the same way: $ mkdir foo $ echo foo foo/bar $ cat EOF foo.mk default: grep foo foo\\bar grep foo foo\\bar $ make -f foo.mk grep foo foo\\bar foo grep foo foo\\bar /usr/bin/sh: foobar: No such file or directory ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Thu, 6 Feb 2014 06:23:46 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net On Wed, Feb 05, 2014 at 11:10:11PM +0200, Eli Zaretskii wrote: Date: Thu, 6 Feb 2014 05:58:08 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net But the thing is there is still inconsistency in how things end up being invoked whether - make calls them directly - make uses sh -c - make uses sh script.sh When the shell is a Unixy one, only the first 2 methods are used, and they are used exactly like on Posix platforms. Except when make is built with BATCH_MODE_SHELL_ONLY. Indeed, that's what that compile time option is for. But if that option messes up Make (does it?), we probably should delete it. Which is needed when sh -c fucks up with double quotes. I asked for test cases where this happens. Does this happen only when backslashes are used in file names? If so, I don't think that matters, since backslashes aren't supposed to be used in file names when a Unixy shell is used. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Thu, 6 Feb 2014 06:31:15 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net Except when make is built with BATCH_MODE_SHELL_ONLY. Which is needed when sh -c fucks up with double quotes. And, more importantly, it's the first two that don't work the same way: $ mkdir foo $ echo foo foo/bar $ cat EOF foo.mk default: grep foo foo\\bar grep foo foo\\bar $ make -f foo.mk grep foo foo\\bar foo grep foo foo\\bar /usr/bin/sh: foobar: No such file or directory If you use forward slashes, do you see any problems in this case? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Thu, Feb 06, 2014 at 07:44:11AM +0200, Eli Zaretskii wrote: Date: Thu, 6 Feb 2014 06:23:46 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net On Wed, Feb 05, 2014 at 11:10:11PM +0200, Eli Zaretskii wrote: Date: Thu, 6 Feb 2014 05:58:08 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net But the thing is there is still inconsistency in how things end up being invoked whether - make calls them directly - make uses sh -c - make uses sh script.sh When the shell is a Unixy one, only the first 2 methods are used, and they are used exactly like on Posix platforms. Except when make is built with BATCH_MODE_SHELL_ONLY. Indeed, that's what that compile time option is for. But if that option messes up Make (does it?), we probably should delete it. Which is needed when sh -c fucks up with double quotes. I asked for test cases where this happens. Does this happen only when backslashes are used in file names? If so, I don't think that matters, since backslashes aren't supposed to be used in file names when a Unixy shell is used. I gave multiple examples already. It doesn't require backslashes to be a problem. Here's another one: $ cat EOF foo.mk default: c:/path/to/python.exe -c 'import sys; sys.argv[1:]' a b c EOF $ make -f foo.mk c:/path/to/python.exe -c 'import sys; sys.argv[1:]' a b c ['a b', 'c'] And that (obviously) doesn't need make to fail: (in a cmd.exe shell): c:/path/to/sh.exe -c c:/path/to/python.exe -c 'import sys; sys.argv[1:]' \a\ b c ['a b', 'c'] (yes, this is how make quotes it, and the quoting is correct) On Thu, Feb 06, 2014 at 07:45:53AM +0200, Eli Zaretskii wrote: Date: Thu, 6 Feb 2014 06:31:15 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bug-make@gnu.org, bo...@kolpackov.net Except when make is built with BATCH_MODE_SHELL_ONLY. Which is needed when sh -c fucks up with double quotes. And, more importantly, it's the first two that don't work the same way: $ mkdir foo $ echo foo foo/bar $ cat EOF foo.mk default: grep foo foo\\bar grep foo foo\\bar $ make -f foo.mk grep foo foo\\bar foo grep foo foo\\bar /usr/bin/sh: foobar: No such file or directory If you use forward slashes, do you see any problems in this case? Obviously, no. But again, this is not a path problem: $ cat EOF foo.mk default: echo foo foo\\bar echo foo foo\\bar EOF $ make -f foo.mk echo foo foo\\bar foo foo\bar echo foo foo\\bar foo foobar $ msysmake -f foo.mk echo foo foo\\bar foo foo\bar echo foo foo\\bar foo foo\bar Mike ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Tue, 4 Feb 2014 13:34:31 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, bo...@kolpackov.net, bug-make@gnu.org On Sat, Feb 01, 2014 at 11:29:19AM +0200, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Cc: Mike Hommey m...@glandium.org, mh+savan...@glandium.org, bo...@kolpackov.net, bug-make@gnu.org Date: Fri, 31 Jan 2014 10:59:13 -0500 My question, or _challenge_ if you like, is whether we can find a way to know, without any hints from the user, whether a given command line will work properly for /bin/sh -c in Windows, or if it needs to use a batch file. Mike, could you please post some details about your use case(s) that required this? I'm not sure why would quotes not work with a Unixy shell. Perhaps this is once again something specific to the MSYS conversion of file names? I'm not sure why it happens, but here's how it manifests: $ cat EOF foo.mk foo: $(LS) c:\\Program Files EOF $ make -f foo.mk LS=$(which ls) C:/mozilla-build/msys/bin/ls.exe c:\\Program Files C:/mozilla-build/msys/bin/ls.exe: c:\Program: No such file or directory C:/mozilla-build/msys/bin/ls.exe: Files: No such file or directory foo.mk:2: recipe for target 'foo' failed make: *** [foo] Error 2 So you are using the MSYS Bash and MSYS ls.exe from the native build of Make, right? Because your recipe succeeds for me both with cmd.exe and with a native build of a Unixy shell used as a shell by Make. This happens for various absolute-path versions of LS, but not when LS=ls. This is something I mentioned in bug 40241 comment 12 and 13. Another issue is with backslashes in paths. For example: $ cat EOF foo.mk foo: grep foo foo\\bar EOF (Note the is just there to trigger sh -c) This executes sh -c grep foo foo\\bar, which fails with: /usr/bin/sh: foobar: No such file or directory But in batch mode shell, this executes sh maken.sh with a content of grep foo foo\\bar and that fails with: maken.sh: line 1: foo\bar: No such file or directory (Note how in one case the backslash is eaten and not in the other case) Why are you using backslashes in file names when your shell is a Unixy shell? That makes little sense to me, and I don't see why Make on Windows should support such use. Unixy shells are supposed to get Unixy file names with forward slashes, including in redirection. When is not there and make executes grep directly, it fails with: grep: foo\bar: No such file or directory Probably because that grep is an MSYS grep, right? In general, if you are using MSYS shell and commands, you are supposed to invoke them from the MSYS build of Make. Otherwise, there can be no guarantee that commands aimed at native tools will work with MSYS programs, which are Posix programs with a twist. I'm willing to support MSYS tools invoked from a native Make, but only up to a point, and these examples go way beyond that point, sorry. If I'm confused, and the same problems happen when MSYS is not involved, please show such examples. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Tue, 2014-02-04 at 18:54 +0200, Eli Zaretskii wrote: Another issue is with backslashes in paths. For example: $ cat EOF foo.mk foo: grep foo foo\\bar EOF (Note the is just there to trigger sh -c) This executes sh -c grep foo foo\\bar, which fails with: /usr/bin/sh: foobar: No such file or directory But in batch mode shell, this executes sh maken.sh with a content of grep foo foo\\bar and that fails with: maken.sh: line 1: foo\bar: No such file or directory (Note how in one case the backslash is eaten and not in the other case) Why are you using backslashes in file names when your shell is a Unixy shell? That makes little sense to me, and I don't see why Make on Windows should support such use. Unixy shells are supposed to get Unixy file names with forward slashes, including in redirection. I agree about using backslashes as directory separators, that obviously cannot work in /bin/sh, even on Windows. But I do see a problem above; what if the literal file 'foo\bar' (a file with a backslash in the name) existed? Then this would work on UNIX but fail on Windows, because (Mike shows) too many backslashes are eaten. On UNIX, grep foo foo\\bar would do as Mike shows the batch mode shell to do, and look for the literal file 'foo\bar', but note his example above where BOTH backslashes are dropped in non-batch mode. That seems wrong to me... ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
From: Paul Smith psm...@gnu.org Cc: Mike Hommey m...@glandium.org, bug-make@gnu.org Date: Tue, 04 Feb 2014 12:28:53 -0500 But I do see a problem above; what if the literal file 'foo\bar' (a file with a backslash in the name) existed? Such a file cannot exist on Windows: the backslash character is not allowed in a file name, unlike on Unix. Then this would work on UNIX but fail on Windows, because (Mike shows) too many backslashes are eaten. On UNIX, grep foo foo\\bar would do as Mike shows the batch mode shell to do, and look for the literal file 'foo\bar', but note his example above where BOTH backslashes are dropped in non-batch mode. That seems wrong to me... There's no alternative: Make on Windows must support both forward- and back-slashes, or else our Windows users will lynch us. That support doesn't come without a price, but I see no way around that, at least not a way that avoids the issue entirely. A backslash is a legitimate directory separator on Windows, and must be handled as such. But it is also an escape (quote) character -- in Unix shells and in Makefiles (e.g., to escape-protect the colon and other special characters). I see no way of supporting the same character in several different and even contradictory roles, without some heuristics which necessarily sacrifices rare and/or unsupported scenarios and use cases, and favors frequent and supported ones. IOW, I don't think the analogy with Unix is valid here, because it sets the bar way too high, so high, in fact, that it cannot be cleared. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Tue, 4 Feb 2014, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Cc: Mike Hommey m...@glandium.org, bug-make@gnu.org Date: Tue, 04 Feb 2014 12:28:53 -0500 But I do see a problem above; what if the literal file 'foo\bar' (a file with a backslash in the name) existed? Such a file cannot exist on Windows: the backslash character is not allowed in a file name, unlike on Unix. This MSDN page lists all the reserved characters, along with the other naming conventions on MSWin. http://msdn.microsoft.com/en-us/library/aa365247.aspx - Daniel ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Tue, Feb 04, 2014 at 12:28:53PM -0500, Paul Smith wrote: On Tue, 2014-02-04 at 18:54 +0200, Eli Zaretskii wrote: Another issue is with backslashes in paths. For example: $ cat EOF foo.mk foo: grep foo foo\\bar EOF (Note the is just there to trigger sh -c) This executes sh -c grep foo foo\\bar, which fails with: /usr/bin/sh: foobar: No such file or directory But in batch mode shell, this executes sh maken.sh with a content of grep foo foo\\bar and that fails with: maken.sh: line 1: foo\bar: No such file or directory (Note how in one case the backslash is eaten and not in the other case) Why are you using backslashes in file names when your shell is a Unixy shell? That makes little sense to me, and I don't see why Make on Windows should support such use. Unixy shells are supposed to get Unixy file names with forward slashes, including in redirection. I agree about using backslashes as directory separators, that obviously cannot work in /bin/sh, even on Windows. Actually, with msys sh, it works. It just didn't work in my case because i didn't have a foo directory with a bar file in it. If i do create them, those commands work as expected (except the rule doesn't run because foo exists, so i need to rename it or make it phony) Mike ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Tue, Feb 04, 2014 at 06:54:36PM +0200, Eli Zaretskii wrote: Why are you using backslashes in file names when your shell is a Unixy shell? That makes little sense to me, and I don't see why Make on Windows should support such use. Unixy shells are supposed to get Unixy file names with forward slashes, including in redirection. I didn't write that makefile, and that makefile is just a pile of awful things, not to say worse, and it does rely on the paths using backslashes for many different reasons. I tried switching to forward slashes, but many pieces broke as a result. The fact that I can get that makefile to work with batch shell mode is kind of a deliverance: i don't have to touch that pile of garbage. Mike ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Sat, Feb 01, 2014 at 11:29:19AM +0200, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Cc: Mike Hommey m...@glandium.org, mh+savan...@glandium.org, bo...@kolpackov.net, bug-make@gnu.org Date: Fri, 31 Jan 2014 10:59:13 -0500 My question, or _challenge_ if you like, is whether we can find a way to know, without any hints from the user, whether a given command line will work properly for /bin/sh -c in Windows, or if it needs to use a batch file. Mike, could you please post some details about your use case(s) that required this? I'm not sure why would quotes not work with a Unixy shell. Perhaps this is once again something specific to the MSYS conversion of file names? I'm not sure why it happens, but here's how it manifests: $ cat EOF foo.mk foo: $(LS) c:\\Program Files EOF $ make -f foo.mk LS=$(which ls) C:/mozilla-build/msys/bin/ls.exe c:\\Program Files C:/mozilla-build/msys/bin/ls.exe: c:\Program: No such file or directory C:/mozilla-build/msys/bin/ls.exe: Files: No such file or directory foo.mk:2: recipe for target 'foo' failed make: *** [foo] Error 2 This happens for various absolute-path versions of LS, but not when LS=ls. This is something I mentioned in bug 40241 comment 12 and 13. Another issue is with backslashes in paths. For example: $ cat EOF foo.mk foo: grep foo foo\\bar EOF (Note the is just there to trigger sh -c) This executes sh -c grep foo foo\\bar, which fails with: /usr/bin/sh: foobar: No such file or directory But in batch mode shell, this executes sh maken.sh with a content of grep foo foo\\bar and that fails with: maken.sh: line 1: foo\bar: No such file or directory (Note how in one case the backslash is eaten and not in the other case) When is not there and make executes grep directly, it fails with: grep: foo\bar: No such file or directory So there is a lack of consistency in how backslashes are treated when make uses sh -c or not. Note the second patch to bug 41246 solves the first issue but not the second. The first patch, with .BATCH_MODE_SHELL added to the makefile fixes both issues. Mike ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
From: Paul Smith psm...@gnu.org Cc: Mike Hommey m...@glandium.org, mh+savan...@glandium.org, bo...@kolpackov.net, bug-make@gnu.org Date: Fri, 31 Jan 2014 10:59:13 -0500 My question, or _challenge_ if you like, is whether we can find a way to know, without any hints from the user, whether a given command line will work properly for /bin/sh -c in Windows, or if it needs to use a batch file. Mike, could you please post some details about your use case(s) that required this? I'm not sure why would quotes not work with a Unixy shell. Perhaps this is once again something specific to the MSYS conversion of file names? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Fri, 31 Jan 2014 09:14:32 + From: Mike Hommey invalid.nore...@gnu.org This is a different approach to the problem, as suggested by Paul: this triggers batch mode shell when there are double quotes in the recipe. This cannot be automatic, as we have deliberately made the double quotes not special for Windows shells. So please make this conditional on some Make option (sorry, I didn't yet have time to look at the patch). ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
Date: Fri, 31 Jan 2014 22:15:56 +0900 From: Mike Hommey m...@glandium.org Cc: psm...@gnu.org, mh+savan...@glandium.org, bo...@kolpackov.net, bug-make@gnu.org On Fri, Jan 31, 2014 at 11:37:47AM +0200, Eli Zaretskii wrote: Date: Fri, 31 Jan 2014 09:14:32 + From: Mike Hommey invalid.nore...@gnu.org This is a different approach to the problem, as suggested by Paul: this triggers batch mode shell when there are double quotes in the recipe. This cannot be automatic, as we have deliberately made the double quotes not special for Windows shells. You mean cmd.exe? The patch only uses batch mode shell when there are double quotes *and* the shell is unixy (so, not cmd.exe) That's better, but still too subtly to be the default, I think. Having some essentially random character trigger very different behavior is bad mojo for a program in general. So please make this conditional on some Make option (sorry, I didn't yet have time to look at the patch). An option as command line argument, or as a special target as the original patch did? The former, as Paul objected to the latter. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Fri, Jan 31, 2014 at 11:37:47AM +0200, Eli Zaretskii wrote: Date: Fri, 31 Jan 2014 09:14:32 + From: Mike Hommey invalid.nore...@gnu.org This is a different approach to the problem, as suggested by Paul: this triggers batch mode shell when there are double quotes in the recipe. This cannot be automatic, as we have deliberately made the double quotes not special for Windows shells. You mean cmd.exe? The patch only uses batch mode shell when there are double quotes *and* the shell is unixy (so, not cmd.exe) So please make this conditional on some Make option (sorry, I didn't yet have time to look at the patch). An option as command line argument, or as a special target as the original patch did? Mike ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Fri, 2014-01-31 at 16:55 +0200, Eli Zaretskii wrote: An option as command line argument, or as a special target as the original patch did? The former, as Paul objected to the latter. I didn't object, per se. I just prefer my tools to DTRT in all cases without me having to use any magical incantations to make it happen :-). My question, or _challenge_ if you like, is whether we can find a way to know, without any hints from the user, whether a given command line will work properly for /bin/sh -c in Windows, or if it needs to use a batch file. It seems, from my lofty vantige point of knowing virtually nothing about it, that this is a solveable problem. If we can try to write down (in text) the rules for when we would or would not be able to use /bin/sh -c, I would think it would quickly become clear if we can solve it or not. As for whether we use a command line option or makefile target, I think probably the makefile target is probably more appropriate. In general I like to use command line options to control user preferences which do not have direct impacts on the build, so if you ran the same build with different flags it would still work, just with different behaviors. But for things which will cause the makefile to not work properly if not specified, those things should be embedded into the makefile itself: I don't want it to be the case that you always have to remember to give a particular command line option or your build will break! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Sat, Jan 18, 2014 at 4:17 PM, Paul D. Smith invalid.nore...@gnu.org wrote: There are limits to the size of command lines that can be invoked. On newer systems like Linux that limit is pretty large (I think it's 16K or so on Linux) but it exists, and on other systems it's much less. It's actually quite a bit bigger than that. On RHEL6 (2.6.32 kernel), getconf ARG_MAX reports 2621440. This, minus the size of the current environment, is the number of bytes available for command lines. It's conceivable that bash imposes its own limit due to fixed buffer sizes; I haven't checked. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
