Re: Another issue with -O?
From: Paul Smith psm...@gnu.org Cc: reinp...@win.tue.nl, bug-make@gnu.org Date: Fri, 03 May 2013 16:51:47 -0400 I think enabling [-O] by default will be a no-brainer if/when we come up with a way to get it to produce the same output as without -j. IOW, run a parallel build, but output its results as if it were a non-parallel build. If you mean literally exactly the same as a non-parallel build, that is enormously difficult. Yes, literally, but with one exception: the order of producing on the screen output from targets that are remade in parallel (i.e. are independent of each other), and are on the same level of recursion, does not have to be preserved. Doing that would require that instead of dumping output in the order that jobs finish, make would have to remember the order in which they were started and dump the output in that order. I meant targets, not jobs (the latter are single command lines, right?). Dump output of a whole target as a single unit. As for the order, as long as the targets are remade in parallel, it doesn't matter. Example: all: a b a: xa @echo $@ xa xb xc: @echo $@ b: xb xc xd @echo $@ xd: $(MAKE) foo @echo $@ In the above case, I'd like to see xa a xb xc $(MAKE) foo xd b but I wouldn't mind to see this instead: xa a xc xb $(MAKE) foo xd b or this: $(MAKE) foo xd xc xb b xa a By contrast, I _would_ mind to see this, for example: xa xb a xc xd $(MAKE) b My position is if we can get output sync to a level where it is no worse than today's parallel build behavior If we want it to be no worse, then why do we need it at all, let alone have it turned on by default? I thought -O should actually improve something, so no worse is too weak to describe that, IMO. And what are our criteria for deciding whether it's no worse? The current default behavior might be confusing and hard to interpret in some cases, but at least it's familiar. -O changes that to a different behavior which is confusing (at least to me) in different situations, and is unfamiliar. How do we decide which one is worse? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
On Sat, 2013-05-04 at 09:57 +0300, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Cc: reinp...@win.tue.nl, bug-make@gnu.org Date: Fri, 03 May 2013 16:51:47 -0400 I think enabling [-O] by default will be a no-brainer if/when we come up with a way to get it to produce the same output as without -j. IOW, run a parallel build, but output its results as if it were a non-parallel build. If you mean literally exactly the same as a non-parallel build, that is enormously difficult. Yes, literally, but with one exception: the order of producing on the screen output from targets that are remade in parallel (i.e. are independent of each other), and are on the same level of recursion, does not have to be preserved. Well that's not literally exactly the same then :-). But I won't agree to the caveats here, in particular the phrase on the same level of recursion. Today with parallel builds and recursion, we don't guarantee anything about the order of execution between different sub-makes and as I mentioned, I see no justification for adding that new requirement to synchronized output. Example: all: a b a: xa @echo $@ xa xb xc: @echo $@ b: xb xc xd @echo $@ xd: $(MAKE) foo @echo $@ By contrast, I _would_ mind to see this, for example: xa xb a xc xd $(MAKE) b I agree that is less than ideal and is one of the two issues I mention below. This will be fixed. However, you may see this: xa xb a $(MAKE) foo xc xd b There's nothing that can be done about that (and that's true of today's parallel build as well). My position is if we can get output sync to a level where it is no worse than today's parallel build behavior If we want it to be no worse, then why do we need it at all, let alone have it turned on by default? I thought -O should actually improve something, so no worse is too weak to describe that, IMO. Obviously we gain synchronized output. I mean, no worse in terms of other behavior. And what are our criteria for deciding whether it's no worse? The current default behavior might be confusing and hard to interpret in some cases, but at least it's familiar. -O changes that to a different behavior which is confusing (at least to me) in different situations, and is unfamiliar. I believe we can get to the point where anyone who can read and understand parallel output can even more easily read and understand the output from -O. Conceptually all we're doing is ensuring the output comes out at the same time, uninterrupted, instead of interleaved. It should not be hard to understand that. There are two known issues right now that are causing confusing output. Hopefully once I fix those the output will make more sense than normal parallel output. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
From: Paul Smith psm...@gnu.org Cc: reinp...@win.tue.nl, bug-make@gnu.org Date: Sat, 04 May 2013 09:04:24 -0400 you may see this: xa xb a $(MAKE) foo xc xd b If a appears before xb, then that's all I ask for. If we want it to be no worse, then why do we need it at all, let alone have it turned on by default? I thought -O should actually improve something, so no worse is too weak to describe that, IMO. Obviously we gain synchronized output. It's not synchronized. It's just has a coarser granularity than what we get without -O. IOW, the chunks are larger, but still interlaced in somewhat random order. I believe we can get to the point where anyone who can read and understand parallel output can even more easily read and understand the output from -O. I'd surely like to get there ;-) ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
I think having this facility built into make is a win, especially as parallel builds become predominant. I would be even more happy about it if we can get it to the point where it can be enabled by default, and users don't even have to worry about it. I agree with Paul. This is something that can't really be done right by an external tool. While I agree that make is a better place to do this than an external tool, something in the back of my mind has been niggling about this still not really being the right answer. I suspect the root of that niggle is a desire to make the stdio FILE / descriptor implement this; there would be a way to fork a file descriptor (or a way to configure what fork() does with inherited file descriptors), such that each child forked off it gets to see something that looks and feels just like the original (e.g. isatty() and kindred functions give the same answers) but with some configuration options passed to the fdfork/ffork function that determine what happens to interleaved I/O among the children of a common parent. One option would then be to buffer all output from the child and deliver it when the child closes. (Another commonly desirable pattern would be to ensure that interleaving happens only at line boundaries, for example, to avoid the case some of us report with -j of one process's error coming in part way through a line of output from another.) When make fork()s its sub-processes, it could then (before exec()ing the sub-process) select which way to set the interleaf options on the file-handles it inherits from its parent. There would be nothing particularly special about make here: any program that invokes fork() would be able to do similar; and any program that fork()s many children is indeed apt to want the same choices you're considering for make (and maybe others). This is undoubtedly too big a thing to change in the standard library, and has been for decades, so we have to make do with doing it in make instead; but maybe it'll stir someone's brain to think of another way to achieve the same result, that's not so specific to make. Perhaps a shim round the standard library might implement ffork/fdfork, or something like them. I have no idea whether it's even possible to preserve descriptor attributes in the sense countenanced, using only the existing standard library facilities. Perhaps a new option to fcntl() could do the trick (but that'd be little use to make, today, as it'd take decades before all fcntl()s implement it). Anyway, enough rattling - I think what you've implemented sounds like a great feature and I await with glee its (suitably tried, tested and refined) release in the next version of GNU make, Eddy. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
Reading this discussion, as a bystander I can't help wondering whether the addition of -O is worthwhile. Unix tools are supposed to be small and dedicated. Using a separate utility seems to be a clean solution here, and that is fact how it was originally done: http://lists.gnu.org/archive/html/bug-make/2011-04/msg00018.html Using a separate utility is less performant and more cumbersome, but it is more modular. The semantics are clear. The utility can be documented and developed separately from GNU Make. As a GNU Make user I worry that with the -O option I won't be sure how it works. -- Reinier Post TU Eindhoven ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
From: Paul Smith psm...@gnu.org Date: Fri, 03 May 2013 08:57:57 -0400 Cc: bug-make@gnu.org I think having this facility built into make is a win, especially as parallel builds become predominant. I would be even more happy about it if we can get it to the point where it can be enabled by default, and users don't even have to worry about it. I think enabling it by default will be a no-brainer if/when we come up with a way to get it to produce the same output as without -j. IOW, run a parallel build, but output its results as if it were a non-parallel build. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
I've done the external utility solution and only because we absolutely had no other choice - it's not much fun and can be done much more effectively by make itself. Regards, Tim On 3 May 2013 14:16, Eli Zaretskii e...@gnu.org wrote: From: Paul Smith psm...@gnu.org Date: Fri, 03 May 2013 08:57:57 -0400 Cc: bug-make@gnu.org I think having this facility built into make is a win, especially as parallel builds become predominant. I would be even more happy about it if we can get it to the point where it can be enabled by default, and users don't even have to worry about it. I think enabling it by default will be a no-brainer if/when we come up with a way to get it to produce the same output as without -j. IOW, run a parallel build, but output its results as if it were a non-parallel build. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Another issue with -O?
With this simple Makefile: all: @echo foobar! true I get: D:\gnu\make-3.82.90_GIT_2013-05-01gnumake -j -f mkfsync1 foobar! true which is expected, but: D:\gnu\make-3.82.90_GIT_2013-05-01gnumake -j -f mkfsync1 -O true foobar! Is this also expected? (I see the same on GNU/Linux.) ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
On Thu, 2013-05-02 at 20:30 +0300, Eli Zaretskii wrote: With this simple Makefile: all: @echo foobar! true Yes this is a bug. I thought of this while we were having our discussion yesterday. Unfortunately in all our tests we were using @ to silence make's output of the command line, so we didn't notice that the command line make prints before it runs the command is not printed using the child's output context, it's just printed directly to stdout. This causes the mis-order you see. I have a solution partly-implemented. I think the solution will actually make things a little more clean. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
