Re: patch to support output synchronization under -j
Hello Paul. Just my 2 cents regarding Automake ... On Tuesday 03 May 2011, Paul Smith wrote: The other thing I was thinking is that this feature might want to be enabled via a command-line argument. All the complex makefiles generated by automake, etc. for example cannot take advantage of this if you have to modify every makefile to add the special target. Not completely true. A developer using Automake who wants to take advantage of this feature could just add the line: .PARALLELSYNC: to his Makefile.am file(s). Automake's preprocessing policy is to pass through the things it doesn't understand (or that are not influential to it). Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, 2011-05-03 at 09:48 +0200, Stefano Lattarini wrote: The other thing I was thinking is that this feature might want to be enabled via a command-line argument. All the complex makefiles generated by automake, etc. for example cannot take advantage of this if you have to modify every makefile to add the special target. Not completely true. A developer using Automake who wants to take advantage of this feature could just add the line: .PARALLELSYNC: to his Makefile.am file(s). Automake's preprocessing policy is to pass through the things it doesn't understand (or that are not influential to it). Sure, I realize that. But that only helps developers. I was thinking about users of open source software, building it on larger machines with -jN and wanted to get clean output when it fails. That's my situation: I have archived vanilla tarballs downloaded directly from the upstream sites, and I have a makefile that does all that's necessary (unpack them, configure, build, install, etc.) I certainly don't want to have to go modify the Makefile.am / Makefile.in files to get this feature enabled. Also apropos of nothing, I don't like this name :-). Maybe .OUTPUTSYNC or something would be better? PARALLELSYNC doesn't help understand what the feature does. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
From: Paul Smith psm...@gnu.org Cc: bug-make@gnu.org Date: Tue, 03 May 2011 01:33:38 -0400 But, I've been playing with a makefile that I have, that builds a complete suite of GCC tools from scratch. I'm building them on larger systems with -j5 or -j10 (not much compared to what some are doing I realize). Sometimes these builds fail and it can be frustrating to see what part of the build is failing. But here's the thing: these makefiles are by their very nature massively recursive. Each top-level rule is an entire build of a major component like gcc, binutils, gdb, bison, etc. As far as I understand it, if I were to enable this feature then all of the output from the entire build of gcc (for example) would be collected into a file and not printed until the entire sub-make had completed. That was exactly the scenario I had in mind when I wrote my message. Recursive Makefiles are the rule nowadays, at least with GNU software, and the top-level Makefile does little more than launch a make all job in each subdirectory. GCC or GDB might be extreme examples, but I do build them from time to time, and there are less extreme examples which nevertheless take considerable time to run each sub-Make. I think we must find some solution to put this under user control, or else this feature will be useful only for small projects. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, May 3, 2011 at 1:33 AM, Paul Smith psm...@gnu.org wrote: I wonder if we can figure out a way to make this work better, as Eli asked. Can we work out a way to handle normal rules (rules with + or $(MAKE)) and sub-make rules differently, so that output from normal rules wasn't collected behind a sub-make? I'm not sure I see exactly how this could work. Thinking, thinking, ... The other thing I was thinking is that this feature might want to be enabled via a command-line argument. All the complex makefiles generated by automake, etc. for example cannot take advantage of this if you have to modify every makefile to add the special target. Hmm, it feels like you've reversed position since last year? When I submitted the patch for .ONESHELL it included a new --one-shell flag and you rejected the flag part saying you didn't want make to end up like GNU tar in the sense of having an overwhelming number of options. Personally I don't see what's so bad about exposing useful features at the command line, and I can't say I ever wished that GNU tar would do less of it, but I figured that was your aesthetic and I'd go with it. That was when you came up with the --eval option, which I found brilliant BTW. So I guess the first-order answer to your point would be there is a way to enable this at the command line and you invented it: --eval=.PARALLELSYNC:. However, I personally like command-line options and if you want a first-class flag you'll get no argument here. [PARALLELSYNC vs OUTPUTSYNC] I'm pretty agnostic on the name. I actually considered .OUTPUTSYNC too but felt the name should expose the fact that it's a no-op unless -j is in use. But I'm happy with it if that's the consensus. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On 3 May 2011 17:39, Eli Zaretskii e...@gnu.org wrote: That was exactly the scenario I had in mind when I wrote my message. Recursive Makefiles are the rule nowadays, at least with GNU software, and the top-level Makefile does little more than launch a make all job in each subdirectory. GCC or GDB might be extreme examples, but I do build them from time to time, and there are less extreme examples which nevertheless take considerable time to run each sub-Make. I think we must find some solution to put this under user control, or else this feature will be useful only for small projects. I wrote a similar descrambler (Talon) and use it on absolutely unbelievably enormous builds but they do only the most minimal amount of recursion. This is actually a problem because it takes so long to read in the makefiles that one wastes 10s of minutes on a single CPU before the cluster can start working. On the other hand the build part is pretty efficient :-). Anyhow I am just saying that this feature is useful huge builds and more so because the huger they are the less one tends to sit there watching the output. Even in smaller ones the boon is that you can be highly parallel, thus not taking too long to get the entire makefile. So you can't watch it with 0 latency while it is working but you do get the complete set of information sooner. We needed more though - we needed the output to be wrapped up and classified and so on so I made up a way of setting variables that controlled this output. In David's implementation one might have the luxury of using target specific variables or something like that. e.g. Talon has a FORCESUCCESS option to tell the shell to output the correct exit code of a command but tell make that it really succeeded. Another option allowed us to get detailed debugging output as the job communicated with the shell so that we could see more precisely when and how some crash happened but without turning on this feature for all recipes in the whole 20 hour build. We use macros but it is like this underneath: mytarget: |TALON_DEBUG=1;FORCESUCCESS=1;TALON_BUILDID=1342;TALON_DESCRAMBLE=1;TALON_TIMEOUT=1;|armcc -o ekern.exe .o .o Notice the DESCRAMBLE which allows me to turn off synchronisation. Now with Talon we are using a systemwide semaphore whose name is based on the TALON_BUILDID. So if we were running a submake we could turn off synchronisation for that recipe and as long as the submake used the same build id it would still be deconflicted. In this case target specific variables might be a neater way to do something similar: mytarget: PARALLELSYNC= mytarget: $(MAKE) -f submake.mk Then submake.mk carries on as normal. The problem is that David is not using semaphores so I haven't thought how to spin it for his case. Regards, Tim -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, 2011-05-03 at 13:00 -0400, David Boyce wrote: The other thing I was thinking is that this feature might want to be enabled via a command-line argument. All the complex makefiles generated by automake, etc. for example cannot take advantage of this if you have to modify every makefile to add the special target. Hmm, it feels like you've reversed position since last year? When I submitted the patch for .ONESHELL it included a new --one-shell flag and you rejected the flag part saying you didn't want make to end up like GNU tar in the sense of having an overwhelming number of options. Heh. Consistency is for suckers :-). However, I still feel that .ONESHELL is not something you can always just set on the command line and have things work just as before. Although _sometimes_ that's true, other times recipes are written in a particular way that can cause problems if .ONESHELL was set. And certainly if you write your makefile assuming .ONESHELL then anyone who runs make _without_ setting it will not be happy... the build will absolutely fail. For these reasons I'm less happy about having the user add it via the command line. To me .ONESHELL is a feature that the makefile AUTHOR chooses for her makefile, not (generally) something that the makefile USER would select (although certainly for some well-formed makefiles you can turn it on and off with no difference in behavior). For this SYNC feature, it will not make any difference whatsoever whether the person invoking the command uses it or not. In fact, in a way it's MORE appropriate for the command line, to allow the user to choose whether or not to use it (just like -j itself); it seems like forcing this to always be on by setting a pseudo target is somewhat antisocial; now if people want to avoid this behavior they have to edit the makefiles. In that way, SYNC is a feature that the makefile USER selects, or not, and not something the makefile AUTHOR would choose. Does that make sense? That was when you came up with the --eval option, which I found brilliant BTW. So I guess the first-order answer to your point would be there is a way to enable this at the command line and you invented it: --eval=.PARALLELSYNC:. However, I personally like command-line options and if you want a first-class flag you'll get no argument here. Except, I don't remember if --eval is passed down to sub-makes? The mind goes first... -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, May 3, 2011 at 1:20 PM, Paul Smith psm...@gnu.org wrote: In that way, SYNC is a feature that the makefile USER selects, or not, and not something the makefile AUTHOR would choose. Does that make sense? It makes perfect sense when you put it that way and I agree wrt to both .ONESHELL and .PARALLELSYNC (or whatever it's called now). Except, I don't remember if --eval is passed down to sub-makes? It is. The mind goes first... Remind me, what were we talking about? David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, 2011-04-15 at 16:57 +0300, Eli Zaretskii wrote: Date: Fri, 15 Apr 2011 12:43:53 +0100 From: Tim Murphy tnmur...@gmail.com I think it's an inevitable consequence that if you have a long-running task then the output from it won't appear until it has completely finished and you won't be able to watch the progress - in this scheme there is no way to show some of the output whilst also ensuring that it is not interspersed with output from other recipes. Well, if that's so, then this feature will be less useful than it could be, at least for me. It reminds me of my DOS days, when running a compilation command from Emacs would show me the output only when the compiler was done. I'm not happy to return to those compiling blind days. Can something be done to alleviate this? When I first read this it seemed to me that (a) there's no good way to solve this, and (b) it's not so important anyway in most builds as compiles are relatively quick. But, I've been playing with a makefile that I have, that builds a complete suite of GCC tools from scratch. I'm building them on larger systems with -j5 or -j10 (not much compared to what some are doing I realize). Sometimes these builds fail and it can be frustrating to see what part of the build is failing. But here's the thing: these makefiles are by their very nature massively recursive. Each top-level rule is an entire build of a major component like gcc, binutils, gdb, bison, etc. As far as I understand it, if I were to enable this feature then all of the output from the entire build of gcc (for example) would be collected into a file and not printed until the entire sub-make had completed. That's not very optimal. Unless I missed something? I wonder if we can figure out a way to make this work better, as Eli asked. Can we work out a way to handle normal rules (rules with + or $(MAKE)) and sub-make rules differently, so that output from normal rules wasn't collected behind a sub-make? I'm not sure I see exactly how this could work. The other thing I was thinking is that this feature might want to be enabled via a command-line argument. All the complex makefiles generated by automake, etc. for example cannot take advantage of this if you have to modify every makefile to add the special target. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
I've filed this as an enhancement request (https://savannah.gnu.org/bugs/index.php?33138) for better tracking. Unfortunately there's a typo in the headline which I can't fix. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Here's an updated version which is much closer to done at least from my POV. I believe most of the requested improvements are there, the semaphore code is factored, there's even a start on documentation. Despite what was said about ifdefs before, the entire patch is currently enclosed in #ifdef POSIX /* PARALLEL-SYNC */ pending any ports. David make-sync3.patch Description: Binary data ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, Apr 19, 2011 at 10:13 AM, Edward Welbourne e...@opera.com wrote: How about if we introduce a variable that can be set for specific targets (using the target-specific variable mechanism). Targets that set this variable to some predefined value will not have their output redirected, so their output will go to the screen. IIUC, the proposal is to make the feature be enabled by a .NAME target (I forget the name); .PARALLELSYNC, as currently coded. perhaps the simplest thing would be to have the feature applied to any target declared to depend on the given .NAME, supplemented with command-line options to turn it on/off everywhere, regardless of such dependencies. Well ... that's no longer the feature I signed up to contribute, and I don't have or see a major use case for it. I think the issue most people have with parallel builds involves debugging scheduled/automated builds with output to a log file. I can't think of a situation where the above feature would be important, but if someone wants to add it as a further enhancement I'm all for it, assuming my enhancement makes it into the codebase in the first place. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, Apr 19, 2011 at 11:06 AM, David Boyce david.s.bo...@gmail.com wrote: Here's an updated version which is much closer to done Ignore this version, I found a bug :-( David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Right, of course :-) That is the obvious reason for using files, thanks! I was biased against writing things to disc which is probably silly, because it's probably not slow when compared to the size of the job that's running. Stdout and stderr will now all be stdout, BTW (no matter what method one uses) but that's not generally such a problem. Regards, Tim On 14 April 2011 21:18, Paul Smith psm...@gnu.org wrote: On Thu, 2011-04-14 at 20:59 +0100, Tim Murphy wrote: To prevent any kind of deadlock you sort of want to empty everything the stderr and stdout pipe buffers may contain. It is conceivable that the stdout pipe might contain two lines of text and stderr pipe might contain 1 by the time your select statement (or waitformultipleobjects) has woken up. This would all be true, if anyone were using pipes and select statements... but we're not :-) The implementation David provided has all the output going to temporary files, which are then read and printed to stdout or stderr (and deleted) when the job gets the output sync lock. To choose between merged and not merged you just choose between creating distinct files, vs. creating one file and setting both stdout/stderr to write to it... same as it would write to the tty device. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Date: Thu, 14 Apr 2011 16:30:42 -0400 From: David Boyce david.s.bo...@gmail.com Cc: Eli Zaretskii e...@gnu.org, bug-make@gnu.org I don't know why this hasn't occurred to me or the authors of similar programs before, but it appears to be possible to get a lock on any writable file descriptor - for instance stdout or stderr, or one of the jobserver-fds. I just changed syncsh to synchronize around a lock on stdout and it seems to work. This looks much more elegant than specifying a writable file. Or is my understanding of file/pipe/locking semantics flawed? And how would Windows/DOS/VMS/OS2 would do with this? If you explain what you mean by file/pipe/locking (e.g., what APIs do you intend to use?), I will look into the possibilities of emulating that on DOS/Windows. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
I replied to this without using reply all and I apologise. :-) The order of the output from separate recipes may be anything - depends on how make schedules them but at least the output from any particular recipe will be grouped together and not mixed with the output from others, hence allowing you to associate errors with th recipes that caused them. I think it's an inevitable consequence that if you have a long-running task then the output from it won't appear until it has completely finished and you won't be able to watch the progress - in this scheme there is no way to show some of the output whilst also ensuring that it is not interspersed with output from other recipes. An alternative might be to mark all lines of text with a unique job number so that they can be reassembled later into a reasonable order. I must say this doesn't suit me at all but it might make someone happier. These features probably need to be switchable anyhow. I come at this from the perspective that my logs are too large for humans to read through and they are being indexed and summarised by processes rather than people - hence the need for the format to be highly parsable. Humans only get called in when there are errors. My longest running tasks are much shorter than the build itself so not getting progress info on them is not a killer. Regards, Tim On 15 April 2011 11:45, Eli Zaretskii e...@gnu.org wrote: Date: Thu, 14 Apr 2011 15:15:15 -0400 From: David Boyce david.s.bo...@gmail.com Cc: psm...@gnu.org, bug-make@gnu.org On Thu, Apr 14, 2011 at 3:00 PM, Eli Zaretskii e...@gnu.org wrote: Yes, but a few words about how is this semaphore supposed to get job done, and in fact what kind of synchronization will this bring to Make, would be appreciated. I don't think you described the feature too much in your original post. No, but what I did say was I won't go into how it works because that's already done at the URLs above. Do you have specific questions not addressed there? They're both pretty short, and I really can't tell what more needs to be said without some context. I lack the higher-level picture. Can you describe in a few words how will this work in a running Make? I don't mean the details of how files are locked and unlocked (I understand that part), I mean the larger picture, like how it will look to the user, how several programs running in parallel will write their output and in what order it will appear eventually on the screen. Thanks, and sorry for not being able to grasp all that from those URLS and the code. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, 2011-04-15 at 07:18 +0100, Tim Murphy wrote: Right, of course :-) That is the obvious reason for using files, thanks! I was biased against writing things to disc which is probably silly, because it's probably not slow when compared to the size of the job that's running. I think you could do the same thing with pipes although you need to spawn a separate process to track each job, most likely. The main reason I prefer solutions using files is that it's the simplest and most portable method. Stdout and stderr will now all be stdout, BTW (no matter what method one uses) but that's not generally such a problem. I don't quite follow that? If you're willing to separate stdout and stderr so that, for example, all stdout output comes first followed by all stderr, then you can still use both. You can just create two files, hook stdout to one and stderr to the other, and after everything is done shuttle the contents of the stdout log file to stdout and the contents of the stderr log file to stderr. The functionality would be the same (as far as what goes to what file descriptor) but they wouldn't be interspersed. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, Apr 15, 2011 at 6:45 AM, Eli Zaretskii e...@gnu.org wrote: I lack the higher-level picture. Can you describe in a few words how will this work in a running Make? I don't mean the details of how files are locked and unlocked (I understand that part), I mean the larger picture, like how it will look to the user, how several programs running in parallel will write their output and in what order it will appear eventually on the screen. I'll try to illustrate with a different example. For a test case I've taken an open-source package called jikes which compiles ~40 C++ files. I had to modify the automake-generated makefile to supply illustrative verbosity; here's the compilation pattern rule: %.o: %.cpp @echo BEGINNING $@ ... \ g++ -DHAVE_CONFIG_H -I. -g -O2 -c -o $@ $ \ echo FINISHED $@ ...; and I've attached output logs without.txt and with.txt, both using -j for unlimited jobs. You'll notice that in without.txt the output of recipes is separated; all the BEGINNING lines show up as soon as the job is forked, and all the ENDING lines show up as it finishes. In the with case, the BEGINNING lines are effectively held and released in sync with their matching ENDING lines. The main problem being solved here is that error and warning messages often get separated from their recipes in build logs. It's hard to debug foo: permission denied when you can't figure out which recipe generated that message. Does that help? David BEGINNING case.o ... FINISHED case.o BEGINNING jikes.o ... FINISHED jikes.o BEGINNING tab.o ... FINISHED tab.o BEGINNING lpginput.o ... FINISHED lpginput.o BEGINNING code.o ... FINISHED code.o BEGINNING dump.o ... FINISHED dump.o BEGINNING op.o ... FINISHED op.o BEGINNING unzip.o ... FINISHED unzip.o BEGINNING unparse.o ... FINISHED unparse.o BEGINNING long.o ... FINISHED long.o BEGINNING segment.o ... FINISHED segment.o BEGINNING jikesapi.o ... FINISHED jikesapi.o BEGINNING init.o ... FINISHED init.o BEGINNING modifier.o ... FINISHED modifier.o BEGINNING zip.o ... FINISHED zip.o BEGINNING set.o ... FINISHED set.o BEGINNING parser.o ... FINISHED parser.o BEGINNING platform.o ... FINISHED platform.o BEGINNING scanner.o ... FINISHED scanner.o BEGINNING option.o ... FINISHED option.o BEGINNING depend.o ... FINISHED depend.o BEGINNING incrmnt.o ... FINISHED incrmnt.o BEGINNING stream.o ... FINISHED stream.o BEGINNING system.o ... FINISHED system.o BEGINNING diagnose.o ... FINISHED diagnose.o BEGINNING error.o ... FINISHED error.o BEGINNING lookup.o ... FINISHED lookup.o BEGINNING definite.o ... FINISHED definite.o BEGINNING double.o ... FINISHED double.o BEGINNING ast.o ... FINISHED ast.o BEGINNING javaact.o ... FINISHED javaact.o BEGINNING control.o ... FINISHED control.o BEGINNING body.o ... FINISHED body.o BEGINNING symbol.o ... FINISHED symbol.o BEGINNING decl.o ... FINISHED decl.o BEGINNING class.o ... FINISHED class.o BEGINNING expr.o ... FINISHED expr.o BEGINNING bytecode.o ... FINISHED bytecode.o rm -f jikes g++ -g -O2 -o jikes ast.o body.o bytecode.o case.o class.o code.o control.o decl.o definite.o depend.o diagnose.o double.o dump.o error.o expr.o incrmnt.o init.o javaact.o jikes.o jikesapi.o long.o lookup.o lpginput.o modifier.o op.o option.o parser.o platform.o scanner.o segment.o set.o stream.o symbol.o system.o tab.o unparse.o unzip.o zip.o BEGINNING ast.o ... BEGINNING body.o ... BEGINNING bytecode.o ... BEGINNING case.o ... BEGINNING class.o ... BEGINNING code.o ... BEGINNING decl.o ... BEGINNING depend.o ... BEGINNING diagnose.o ... BEGINNING double.o ... BEGINNING dump.o ... BEGINNING error.o ... BEGINNING expr.o ... BEGINNING incrmnt.o ... BEGINNING init.o ... BEGINNING javaact.o ... BEGINNING jikes.o ... BEGINNING long.o ... BEGINNING jikesapi.o ... BEGINNING lookup.o ... BEGINNING lpginput.o ... BEGINNING op.o ... BEGINNING option.o ... BEGINNING modifier.o ... BEGINNING parser.o ... BEGINNING platform.o ... BEGINNING segment.o ... BEGINNING set.o ... BEGINNING scanner.o ... BEGINNING stream.o ... BEGINNING symbol.o ... BEGINNING system.o ... BEGINNING tab.o ... BEGINNING unparse.o ... BEGINNING unzip.o ... BEGINNING zip.o ... FINISHED tab.o FINISHED case.o FINISHED dump.o FINISHED jikes.o FINISHED op.o FINISHED code.o FINISHED lpginput.o FINISHED unparse.o FINISHED long.o FINISHED unzip.o FINISHED jikesapi.o FINISHED segment.o FINISHED zip.o FINISHED set.o FINISHED parser.o FINISHED modifier.o FINISHED init.o FINISHED platform.o FINISHED scanner.o FINISHED depend.o FINISHED incrmnt.o FINISHED error.o FINISHED option.o FINISHED stream.o FINISHED system.o FINISHED double.o FINISHED diagnose.o FINISHED definite.o FINISHED lookup.o FINISHED javaact.o FINISHED ast.o FINISHED body.o FINISHED control.o FINISHED class.o FINISHED symbol.o FINISHED decl.o FINISHED expr.o FINISHED bytecode.o rm -f jikes g++ -g -O2 -o jikes ast.o body.o bytecode.o case.o class.o code.o control.o decl.o definite.o depend.o diagnose.o double.o dump.o error.o
Re: patch to support output synchronization under -j
Date: Fri, 15 Apr 2011 12:43:53 +0100 From: Tim Murphy tnmur...@gmail.com Cc: David Boyce david.s.bo...@gmail.com, bug-make@gnu.org I think it's an inevitable consequence that if you have a long-running task then the output from it won't appear until it has completely finished and you won't be able to watch the progress - in this scheme there is no way to show some of the output whilst also ensuring that it is not interspersed with output from other recipes. Well, if that's so, then this feature will be less useful than it could be, at least for me. It reminds me of my DOS days, when running a compilation command from Emacs would show me the output only when the compiler was done. I'm not happy to return to those compiling blind days. Can something be done to alleviate this? Btw, there will be other side effects, at least on non-Posix platforms, due to the fact that stuff that was supposed to go to the screen is redirected to a file instead. Some programs sense that and behave differently, e.g. with binary non-printable characters or with special character sequences. By redirecting the output to files, then displaying it on the screen, the output may look strangely, or have some more grave effect, like terminating output prematurely. We should consider these potential problems, and in any case the simple loop that reads from temporary files and writes to stdout/stderr may need to become much more complicated, at least on DOS/Windows. (I believe similar, though less serious, problems could happen on Unix as well.) ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 4:30 PM, David Boyce david.s.bo...@gmail.com wrote: I don't know why this hasn't occurred to me or the authors of similar programs before, but it appears to be possible to get a lock on any writable file descriptor - for instance stdout or stderr, or one of the jobserver-fds. I just changed syncsh to synchronize around a lock on stdout and it seems to work. This looks much more elegant than specifying a writable file. Attached is an updated patch with some improvements: - The feature is turned on by a special target currently called .PARALLELSYNC. No filename required. - It uses stdout as its semaphore. - It combines output into one temp file iff the inherited stdout and stderr are already combined, as suggested by Howard Chu. Very lightly tested, of course. See the Makefile.synctest for a usage example. David make-sync.patch Description: Binary data Makefile.synctest Description: Binary data ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Hello everybody. On Friday 15 April 2011, Eli Zaretskii wrote: Btw, there will be other side effects, at least on non-Posix platforms, due to the fact that stuff that was supposed to go to the screen is redirected to a file instead. Some programs sense that and behave differently, e.g. with binary non-printable characters or with special character sequences. By redirecting the output to files, then displaying it on the screen, the output may look strangely, or have some more grave effect, like terminating output prematurely. We should consider these potential problems, and in any case the simple loop that reads from temporary files and writes to stdout/stderr may need to become much more complicated, at least on DOS/Windows. (I believe similar, though less serious, problems could happen on Unix as well.) For a real-life example of a (minor) problem of this kind that we have encountered in Automake, see: http://www.mail-archive.com/automake-patches@gnu.org/msg03641.html Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, Apr 15, 2011 at 9:48 AM, Eli Zaretskii e...@gnu.org wrote: So effectively, whenever a job finishes, its parent Make takes the semaphore, outputs all of the output of that job to the screen, then releases the semaphore, is that right? Yes, exactly. And how do you communicate the name of the semaphore file to sub-Make's? Is that by assuming that its file descriptor is inherited? I also don't understand the need in the MAKESYNCFILE variable: why is it important where the semaphore file will live and how will it be named? Addressed by second iteration. There is no sync file anymore. Finally, wouldn't it be a potential problem top inherit so many handles to subordinate processes (2 for each running job)? We could run out of available handles in deeply recursive jobs, couldn't we? Just to be completely clear, we're not talking about a change to default behavior - it's merely a new option. But yes, any feature which consumes additional resources increases the risk of running out of that resource. I don't know that recursion comes into it though, it's more a question of how many parallel jobs a given make process is managing since limits on file handles/descriptors are per process. In other words if using the jobserver you're limited to the number of jobs specified at the top level even in a recursive setup. If not, make tries to work around that by forcing -j1, and if that doesn't work you're in trouble anyway. The strictest file limit I know of is on old Unixy platforms. The tmpfile() API returns a FILE *, not a file descriptor, and older Unices have a limit of 256 of these. But even in that corner case, how many people are going to run make -j200+? How many 64-CPU processors are running really old kernels? So yes, while there is a theoretical limit there are many kinds of resource exhaustion which can be triggered by large values of -j and as far as I can tell this is just another, and optional. Parenthetically, I only use FILE * because that's what tmpfile hands back. I believe it would be possible to dup the file descriptor out of the FILE *, close it right away, and work with an integral file descriptor from there. That would avoid the above limit. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Date: Fri, 15 Apr 2011 10:37:13 -0400 From: David Boyce david.s.bo...@gmail.com Cc: psm...@gnu.org, bug-make@gnu.org Finally, wouldn't it be a potential problem top inherit so many handles to subordinate processes (2 for each running job)? We could run out of available handles in deeply recursive jobs, couldn't we? Just to be completely clear, we're not talking about a change to default behavior - it's merely a new option. But this new option uses up 2 additional files per job, doesn't it? But yes, any feature which consumes additional resources increases the risk of running out of that resource. I don't know that recursion comes into it though, Each sub-Make inherits all the file descriptors of all its parents, grandparents, etc. If a sub-Make was launched when 4 other jobs were running in parallel, the sub-Make will inherit 8 file descriptors that it won't use. The deeper the recursion, the more inherited descriptors. it's more a question of how many parallel jobs a given make process is managing since limits on file handles/descriptors are per process. What about people who use make -j without limits? The strictest file limit I know of is on old Unixy platforms. The tmpfile() API returns a FILE *, not a file descriptor, and older Unices have a limit of 256 of these. But even in that corner case, how many people are going to run make -j200+? How many 64-CPU processors are running really old kernels? It's not like having a 256-core machine is a fantasy that will not happen in a few years. On my day job, we are already using 12 cores and don't consider that living on the edge. Parenthetically, I only use FILE * because that's what tmpfile hands back. I believe it would be possible to dup the file descriptor out of the FILE *, close it right away, and work with an integral file descriptor from there. That would avoid the above limit. I think the real kernel limitation, if there is one, is on file descriptors, not FILE objects. The latter live in the application. Anyway, if no one is worried about the inherited handles, just ignore me on that. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, Apr 15, 2011 at 11:09 AM, Eli Zaretskii e...@gnu.org wrote: But this new option uses up 2 additional files per job, doesn't it? One or two, as discussed elsewhere in this thread. Each sub-Make inherits all the file descriptors of all its parents, grandparents, etc. If a sub-Make was launched when 4 other jobs were running in parallel, the sub-Make will inherit 8 file descriptors that it won't use. The deeper the recursion, the more inherited descriptors. You have a good point. I should probably turn on FD_CLOEXEC for these. What about people who use make -j without limits? As noted, those people are at risk of exhausting many resources. It's not like having a 256-core machine is a fantasy that will not happen in a few years. On my day job, we are already using 12 cores and don't consider that living on the edge. Right, but my point was the limits of which I speak are old. I don't think a 256-core processor will ever run on a kernel from the 1980's. Or in other words, part of scaling an OS to many cores involves removing or at least upping artificial resource constraints. The natural trend toward 64-bit systems is part of that of course. I think the real kernel limitation, if there is one, is on file descriptors, not FILE objects. The latter live in the application. On that you are wrong, at least WRT POSIX machines. The old limit on streams was a function of an 8-bit field in the FILE structure, whereas every system I've ever used can allocate at least 1024 descriptors. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Date: Fri, 15 Apr 2011 12:39:56 -0400 From: David Boyce david.s.bo...@gmail.com Cc: psm...@gnu.org, bug-make@gnu.org On Fri, Apr 15, 2011 at 11:09 AM, Eli Zaretskii e...@gnu.org wrote: But this new option uses up 2 additional files per job, doesn't it? One or two, as discussed elsewhere in this thread. Each sub-Make inherits all the file descriptors of all its parents, grandparents, etc. If a sub-Make was launched when 4 other jobs were running in parallel, the sub-Make will inherit 8 file descriptors that it won't use. The deeper the recursion, the more inherited descriptors. You have a good point. I should probably turn on FD_CLOEXEC for these. I agree. What about the other issue: with the fact that output from a recipe is only shown when the entire recipe is done. I think this is a serious drawback, at least in some use cases. Can anything be done about that? ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
What about the other issue: with the fact that output from a recipe is only shown when the entire recipe is done. I think this is a serious drawback, at least in some use cases. Can anything be done about that? The only thing that I can imagine is that one would allow one recipe to have the console for as long as it lasts and to postpone output from all other recipes till it finished. Having finished, the output from all other recipes that might have completed in the meantime would be flushed. The console would be given to the next uncompleted task. In this way many tasks would be so short as to give up the console quickly but a long running task would be sure to get the console as much as it needed it. Regards, Tim -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, Apr 15, 2011 at 1:03 PM, Eli Zaretskii e...@gnu.org wrote: Can Make be invoked with its stdout closed by the parent process? If it is, will this still work? I've tried to code it such that if anything goes wrong setting up the sync, of which a closed stdout would be one example, it prints a message and continues on without the sync feature but does not fail the build. Such corner cases have not been tested much. + memset(fl, 0, sizeof(fl)); + fl.l_type = F_WRLCK; + fl.l_whence = SEEK_SET; + fl.l_pid = getpid(); + fl.l_start = fl.l_len = 0; + if (fcntl(syncfd, F_SETLKW, fl) == -1) + perror(fcntl); As discussed, please factor this out into a separate function. As soon as things settle down. + if (c-tempout) + while ((nread = fread(buffer, 1, sizeof(buffer), c-tempout)) 0) + write(fileno(stdout), buffer, nread); Are we sure c-tempout will be zeroed out by initialization? They're explicitly initialized: + child-tempout = child-temperr = NULL; + if (c-tempout) + fclose(c-tempout); + if (c-temperr) + fclose(c-temperr); Should we assign NULL to these once they are closed? Yes. I'm working on a new version which replaces these with descriptors anyway. Same principal applies though. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Eli Zaretskii wrote: Date: Fri, 15 Apr 2011 10:37:13 -0400 From: David Boycedavid.s.bo...@gmail.com it's more a question of how many parallel jobs a given make process is managing since limits on file handles/descriptors are per process. What about people who use make -j without limits? They're idiots. I've never seen a machine that can handle more than 1.5x number of cores without getting bottlenecked by I/O or system scheduler. (And I've run this stuff on 1024 node Connection Machines, among other things...) Go above a certain threshold and you're only slowing yourself down. It's not like having a 256-core machine is a fantasy that will not happen in a few years. On my day job, we are already using 12 cores and don't consider that living on the edge. We've already seen bug reports from people running on machines with 4096 cores, trying to run with -j8192 or somesuch. (Default pipe buffer is 4K which sets -j4096 as the practical limit without mucking with kernel params.) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, 2011-04-15 at 19:54 +0300, Eli Zaretskii wrote: What about the other issue: with the fact that output from a recipe is only shown when the entire recipe is done. I think this is a serious drawback, at least in some use cases. Can anything be done about that? I don't see how. The entire point of this feature is that all the output for each job is shown together as one group, rather than intermixed with the output of other jobs. I can't see how you can see the output from all jobs immediately and still keep the output from being mixed together; that seems akin to squaring a circle :-). If you want the output immediately, don't enable this feature. If you want the output to be grouped together, then enable this feature. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
I wish I could play with one of those Connection machines and find out about the bottlenecks at j=0.25*cpus that I can see on some machines :-( Regards, Tim On 15 April 2011 19:32, Howard Chu h...@highlandsun.com wrote: Eli Zaretskii wrote: Date: Fri, 15 Apr 2011 10:37:13 -0400 From: David Boycedavid.s.bo...@gmail.com it's more a question of how many parallel jobs a given make process is managing since limits on file handles/descriptors are per process. What about people who use make -j without limits? They're idiots. I've never seen a machine that can handle more than 1.5x number of cores without getting bottlenecked by I/O or system scheduler. (And I've run this stuff on 1024 node Connection Machines, among other things...) Go above a certain threshold and you're only slowing yourself down. It's not like having a 256-core machine is a fantasy that will not happen in a few years. On my day job, we are already using 12 cores and don't consider that living on the edge. We've already seen bug reports from people running on machines with 4096 cores, trying to run with -j8192 or somesuch. (Default pipe buffer is 4K which sets -j4096 as the practical limit without mucking with kernel params.) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
From: Paul Smith psm...@gnu.org CC: David Boyce david.s.bo...@gmail.com, bug-make@gnu.org Date: Fri, 15 Apr 2011 14:53:52 -0400 On Fri, 2011-04-15 at 19:54 +0300, Eli Zaretskii wrote: What about the other issue: with the fact that output from a recipe is only shown when the entire recipe is done. I think this is a serious drawback, at least in some use cases. Can anything be done about that? I don't see how. The entire point of this feature is that all the output for each job is shown together as one group, rather than intermixed with the output of other jobs. I can't see how you can see the output from all jobs immediately and still keep the output from being mixed together; that seems akin to squaring a circle :-). How about if we introduce a variable that can be set for specific targets (using the target-specific variable mechanism). Targets that set this variable to some predefined value will not have their output redirected, so their output will go to the screen. Make will hold the semaphore for the entire time the job for one of these targets runs, so that no other target can show its output until the last of the jobs for those targets exits. Would something like that work? ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, Apr 12, 2011 at 1:46 PM, David Boyce david.s.bo...@gmail.com wrote: So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? Ping? The original patch attachment was made by hand using diff -u and I had some trouble applying it myself. So here's a version created with cvs -q diff -uN which should work better. I've done copyright assignment paperwork in the past, BTW. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, 2011-04-14 at 11:01 -0400, David Boyce wrote: On Tue, Apr 12, 2011 at 1:46 PM, David Boyce david.s.bo...@gmail.com wrote: So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? Ping? The original patch attachment was made by hand using diff -u and I had some trouble applying it myself. So here's a version created with cvs -q diff -uN which should work better. I've looked at it and as a concept I don't have too many issues with it (although I'd like to hear from the ports) but I think there some outstanding issues that should be considered. One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
From: Paul Smith psm...@gnu.org Date: Thu, 14 Apr 2011 13:29:09 -0400 On Thu, 2011-04-14 at 11:01 -0400, David Boyce wrote: On Tue, Apr 12, 2011 at 1:46 PM, David Boyce david.s.bo...@gmail.com wrote: So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? Ping? The original patch attachment was made by hand using diff -u and I had some trouble applying it myself. So here's a version created with cvs -q diff -uN which should work better. I've looked at it and as a concept I don't have too many issues with it (although I'd like to hear from the ports) David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. TIA ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 1:29 PM, Paul Smith psm...@gnu.org wrote: I've looked at it and as a concept I don't have too many issues with it Thanks. One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. I actually think this is unavoidable. At least it's unavoidable when using a separate shell wrapper; it may be possible to better internally to make with some extra work. The reason is that the SHELL variable is used not only for recipes but also for the $(shell) function. Intermingling stdout and stderr in the result of $(shell) is just disastrously wrong (as I found in the first iteration of syncsh). I spent some time trying to find a way to determine, from inside a child shell, whether we were forked by a recipe or by $(shell) but could find no reliable way. Thus syncsh was forced to keep them in separate files. Since $(shell) invocations are not jobs according to make's process model there's no need for them to participate in synchronization at all, so it may be that within make there's a way to only sync on recipes which in turn would allow 21. Of course, either way some context is lost. If you put both into one temp file you lose track of which was which; if you keep them in separate files you lose ordering instead. So it becomes a matter of taste, or perhaps an option though that seems like a bit too much to me. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, 2011-04-14 at 20:48 +0300, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Date: Thu, 14 Apr 2011 13:29:09 -0400 On Thu, 2011-04-14 at 11:01 -0400, David Boyce wrote: On Tue, Apr 12, 2011 at 1:46 PM, David Boyce david.s.bo...@gmail.com wrote: So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? Ping? The original patch attachment was made by hand using diff -u and I had some trouble applying it myself. So here's a version created with cvs -q diff -uN which should work better. I've looked at it and as a concept I don't have too many issues with it (although I'd like to hear from the ports) David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. David wants to interlock between ALL instances of make printing output, so that even during recursive makes no matter how many you have running concurrently, only one will print its output at a time. There is no specific region of the file that's locked: the lockfile is basically a file-based, system-wide semaphore. The entire file is locked; it's empty and has no content. I'm not sure I like the idea of having to define a lockfile to enable this feature though. It feels a little too much like exposing the implementation details to the user. Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. Definitely. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 1:48 PM, Eli Zaretskii e...@gnu.org wrote: David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. I was about to write this up but I see Paul beat me to it ... Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. Agree. Ifdef bad. Just wanted to see if the feature would be favorably looked on first. I can't actually do the Windows etc ports but I could certainly refactor it with stubs. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
File locking would be the exclusion mechanism with a shell implementation, right? With Talon we used a system semaphore with timeout-wait semantics . I would assume that one doesn't need it if the code is in make since make can choose when to start and stop reading from any particular file handle. In Talon we don't use files but an expanding memory buffer instead. Is this better/worse? I don't know. Our huge builds don't really seem to suffer memory shortages from this particular source. I have seen a commercial system that limits the total output - i.e. it won't take more than X megabytes of output per task and in very large parallel builds there are good reasons for having limits like this anyhow. Regards, Tim On 14 April 2011 19:12, David Boyce david.s.bo...@gmail.com wrote: On Thu, Apr 14, 2011 at 1:48 PM, Eli Zaretskii e...@gnu.org wrote: David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. I was about to write this up but I see Paul beat me to it ... Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. Agree. Ifdef bad. Just wanted to see if the feature would be favorably looked on first. I can't actually do the Windows etc ports but I could certainly refactor it with stubs. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Date: Thu, 14 Apr 2011 14:12:16 -0400 From: David Boyce david.s.bo...@gmail.com Cc: psm...@gnu.org, bug-make@gnu.org On Thu, Apr 14, 2011 at 1:48 PM, Eli Zaretskii e...@gnu.org wrote: David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. I was about to write this up but I see Paul beat me to it ... Yes, but a few words about how is this semaphore supposed to get job done, and in fact what kind of synchronization will this bring to Make, would be appreciated. I don't think you described the feature too much in your original post. Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. Agree. Ifdef bad. Just wanted to see if the feature would be favorably looked on first. I can't actually do the Windows etc ports but I could certainly refactor it with stubs. If you refactor the Unix code, adding the Windows equivalent is easy, so you don't even need to add stubs. Thanks. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Paul Smith psm...@gnu.org writes: One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. FWIW, A commercial distributed make system I use at work has a --x-mergestreams command line option to control whether stdout stderr are merged or not. By default, the streams are merged. I tend to agree that this is the more useful behavior. --jtc -- J.T. Conklin ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 2:10 PM, Paul Smith psm...@gnu.org wrote: There is no specific region of the file that's locked: the lockfile is basically a file-based, system-wide semaphore. Yes, it's conceptually a semaphore. In fact a Windows port might prefer to use real semaphores. The reason I stayed away from them on the POSIX side is that, according to my reading, POSIX semaphores don't go away automatically when the creating process exits; they need to be explicitly released. Which entails a lot of responsibility and complexity in terms of signal handling and so on. An exclusive file lock produces the same result but goes away with its file descriptor. File locking is also older and thus likely a little more portable, though both are pretty old. The entire file is [...] empty and has no content. That's not actually the case with the current implementation. The file can have content, in fact the obvious candidate is often the Makefile as shown in the test case. It must be writable according to fcntl locking semantics but it is never written to. I'm not sure I like the idea of having to define a lockfile to enable this feature though. It feels a little too much like exposing the implementation details to the user. Agree. This needs a little more thought. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 3:00 PM, Eli Zaretskii e...@gnu.org wrote: Yes, but a few words about how is this semaphore supposed to get job done, and in fact what kind of synchronization will this bring to Make, would be appreciated. I don't think you described the feature too much in your original post. No, but what I did say was I won't go into how it works because that's already done at the URLs above. Do you have specific questions not addressed there? They're both pretty short, and I really can't tell what more needs to be said without some context. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Ok, To prevent any kind of deadlock you sort of want to empty everything the stderr and stdout pipe buffers may contain. It is conceivable that the stdout pipe might contain two lines of text and stderr pipe might contain 1 by the time your select statement (or waitformultipleobjects) has woken up. How do you know what order they should be in in the output? SeS, SSe or eSS? So one theory goes -ah forget it. Regards Tim On 14 April 2011 20:46, Tim Murphy tnmur...@gmail.com wrote: That was a confused email - apologies. I will rethink it and return. :-) On 14 April 2011 20:43, Tim Murphy tnmur...@gmail.com wrote: -- Forwarded message -- From: Tim Murphy tnmur...@gmail.com Date: 14 April 2011 20:43 Subject: Re: patch to support output synchronization under -j To: psm...@gnu.org The reason for splitting stderr and stdout is to do with deadlock and reading pipes. IIRC. e.g. blocking on a read to stderr which will never return because the process is stuck waiting for you to read from it's stdout. I think it's all easier on an os where you can create ptys (is that the term?) i.e. fake consoles to which stdout and stderr are both attached. I am going to be lazy and let someone else justify this or shoot it down. Perhaps I will remember why I got into trouble with it a long time ago. In practice the non-interleaving is actually nice in examples I've seen because the context is obvious from the error message itself (otherwise its' a crap error message and needs updating) and the division makes it a little easier to scan for error messages automatically. It's more of a problem where you're running a task that executes some long sub-build. Regards, Tim On 14 April 2011 20:16, Paul Smith psm...@gnu.org wrote: On Thu, 2011-04-14 at 14:08 -0400, David Boyce wrote: On Thu, Apr 14, 2011 at 1:29 PM, Paul Smith psm...@gnu.org wrote: One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. The reason is that the SHELL variable is used not only for recipes but also for the $(shell) function. Intermingling stdout and stderr in the result of $(shell) is just disastrously wrong (as I found in the first iteration of syncsh). I spent some time trying to find a way to determine, from inside a child shell, whether we were forked by a recipe or by $(shell) but could find no reliable way. Thus syncsh was forced to keep them in separate files. Since $(shell) invocations are not jobs according to make's process model there's no need for them to participate in synchronization at all, so it may be that within make there's a way to only sync on recipes which in turn would allow 21. Your latter statement is absolutely correct: it's wrong for $(shell ...) to synchronize. Shell function output is captured by make, not printed to stdout, so synchronizing it doesn't make much sense. Of course, either way some context is lost. If you put both into one temp file you lose track of which was which; if you keep them in separate files you lose ordering instead. So it becomes a matter of taste, or perhaps an option though that seems like a bit too much to me. I agree that adding an option seems like a lot. I think it's more important to maintain ordering of stdout/stderr than it is to allow individual redirection. However, you could do both with some heuristics. Hm. Maybe not. I was going to say you could merge them if stdout and stderr were going to the same tty or file, but I don't think there's any good way in UNIX to know whether two file descriptors are pointing at the same file/device. Hrm. In Linux you can find out via /proc but that's a pretty special case. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk
Re: patch to support output synchronization under -j
On Thu, 2011-04-14 at 20:59 +0100, Tim Murphy wrote: To prevent any kind of deadlock you sort of want to empty everything the stderr and stdout pipe buffers may contain. It is conceivable that the stdout pipe might contain two lines of text and stderr pipe might contain 1 by the time your select statement (or waitformultipleobjects) has woken up. This would all be true, if anyone were using pipes and select statements... but we're not :-) The implementation David provided has all the output going to temporary files, which are then read and printed to stdout or stderr (and deleted) when the job gets the output sync lock. To choose between merged and not merged you just choose between creating distinct files, vs. creating one file and setting both stdout/stderr to write to it... same as it would write to the tty device. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 3:08 PM, David Boyce david.s.bo...@gmail.com wrote: On Thu, Apr 14, 2011 at 2:10 PM, Paul Smith psm...@gnu.org wrote: I'm not sure I like the idea of having to define a lockfile to enable this feature though. It feels a little too much like exposing the implementation details to the user. Agree. This needs a little more thought. I don't know why this hasn't occurred to me or the authors of similar programs before, but it appears to be possible to get a lock on any writable file descriptor - for instance stdout or stderr, or one of the jobserver-fds. I just changed syncsh to synchronize around a lock on stdout and it seems to work. This looks much more elegant than specifying a writable file. Or is my understanding of file/pipe/locking semantics flawed? And how would Windows/DOS/VMS/OS2 would do with this? David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
