Re: patch to support output synchronization under -j

2011-04-15 Thread Tim Murphy
Right, of course :-) That is the obvious reason for using files, thanks! I was biased against writing things to disc which is probably silly, because it's probably not slow when compared to the size of the job that's running. Stdout and stderr will now all be stdout, BTW (no matter what method

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
Date: Thu, 14 Apr 2011 16:30:42 -0400 From: David Boyce david.s.bo...@gmail.com Cc: Eli Zaretskii e...@gnu.org, bug-make@gnu.org I don't know why this hasn't occurred to me or the authors of similar programs before, but it appears to be possible to get a lock on any writable file

Re: patch to support output synchronization under -j

2011-04-15 Thread Tim Murphy
I replied to this without using reply all and I apologise. :-) The order of the output from separate recipes may be anything - depends on how make schedules them but at least the output from any particular recipe will be grouped together and not mixed with the output from others, hence allowing

Re: patch to support output synchronization under -j

2011-04-15 Thread Paul Smith
On Fri, 2011-04-15 at 07:18 +0100, Tim Murphy wrote: Right, of course :-) That is the obvious reason for using files, thanks! I was biased against writing things to disc which is probably silly, because it's probably not slow when compared to the size of the job that's running. I think you

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Fri, Apr 15, 2011 at 6:45 AM, Eli Zaretskii e...@gnu.org wrote: I lack the higher-level picture.  Can you describe in a few words how will this work in a running Make?  I don't mean the details of how files are locked and unlocked (I understand that part), I mean the larger picture, like

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
Date: Fri, 15 Apr 2011 12:43:53 +0100 From: Tim Murphy tnmur...@gmail.com Cc: David Boyce david.s.bo...@gmail.com, bug-make@gnu.org I think it's an inevitable consequence that if you have a long-running task then the output from it won't appear until it has completely finished and you

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Thu, Apr 14, 2011 at 4:30 PM, David Boyce david.s.bo...@gmail.com wrote: I don't know why this hasn't occurred to me or the authors of similar programs before, but it appears to be possible to get a lock on any writable file descriptor - for instance stdout or stderr, or one of the

Re: patch to support output synchronization under -j

2011-04-15 Thread Stefano Lattarini
Hello everybody. On Friday 15 April 2011, Eli Zaretskii wrote: Btw, there will be other side effects, at least on non-Posix platforms, due to the fact that stuff that was supposed to go to the screen is redirected to a file instead. Some programs sense that and behave differently, e.g. with

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Fri, Apr 15, 2011 at 9:48 AM, Eli Zaretskii e...@gnu.org wrote: So effectively, whenever a job finishes, its parent Make takes the semaphore, outputs all of the output of that job to the screen, then releases the semaphore, is that right? Yes, exactly. And how do you communicate the name

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
Date: Fri, 15 Apr 2011 10:37:13 -0400 From: David Boyce david.s.bo...@gmail.com Cc: psm...@gnu.org, bug-make@gnu.org Finally, wouldn't it be a potential problem top inherit so many handles to subordinate processes (2 for each running job)?  We could run out of available handles in

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Fri, Apr 15, 2011 at 11:09 AM, Eli Zaretskii e...@gnu.org wrote: But this new option uses up 2 additional files per job, doesn't it? One or two, as discussed elsewhere in this thread. Each sub-Make inherits all the file descriptors of all its parents, grandparents, etc.  If a sub-Make was

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
Date: Fri, 15 Apr 2011 12:39:56 -0400 From: David Boyce david.s.bo...@gmail.com Cc: psm...@gnu.org, bug-make@gnu.org On Fri, Apr 15, 2011 at 11:09 AM, Eli Zaretskii e...@gnu.org wrote: But this new option uses up 2 additional files per job, doesn't it? One or two, as discussed elsewhere

Re: patch to support output synchronization under -j

2011-04-15 Thread Tim Murphy
What about the other issue: with the fact that output from a recipe is only shown when the entire recipe is done. I think this is a serious drawback, at least in some use cases. Can anything be done about that? The only thing that I can imagine is that one would allow one recipe to have the

Re: patch to support output synchronization under -j

2011-04-15 Thread David Boyce
On Fri, Apr 15, 2011 at 1:03 PM, Eli Zaretskii e...@gnu.org wrote: Can Make be invoked with its stdout closed by the parent process?  If it is, will this still work? I've tried to code it such that if anything goes wrong setting up the sync, of which a closed stdout would be one example, it

Re: patch to support output synchronization under -j

2011-04-15 Thread Howard Chu
Eli Zaretskii wrote: Date: Fri, 15 Apr 2011 10:37:13 -0400 From: David Boycedavid.s.bo...@gmail.com it's more a question of how many parallel jobs a given make process is managing since limits on file handles/descriptors are per process. What about people who use make -j without limits?

Re: patch to support output synchronization under -j

2011-04-15 Thread Paul Smith
On Fri, 2011-04-15 at 19:54 +0300, Eli Zaretskii wrote: What about the other issue: with the fact that output from a recipe is only shown when the entire recipe is done. I think this is a serious drawback, at least in some use cases. Can anything be done about that? I don't see how. The

Re: patch to support output synchronization under -j

2011-04-15 Thread Tim Murphy
I wish I could play with one of those Connection machines and find out about the bottlenecks at j=0.25*cpus that I can see on some machines :-( Regards, Tim On 15 April 2011 19:32, Howard Chu h...@highlandsun.com wrote: Eli Zaretskii wrote: Date: Fri, 15 Apr 2011 10:37:13 -0400 From: David

Re: patch to support output synchronization under -j

2011-04-15 Thread Eli Zaretskii
From: Paul Smith psm...@gnu.org CC: David Boyce david.s.bo...@gmail.com, bug-make@gnu.org Date: Fri, 15 Apr 2011 14:53:52 -0400 On Fri, 2011-04-15 at 19:54 +0300, Eli Zaretskii wrote: What about the other issue: with the fact that output from a recipe is only shown when the entire recipe