Re: patch to support output synchronization under -j
On Tue, Apr 12, 2011 at 1:46 PM, David Boyce david.s.bo...@gmail.com wrote: So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? Ping? The original patch attachment was made by hand using diff -u and I had some trouble applying it myself. So here's a version created with cvs -q diff -uN which should work better. I've done copyright assignment paperwork in the past, BTW. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, 2011-04-14 at 11:01 -0400, David Boyce wrote: On Tue, Apr 12, 2011 at 1:46 PM, David Boyce david.s.bo...@gmail.com wrote: So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? Ping? The original patch attachment was made by hand using diff -u and I had some trouble applying it myself. So here's a version created with cvs -q diff -uN which should work better. I've looked at it and as a concept I don't have too many issues with it (although I'd like to hear from the ports) but I think there some outstanding issues that should be considered. One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
From: Paul Smith psm...@gnu.org Date: Thu, 14 Apr 2011 13:29:09 -0400 On Thu, 2011-04-14 at 11:01 -0400, David Boyce wrote: On Tue, Apr 12, 2011 at 1:46 PM, David Boyce david.s.bo...@gmail.com wrote: So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? Ping? The original patch attachment was made by hand using diff -u and I had some trouble applying it myself. So here's a version created with cvs -q diff -uN which should work better. I've looked at it and as a concept I don't have too many issues with it (although I'd like to hear from the ports) David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. TIA ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 1:29 PM, Paul Smith psm...@gnu.org wrote: I've looked at it and as a concept I don't have too many issues with it Thanks. One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. I actually think this is unavoidable. At least it's unavoidable when using a separate shell wrapper; it may be possible to better internally to make with some extra work. The reason is that the SHELL variable is used not only for recipes but also for the $(shell) function. Intermingling stdout and stderr in the result of $(shell) is just disastrously wrong (as I found in the first iteration of syncsh). I spent some time trying to find a way to determine, from inside a child shell, whether we were forked by a recipe or by $(shell) but could find no reliable way. Thus syncsh was forced to keep them in separate files. Since $(shell) invocations are not jobs according to make's process model there's no need for them to participate in synchronization at all, so it may be that within make there's a way to only sync on recipes which in turn would allow 21. Of course, either way some context is lost. If you put both into one temp file you lose track of which was which; if you keep them in separate files you lose ordering instead. So it becomes a matter of taste, or perhaps an option though that seems like a bit too much to me. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, 2011-04-14 at 20:48 +0300, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Date: Thu, 14 Apr 2011 13:29:09 -0400 On Thu, 2011-04-14 at 11:01 -0400, David Boyce wrote: On Tue, Apr 12, 2011 at 1:46 PM, David Boyce david.s.bo...@gmail.com wrote: So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? Ping? The original patch attachment was made by hand using diff -u and I had some trouble applying it myself. So here's a version created with cvs -q diff -uN which should work better. I've looked at it and as a concept I don't have too many issues with it (although I'd like to hear from the ports) David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. David wants to interlock between ALL instances of make printing output, so that even during recursive makes no matter how many you have running concurrently, only one will print its output at a time. There is no specific region of the file that's locked: the lockfile is basically a file-based, system-wide semaphore. The entire file is locked; it's empty and has no content. I'm not sure I like the idea of having to define a lockfile to enable this feature though. It feels a little too much like exposing the implementation details to the user. Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. Definitely. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 1:48 PM, Eli Zaretskii e...@gnu.org wrote: David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. I was about to write this up but I see Paul beat me to it ... Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. Agree. Ifdef bad. Just wanted to see if the feature would be favorably looked on first. I can't actually do the Windows etc ports but I could certainly refactor it with stubs. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
File locking would be the exclusion mechanism with a shell implementation, right? With Talon we used a system semaphore with timeout-wait semantics . I would assume that one doesn't need it if the code is in make since make can choose when to start and stop reading from any particular file handle. In Talon we don't use files but an expanding memory buffer instead. Is this better/worse? I don't know. Our huge builds don't really seem to suffer memory shortages from this particular source. I have seen a commercial system that limits the total output - i.e. it won't take more than X megabytes of output per task and in very large parallel builds there are good reasons for having limits like this anyhow. Regards, Tim On 14 April 2011 19:12, David Boyce david.s.bo...@gmail.com wrote: On Thu, Apr 14, 2011 at 1:48 PM, Eli Zaretskii e...@gnu.org wrote: David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. I was about to write this up but I see Paul beat me to it ... Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. Agree. Ifdef bad. Just wanted to see if the feature would be favorably looked on first. I can't actually do the Windows etc ports but I could certainly refactor it with stubs. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Date: Thu, 14 Apr 2011 14:12:16 -0400 From: David Boyce david.s.bo...@gmail.com Cc: psm...@gnu.org, bug-make@gnu.org On Thu, Apr 14, 2011 at 1:48 PM, Eli Zaretskii e...@gnu.org wrote: David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. I was about to write this up but I see Paul beat me to it ... Yes, but a few words about how is this semaphore supposed to get job done, and in fact what kind of synchronization will this bring to Make, would be appreciated. I don't think you described the feature too much in your original post. Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. Agree. Ifdef bad. Just wanted to see if the feature would be favorably looked on first. I can't actually do the Windows etc ports but I could certainly refactor it with stubs. If you refactor the Unix code, adding the Windows equivalent is easy, so you don't even need to add stubs. Thanks. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Paul Smith psm...@gnu.org writes: One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. FWIW, A commercial distributed make system I use at work has a --x-mergestreams command line option to control whether stdout stderr are merged or not. By default, the streams are merged. I tend to agree that this is the more useful behavior. --jtc -- J.T. Conklin ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 2:10 PM, Paul Smith psm...@gnu.org wrote: There is no specific region of the file that's locked: the lockfile is basically a file-based, system-wide semaphore. Yes, it's conceptually a semaphore. In fact a Windows port might prefer to use real semaphores. The reason I stayed away from them on the POSIX side is that, according to my reading, POSIX semaphores don't go away automatically when the creating process exits; they need to be explicitly released. Which entails a lot of responsibility and complexity in terms of signal handling and so on. An exclusive file lock produces the same result but goes away with its file descriptor. File locking is also older and thus likely a little more portable, though both are pretty old. The entire file is [...] empty and has no content. That's not actually the case with the current implementation. The file can have content, in fact the obvious candidate is often the Makefile as shown in the test case. It must be writable according to fcntl locking semantics but it is never written to. I'm not sure I like the idea of having to define a lockfile to enable this feature though. It feels a little too much like exposing the implementation details to the user. Agree. This needs a little more thought. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 3:00 PM, Eli Zaretskii e...@gnu.org wrote: Yes, but a few words about how is this semaphore supposed to get job done, and in fact what kind of synchronization will this bring to Make, would be appreciated. I don't think you described the feature too much in your original post. No, but what I did say was I won't go into how it works because that's already done at the URLs above. Do you have specific questions not addressed there? They're both pretty short, and I really can't tell what more needs to be said without some context. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Fwd: patch to support output synchronization under -j
-- Forwarded message -- From: Tim Murphy tnmur...@gmail.com Date: 14 April 2011 20:43 Subject: Re: patch to support output synchronization under -j To: psm...@gnu.org The reason for splitting stderr and stdout is to do with deadlock and reading pipes. IIRC. e.g. blocking on a read to stderr which will never return because the process is stuck waiting for you to read from it's stdout. I think it's all easier on an os where you can create ptys (is that the term?) i.e. fake consoles to which stdout and stderr are both attached. I am going to be lazy and let someone else justify this or shoot it down. Perhaps I will remember why I got into trouble with it a long time ago. In practice the non-interleaving is actually nice in examples I've seen because the context is obvious from the error message itself (otherwise its' a crap error message and needs updating) and the division makes it a little easier to scan for error messages automatically. It's more of a problem where you're running a task that executes some long sub-build. Regards, Tim On 14 April 2011 20:16, Paul Smith psm...@gnu.org wrote: On Thu, 2011-04-14 at 14:08 -0400, David Boyce wrote: On Thu, Apr 14, 2011 at 1:29 PM, Paul Smith psm...@gnu.org wrote: One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. The reason is that the SHELL variable is used not only for recipes but also for the $(shell) function. Intermingling stdout and stderr in the result of $(shell) is just disastrously wrong (as I found in the first iteration of syncsh). I spent some time trying to find a way to determine, from inside a child shell, whether we were forked by a recipe or by $(shell) but could find no reliable way. Thus syncsh was forced to keep them in separate files. Since $(shell) invocations are not jobs according to make's process model there's no need for them to participate in synchronization at all, so it may be that within make there's a way to only sync on recipes which in turn would allow 21. Your latter statement is absolutely correct: it's wrong for $(shell ...) to synchronize. Shell function output is captured by make, not printed to stdout, so synchronizing it doesn't make much sense. Of course, either way some context is lost. If you put both into one temp file you lose track of which was which; if you keep them in separate files you lose ordering instead. So it becomes a matter of taste, or perhaps an option though that seems like a bit too much to me. I agree that adding an option seems like a lot. I think it's more important to maintain ordering of stdout/stderr than it is to allow individual redirection. However, you could do both with some heuristics. Hm. Maybe not. I was going to say you could merge them if stdout and stderr were going to the same tty or file, but I don't think there's any good way in UNIX to know whether two file descriptors are pointing at the same file/device. Hrm. In Linux you can find out via /proc but that's a pretty special case. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Ok, To prevent any kind of deadlock you sort of want to empty everything the stderr and stdout pipe buffers may contain. It is conceivable that the stdout pipe might contain two lines of text and stderr pipe might contain 1 by the time your select statement (or waitformultipleobjects) has woken up. How do you know what order they should be in in the output? SeS, SSe or eSS? So one theory goes -ah forget it. Regards Tim On 14 April 2011 20:46, Tim Murphy tnmur...@gmail.com wrote: That was a confused email - apologies. I will rethink it and return. :-) On 14 April 2011 20:43, Tim Murphy tnmur...@gmail.com wrote: -- Forwarded message -- From: Tim Murphy tnmur...@gmail.com Date: 14 April 2011 20:43 Subject: Re: patch to support output synchronization under -j To: psm...@gnu.org The reason for splitting stderr and stdout is to do with deadlock and reading pipes. IIRC. e.g. blocking on a read to stderr which will never return because the process is stuck waiting for you to read from it's stdout. I think it's all easier on an os where you can create ptys (is that the term?) i.e. fake consoles to which stdout and stderr are both attached. I am going to be lazy and let someone else justify this or shoot it down. Perhaps I will remember why I got into trouble with it a long time ago. In practice the non-interleaving is actually nice in examples I've seen because the context is obvious from the error message itself (otherwise its' a crap error message and needs updating) and the division makes it a little easier to scan for error messages automatically. It's more of a problem where you're running a task that executes some long sub-build. Regards, Tim On 14 April 2011 20:16, Paul Smith psm...@gnu.org wrote: On Thu, 2011-04-14 at 14:08 -0400, David Boyce wrote: On Thu, Apr 14, 2011 at 1:29 PM, Paul Smith psm...@gnu.org wrote: One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. The reason is that the SHELL variable is used not only for recipes but also for the $(shell) function. Intermingling stdout and stderr in the result of $(shell) is just disastrously wrong (as I found in the first iteration of syncsh). I spent some time trying to find a way to determine, from inside a child shell, whether we were forked by a recipe or by $(shell) but could find no reliable way. Thus syncsh was forced to keep them in separate files. Since $(shell) invocations are not jobs according to make's process model there's no need for them to participate in synchronization at all, so it may be that within make there's a way to only sync on recipes which in turn would allow 21. Your latter statement is absolutely correct: it's wrong for $(shell ...) to synchronize. Shell function output is captured by make, not printed to stdout, so synchronizing it doesn't make much sense. Of course, either way some context is lost. If you put both into one temp file you lose track of which was which; if you keep them in separate files you lose ordering instead. So it becomes a matter of taste, or perhaps an option though that seems like a bit too much to me. I agree that adding an option seems like a lot. I think it's more important to maintain ordering of stdout/stderr than it is to allow individual redirection. However, you could do both with some heuristics. Hm. Maybe not. I was going to say you could merge them if stdout and stderr were going to the same tty or file, but I don't think there's any good way in UNIX to know whether two file descriptors are pointing at the same file/device. Hrm. In Linux you can find out via /proc but that's a pretty special case. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___
Re: patch to support output synchronization under -j
On Thu, 2011-04-14 at 20:59 +0100, Tim Murphy wrote: To prevent any kind of deadlock you sort of want to empty everything the stderr and stdout pipe buffers may contain. It is conceivable that the stdout pipe might contain two lines of text and stderr pipe might contain 1 by the time your select statement (or waitformultipleobjects) has woken up. This would all be true, if anyone were using pipes and select statements... but we're not :-) The implementation David provided has all the output going to temporary files, which are then read and printed to stdout or stderr (and deleted) when the job gets the output sync lock. To choose between merged and not merged you just choose between creating distinct files, vs. creating one file and setting both stdout/stderr to write to it... same as it would write to the tty device. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 3:08 PM, David Boyce david.s.bo...@gmail.com wrote: On Thu, Apr 14, 2011 at 2:10 PM, Paul Smith psm...@gnu.org wrote: I'm not sure I like the idea of having to define a lockfile to enable this feature though. It feels a little too much like exposing the implementation details to the user. Agree. This needs a little more thought. I don't know why this hasn't occurred to me or the authors of similar programs before, but it appears to be possible to get a lock on any writable file descriptor - for instance stdout or stderr, or one of the jobserver-fds. I just changed syncsh to synchronize around a lock on stdout and it seems to work. This looks much more elegant than specifying a writable file. Or is my understanding of file/pipe/locking semantics flawed? And how would Windows/DOS/VMS/OS2 would do with this? David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
