Re: [bug #33138] .PARLLELSYNC enhancement with patch
Date: Fri, 19 Apr 2013 11:54:05 +0200 Cc: bo...@kolpackov.net, bug-make@gnu.org From: Frank Heckenbach f.heckenb...@fh-soft.de Eli Zaretskii wrote: Initial investigation indicates that tmpfile should do the job just fine: the file is deleted only when the last descriptor for it is closed. That includes any duplicated descriptors. Great. As for fcntl, F_SETLKW, and F_GETFD, they will need to be emulated. In particular, it looks like LockFileEx with LOCKFILE_EXCLUSIVE_LOCK flag set and LOCKFILE_FAIL_IMMEDIATELY flag cleared should do the job. I will need to see how it works in reality, though. OK. Upon a second look, I'm not sure I understand how this feature works, exactly, and why you-all thought making it work on Windows is a matter of a few functions. I sincerely hope I'm missing something, please bear with me. First, most of the meat of OUTPUT_SYNC code, which sets up the stage when running child jobs, is in a branch that isn't compiled on Windows (#if !defined(__MSDOS__) !defined(_AMIGA) !defined(WINDOWS32) on line 1482 of job.c). So currently that part is not even run on Windows. Please tell me that nothing in this feature relies on 'fork', with its copying of handles and other data structures. Because if it does, we have no hope of making it work on Windows, at least not using the same algorithms as on Unix. More importantly, how exactly locking the (redirected) stdout/stderr of the child is supposed to cause synchronization, and why do we need it at all? Isn't synchronization already achieved by redirecting child's output to a file, and only dumping it to screen when the child exits? What does lock add to this? Who else will be writing what to where, that we want to prevent by holding the lock/semaphore? In an old thread, Paul explained something similar: David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. David wants to interlock between ALL instances of make printing output, so that even during recursive makes no matter how many you have running concurrently, only one will print its output at a time. There is no specific region of the file that's locked: the lockfile is basically a file-based, system-wide semaphore. The entire file is locked; it's empty and has no content. Assuming this all is still basically true, I guess I still don't understand what exactly is being locked and why. E.g., why do we only want to interlock instances of Make, but not the programs they run? Also, acquire_semaphore is used only in sync_output, which is called only when a child exits. IOW, nothing is locked while the child runs, only when its output is ready. In addition, we are locking stdout. But doesn't each instance of Make have, or can have, its own stdout? If so, how will the interlock work? What am I missing? Probably a lot. TIA ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
Since you asked basic questions I'm going to start this at a basic level. Apologies if it covers some stuff you already know or if I misinterpreted the questions. Note that I haven't actually looked at the patch that went in so this is generally wrt the original. The first thing is get the word lock out of your mind because we aren't really locking anything. Yes, that API is in use but it's only to create a semaphore or baton. Nobody is ever prevented from doing anything. It just happens that on Unix the most portable (i.e. oldest) way of implementing a semaphore is with the advisory locking API. All cooperating processes agree not to proceed unless and until they are able to acquire the exclusive lock on a shared file descriptor, but it's not necessary to ever actually write anything to that descriptor. Second, the original implementation (not sure if I ever sent that one one in though) actually created a temp file to use as the semaphore fd. But then I discovered that stdout can be locked in the same way, which is simpler. But applying the lock to stdout is just a frill; it could be a temp file, especially if some platform turned out to need it that way. I just figured that stdout is always available, or at least if it's closed you don't have to worry about synchronizing output. Third, yes, nothing is locked while the child runs. If a shared resource was locked during child runs it would have the effect of re-serializing the build as each supposedly parallel child waited on the lock. So what happens here is really very simple: each child (aka recipe) runs asynchronously, assuming -j of course, and dumps its output to one or two temp files. Only when the child has finished and wants to report results does it enter the queue waiting for the baton. When it gets it, it holds it just long enough to copy its output from the temp files to stdout/stderr and then lets the next guy have his turn. Thus, assuming the average job runs for a significant amount of time (multiples of a write() system call anyway) there will not be much contention on the semaphore and it won't be a bottleneck. You're right that simply writing to temp files and dumping everything at once when the job finished would be likely to reduce the incidence of garbling even without the semaphore, but not to zero. It may be that the locking of stdout is only useful on Unix due to the fact that it's inherited into child processes. I don't know what Paul or Frank is thinking, and as mentioned I haven't looked at the current version, but my thinking originally was that Windows could easily handle this using its own far richer set of semaphore/locking APIs. I'd actually expect this to be easier and more natural on Windows than Unix. All that's required is to choose a semaphore to synchronize on, dump output to temp files, and copy it to stdout/stderr only after acquiring the semaphore. And remove the temp files of course. -David Boyce On Tue, Apr 23, 2013 at 10:50 AM, Eli Zaretskii e...@gnu.org wrote: Date: Fri, 19 Apr 2013 11:54:05 +0200 Cc: bo...@kolpackov.net, bug-make@gnu.org From: Frank Heckenbach f.heckenb...@fh-soft.de Eli Zaretskii wrote: Initial investigation indicates that tmpfile should do the job just fine: the file is deleted only when the last descriptor for it is closed. That includes any duplicated descriptors. Great. As for fcntl, F_SETLKW, and F_GETFD, they will need to be emulated. In particular, it looks like LockFileEx with LOCKFILE_EXCLUSIVE_LOCK flag set and LOCKFILE_FAIL_IMMEDIATELY flag cleared should do the job. I will need to see how it works in reality, though. OK. Upon a second look, I'm not sure I understand how this feature works, exactly, and why you-all thought making it work on Windows is a matter of a few functions. I sincerely hope I'm missing something, please bear with me. First, most of the meat of OUTPUT_SYNC code, which sets up the stage when running child jobs, is in a branch that isn't compiled on Windows (#if !defined(__MSDOS__) !defined(_AMIGA) !defined(WINDOWS32) on line 1482 of job.c). So currently that part is not even run on Windows. Please tell me that nothing in this feature relies on 'fork', with its copying of handles and other data structures. Because if it does, we have no hope of making it work on Windows, at least not using the same algorithms as on Unix. More importantly, how exactly locking the (redirected) stdout/stderr of the child is supposed to cause synchronization, and why do we need it at all? Isn't synchronization already achieved by redirecting child's output to a file, and only dumping it to screen when the child exits? What does lock add to this? Who else will be writing what to where, that we want to prevent by holding the lock/semaphore? In an old thread, Paul explained something similar: David, can you explain why you needed to lock the files? Also, what region(s) of the file you
Re: [bug #33138] .PARLLELSYNC enhancement with patch
Date: Tue, 23 Apr 2013 11:29:35 -0700 From: David Boyce david.s.bo...@gmail.com Cc: Frank Heckenbach f.heckenb...@fh-soft.de, bug-make bug-make@gnu.org Since you asked basic questions I'm going to start this at a basic level. Apologies if it covers some stuff you already know or if I misinterpreted the questions. Note that I haven't actually looked at the patch that went in so this is generally wrt the original. [...] Thanks, I will dwell on this. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Tue, 2013-04-23 at 21:40 +0300, Eli Zaretskii wrote: Date: Tue, 23 Apr 2013 11:29:35 -0700 From: David Boyce david.s.bo...@gmail.com Cc: Frank Heckenbach f.heckenb...@fh-soft.de, bug-make bug-make@gnu.org Since you asked basic questions I'm going to start this at a basic level. Apologies if it covers some stuff you already know or if I misinterpreted the questions. Note that I haven't actually looked at the patch that went in so this is generally wrt the original. [...] Thanks, I will dwell on this. When thinking about this, remember that it's not enough to consider how a single make invocation will work. If you run with a single make instance under -j, then redirecting each job's output to a temp file and then when make reaps each job, copying the contents of that temp file to stdout, is a sufficient solution. You just need to be able to redirect stdout/stderr of a given job to temporary files. In UNIX of course this is done by dup()'ing the file descriptors after the fork and before the exec. Presumably on Windows there's some other way to do this. However, in a situation where we have recursive make instances all running at the same time under -j then each one of those make instances is also running one or more jobs in parallel. In this case it's not good enough for each make to synchronize its own jobs' output. So in addition to the temp file change above, you ALSO need a way to synchronize the use of the single resource (stdout) that is being shared by all instances of recursive make. On UNIX we have chosen to use an advisory lock on the stdout file descriptor: it's handy, and it's the resource being contended for, so it makes sense. You asked, what if someone redirected the stdout of a sub-make. In that case things still work: that sub-make will not participate in the same locking as the other sub-makes, it's true, but that's OK because the output is going to a different location from the other sub-makes so there's no need to synchronize them. Meanwhile any sub-sub-makes using the same output file will still be synchronous with each other. I'm not sure if the lock locks the FD (so that if you dup'd the FD but it still pointed to the same output, you could take exclusive locks on both), or if it locks the underlying resource. If the former I guess it's possible to break the synchrony if you worked at it. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
From: Paul Smith psm...@gnu.org Cc: David Boyce david.s.bo...@gmail.com, f.heckenb...@fh-soft.de, bug-make@gnu.org Date: Tue, 23 Apr 2013 15:04:39 -0400 When thinking about this, remember that it's not enough to consider how a single make invocation will work. If you run with a single make instance under -j, then redirecting each job's output to a temp file and then when make reaps each job, copying the contents of that temp file to stdout, is a sufficient solution. You just need to be able to redirect stdout/stderr of a given job to temporary files. In UNIX of course this is done by dup()'ing the file descriptors after the fork and before the exec. Presumably on Windows there's some other way to do this. It's the same way, actually (modulo the fork/exec dance). So in addition to the temp file change above, you ALSO need a way to synchronize the use of the single resource (stdout) that is being shared by all instances of recursive make. On UNIX we have chosen to use an advisory lock on the stdout file descriptor: it's handy, and it's the resource being contended for, so it makes sense. I still don't know how does Make achieve on Unix the interlocking with its sub-Make's. Is it because the lock is inherited as part of fork? If so, we will need a special command-line argument on Windows to pass the name of the mutex, or the value of its handle, down the sub-make chain, because even if the handle is inherited, its value needs to be communicated. (I wish the design and implementation of this feature were less Posix-centric...) I'm not sure if the lock locks the FD (so that if you dup'd the FD but it still pointed to the same output, you could take exclusive locks on both), or if it locks the underlying resource. If the former I guess it's possible to break the synchrony if you worked at it. Sorry, you lost me here. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Tue, Apr 23, 2013 at 12:04 PM, Paul Smith psm...@gnu.org wrote: I'm not sure if the lock locks the FD (so that if you dup'd the FD but it still pointed to the same output, you could take exclusive locks on both), or if it locks the underlying resource. I'm pretty sure it's the underlying resource that gets locked. The man page is not quite explicit but implicitly fairly clear. The opening paragraph says: Apply, test or remove a POSIX lock on a section of an open file. The file is specified by fd, a file descriptor open for writing David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
(TL;DR: Probably irrelevant.)
Paul Smith wrote:
I'm not sure if the lock locks the FD (so that if you dup'd the FD but
it still pointed to the same output, you could take exclusive locks on
both), or if it locks the underlying resource. If the former I guess
it's possible to break the synchrony if you worked at it.
The Linux manpage says:
: As well as being removed by an explicit F_UNLCK, record locks are auto-
: matically released when the process terminates or if it closes any file
: descriptor referring to a file on which locks are held. This is bad:
: it means that a process can lose the locks on a file like /etc/passwd
: or /etc/mtab when for some reason a library function decides to open,
: read and close it.
:
: Record locks are not inherited by a child created via fork(2), but are
: preserved across an execve(2).
A quick test indicates that the lock indeed refers to the resource,
but several locks to the same resource are merged by the system
(rather than deadlocking itself):
#include fcntl.h
int main ()
{
int a = open (/dev/null, O_RDWR);
int b = open (/dev/null, O_RDWR); // or b = dup (a)
struct flock fl;
fl.l_type = F_WRLCK;
fl.l_whence = SEEK_SET;
fl.l_start = 0;
fl.l_len = 1;
if (fcntl (a, F_SETLKW, fl))
perror (a);
if (fcntl (b, F_SETLKW, fl))
perror (b);
return 0;
}
Though I don't think any of this really matters, at least with our
current implementation, since we don't dup the FD locked or fork or
exec during this time. And it's possible break the synchrony if you
worked at it since our locking is only advisory, anyway. (I.e.,
within make; not so easy in the jobs run since they don't see the FD
used for locking, i.e. the original stdout.)
PS: Paul, the following comment is obsolete; I forgot to remove it:
fl.l_start = 0; /* lock just one byte according to pid */
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
David Boyce wrote: The first thing is get the word lock out of your mind because we aren't really locking anything. Yes, that API is in use but it's only to create a semaphore or baton. Nobody is ever prevented from doing anything. It just happens that on Unix the most portable (i.e. oldest) way of implementing a semaphore is with the advisory locking API. All cooperating processes agree not to proceed unless and until they are able to acquire the exclusive lock on a shared file descriptor, but it's not necessary to ever actually write anything to that descriptor. Just to clarify: It's not necessary to write anything to that descriptor in order for the locking to work. We do actually write to the descriptor when we have the lock, but that's just an implementation detail, i.e. the lock could be something else, as long as the different make instances agree not to write to the descriptor without holding the lock. You're right that simply writing to temp files and dumping everything at once when the job finished would be likely to reduce the incidence of garbling even without the semaphore, but not to zero. It may be that the locking of stdout is only useful on Unix due to the fact that it's inherited into child processes. I don't know what Paul or Frank is thinking, and as mentioned I haven't looked at the current version, but my thinking originally was that Windows could easily handle this using its own far richer set of semaphore/locking APIs. I'd actually expect this to be easier and more natural on Windows than Unix. All that's required is to choose a semaphore to synchronize on, dump output to temp files, and copy it to stdout/stderr only after acquiring the semaphore. And remove the temp files of course. Yes, as I wrote in another mail, even a completely global semaphore should do. Sure, it's excluding much more than necessary, but since the critical section is very short, this shouldn't hurt much. (In other words, if make jobs produce such huge output that copying it around takes a significant amount of time, nobody's ever going to read that output anyway, or someone is post-processing it which will take much longer than copying it, anyway.) Indeed, as David wrote, under Unix, locking stdout/stderr is most straightforward because it's available without further setup. Incidentally, this way of locking is also as fine-grained as possible (considering recursive makes). However, as I wrote, this fine-grained locking is not really necessary, so it's not worth worrying about replicating it on Windows if this causes trouble. On Tue, Apr 23, 2013 at 10:50 AM, Eli Zaretskii e...@gnu.org wrote: Please tell me that nothing in this feature relies on 'fork', with its copying of handles and other data structures. All it requires is inheriting the redirected stdout/stderr to child processes. This was already possible under Dos (with the exception that since there was no fork, you had to redirect in the parent process, call the child, then undirect in the parent, IIRC). It's just like shell redirections, i.e. if you do foo bar baz, the stdout of foo and all processes called by foo is redirected to a file called bar, while baz and the rest of the shell continue running with their original stdout. If that's the same under Windows (at least using bash or a similar shell; no idea what Windows's own shell does), there should be no problem (or the bash sources should reveal what needs to be done). Perhaps you know all of this already and perhaps it's trivial, or perhaps it's impossible ... (I really don't know how different things are under Windows.) In an old thread, Paul explained something similar: David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. David wants to interlock between ALL instances of make printing output, so that even during recursive makes no matter how many you have running concurrently, only one will print its output at a time. There is no specific region of the file that's locked: the lockfile is basically a file-based, system-wide semaphore. The entire file is locked; it's empty and has no content. Assuming this all is still basically true, It's almost still true. As written above, we now don't use an extra, empty, file for locking, but stdout itself. Otherwise it's still so, we don't lock a particular region of the file, but use it as a semaphore. I guess I still don't understand what exactly is being locked and why. E.g., why do we only want to interlock instances of Make, but not the programs they run? Also, acquire_semaphore is used only in sync_output, which is called only when a child exits. IOW, nothing is locked while the child runs, only when its output is ready. As David wrote, that's necessary to preserve parallelism. In
Re: [bug #33138] .PARLLELSYNC enhancement with patch
Date: Tue, 23 Apr 2013 21:41:22 +0200 From: Frank Heckenbach f.heckenb...@fh-soft.de Yes, as I wrote in another mail, even a completely global semaphore should do. Not healthy, IMHO. Some snafu could inadvertently and completely silently stop an unrelated build somewhere on the same system. Sure, it's excluding much more than necessary, but since the critical section is very short, this shouldn't hurt much. (In other words, if make jobs produce such huge output that copying it around takes a significant amount of time, nobody's ever going to read that output anyway, or someone is post-processing it which will take much longer than copying it, anyway.) It doesn't matter whether someone reads the output or not, that output gets written and takes its time. We sometimes invoke tools with options we later regret, because we don't anticipate the results. Indeed, as David wrote, under Unix, locking stdout/stderr is most straightforward because it's available without further setup. Incidentally, this way of locking is also as fine-grained as possible (considering recursive makes). However, as I wrote, this fine-grained locking is not really necessary, so it's not worth worrying about replicating it on Windows if this causes trouble. It is not easy to know what needs to be replicated and what doesn't, because the fine print of the implementation is not clear, at least to me, and therefore it could easily happen that non-replication of some seemingly secondary detail completely breaks the design. After all, the basic models of Unix and Windows regarding processes and signals are very different. On Tue, Apr 23, 2013 at 10:50 AM, Eli Zaretskii e...@gnu.org wrote: Please tell me that nothing in this feature relies on 'fork', with its copying of handles and other data structures. All it requires is inheriting the redirected stdout/stderr to child processes. This was already possible under Dos (with the exception that since there was no fork, you had to redirect in the parent process, call the child, then undirect in the parent, IIRC). Inheritance is not the problem; _finding_ the inherited object or its handle is. With stdout, its name is global, but not so with a handle to some mutex I can create or its name (unless we indeed make the name fixed and thus system-global, which I don't like, if only because it's not how Make works on Unix). It's just like shell redirections, i.e. if you do foo bar baz, the stdout of foo and all processes called by foo is redirected to a file called bar, while baz and the rest of the shell continue running with their original stdout. If that's the same under Windows (at least using bash or a similar shell; no idea what Windows's own shell does), there should be no problem (or the bash sources should reveal what needs to be done). Perhaps you know all of this already and perhaps it's trivial, or perhaps it's impossible ... (I really don't know how different things are under Windows.) They aren't different, and all this is pretty basic. It's not what worries me. They can have their own stdout, in particular with the --output-sync=make option. But that's actually the harmless case: Each sub-make runs with its stdout already redirected to a temp file by the main make. In turn, it redirects the stdout of its children to separate temp files, and when they are done, collects the data to its stdout, i.e. the temp file from the main make. When the sub make is finished, the main make collects its output to the original stdout. So unless I'm mistaken, no locking is actually required in this case. But we still acquire the semaphore in this case, don't we? Perhaps we shouldn't. I still don't know how does Make achieve on Unix the interlocking with its sub-Make's. Is it because the lock is inherited as part of fork? The fd is inherited as part of fork. Each make that needs to takes a lock on the fd. On the same fd, or, rather, on an fd connected to the same resource. That's what I thought (it isn't spelled out anywhere in the comments, and the code that implements that is quite scattered). I need to check whether LockFileEx works on handles connected to the console; somehow I'm not sure. And LockFileEx is not ideal because it doesn't work on Windows 9X. So maybe I will have to use a mutex after all. (I wish the design and implementation of this feature were less Posix-centric...) The implementation my be (though it's only two functions, acquire_semaphore(), release_semaphore()) that can be completely replaced (note, again, the fact that the stdout or stderr fd also serves for locking, is just an implementation detail and not central to the design). I beg to differ. The implementation relies heavily on the fact that the semaphore's name or descriptor is globally known to every program in the chain. Writing a replacement for F_SETLKW using LockFileEx took me no more than 5 minutes; it is only when I
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Tue, 2013-04-23 at 23:16 +0300, Eli Zaretskii wrote: All it requires is inheriting the redirected stdout/stderr to child processes. This was already possible under Dos (with the exception that since there was no fork, you had to redirect in the parent process, call the child, then undirect in the parent, IIRC). Inheritance is not the problem; _finding_ the inherited object or its handle is. With stdout, its name is global, but not so with a handle to some mutex I can create or its name (unless we indeed make the name fixed and thus system-global, which I don't like, if only because it's not how Make works on Unix). You're right: we cannot use a fixed name. That would allow only one (top-level) invocation of make per system to use -O. That's not acceptable. They can have their own stdout, in particular with the --output-sync=make option. But that's actually the harmless case: Each sub-make runs with its stdout already redirected to a temp file by the main make. In turn, it redirects the stdout of its children to separate temp files, and when they are done, collects the data to its stdout, i.e. the temp file from the main make. When the sub make is finished, the main make collects its output to the original stdout. So unless I'm mistaken, no locking is actually required in this case. But we still acquire the semaphore in this case, don't we? Perhaps we shouldn't. We do acquire it but it's on a different resource. We still do need the semaphore because we want the output directed to the other file to be synchronized as well, at least within itself. If you run make -O and your output to stdout is synchronized by you run make -O out and the contents of the out file are not synchronized, that would be bogus. However, I'm not worried about losing this ability to change the lock domain for sub-makes based on redirection. If all the sub-makes were synchronized together even though some redirected to a different file that is fine. First, I sincerely doubt there are many makefiles which run sub-makes with IO redirected like that. And second, the slight loss in parallelization is not worth a huge increase in complexity. It's a nice thing that we get it for free with stdout but I wouldn't expend days trying to implement it if we didn't... in fact without stdout I just don't see that it's even possible. I guess if the child could somehow detect that it was using a different stdout it could be done, but that would require the parent passing information about stdout to the child... ugh. Not worth it. IOW, the top-level design is indeed quite general, but the low-level algorithmic details are not, and therefore just replacing these functions will not necessarily do the job. Right. The locking is the part that's not portable, and we need a lock that can be shared between all the make instances for one top-level make, but not shared with another top-level make. Without knowing what kind of resource Windows can take locks on, we can't really know how to help with that. There must be precedent for this kind of shared between a subset of processes lock on Windows. What handle do we need to know, and how can that handle be communicated? We can pass a filename through the environment, we can add a command-line flag to the sub-makes, we can do whatever the equivalent of inheriting an open file descriptor is on Windows... If we really want to make this reasonably portable (and without that, I cannot see how Paul's dream about less ifdef's is going to materialize), this code needs IMO to be refactored to make the algorithm know less about the implementation details. Personally I've never had any luck trying to create portable code from scratch, at least not unless I'm very familiar with all the platforms which is certainly not the case here. Once we see the second implementation it will be a lot more obvious what needs to be done to make this more portable. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
Eli Zaretskii wrote: Date: Tue, 23 Apr 2013 21:41:22 +0200 From: Frank Heckenbach f.heckenb...@fh-soft.de Sure, it's excluding much more than necessary, but since the critical section is very short, this shouldn't hurt much. (In other words, if make jobs produce such huge output that copying it around takes a significant amount of time, nobody's ever going to read that output anyway, or someone is post-processing it which will take much longer than copying it, anyway.) It doesn't matter whether someone reads the output or not, that output gets written and takes its time. We sometimes invoke tools with options we later regret, because we don't anticipate the results. I was just saying we don't need to optimize for such a strange case. I'm not saying we should fail in such a case, just that it may take a little longer if locking is too coarse. Producing the output by the jobs takes some time anyway, and if the additional delay caused by locking is too much, they just shouldn't use this option. It's an option after all. Indeed, as David wrote, under Unix, locking stdout/stderr is most straightforward because it's available without further setup. Incidentally, this way of locking is also as fine-grained as possible (considering recursive makes). However, as I wrote, this fine-grained locking is not really necessary, so it's not worth worrying about replicating it on Windows if this causes trouble. It is not easy to know what needs to be replicated and what doesn't, because the fine print of the implementation is not clear, at least to me, and therefore it could easily happen that non-replication of some seemingly secondary detail completely breaks the design. I think it was discussed extensively in the original discussion (in which you also participated) and here. If anything is still unclear, feel free to ask. But just to repeat: The only thing really necessary is that two makes with the same stdout/stderr never run sync_output() simultaneously. If *any* two sub-makes never run sync_output() simultaneously, that is also fine. On Tue, Apr 23, 2013 at 10:50 AM, Eli Zaretskii e...@gnu.org wrote: Please tell me that nothing in this feature relies on 'fork', with its copying of handles and other data structures. All it requires is inheriting the redirected stdout/stderr to child processes. This was already possible under Dos (with the exception that since there was no fork, you had to redirect in the parent process, call the child, then undirect in the parent, IIRC). Inheritance is not the problem; _finding_ the inherited object or its handle is. With stdout, its name is global, but not so with a handle to some mutex I can create or its name (unless we indeed make the name fixed and thus system-global, which I don't like, if only because it's not how Make works on Unix). Now we're talking about different things. I was replying to your question what it requires about fork etc. (answer: just stdout/stderr redirection like in the shell). You're worried about how to pass a reference to a mutex around. Paul just replied to that. They can have their own stdout, in particular with the --output-sync=make option. But that's actually the harmless case: Each sub-make runs with its stdout already redirected to a temp file by the main make. In turn, it redirects the stdout of its children to separate temp files, and when they are done, collects the data to its stdout, i.e. the temp file from the main make. When the sub make is finished, the main make collects its output to the original stdout. So unless I'm mistaken, no locking is actually required in this case. But we still acquire the semaphore in this case, don't we? Perhaps we shouldn't. It wouldn't really make things easier, just add another conditional which needs to be tested etc. I still don't know how does Make achieve on Unix the interlocking with its sub-Make's. Is it because the lock is inherited as part of fork? The fd is inherited as part of fork. Each make that needs to takes a lock on the fd. On the same fd, or, rather, on an fd connected to the same resource. That's what I thought (it isn't spelled out anywhere in the comments, and the code that implements that is quite scattered). It's the same fd (namely, stdout or stderr) via inheritance. Of course, you can say it's not the same fd, but two different fd's (i.e., stdout of different processes) referring to the same resource, but that's splitting hairs (and, again, probably irrelevant since you're going to implement it differently anyway). (I wish the design and implementation of this feature were less Posix-centric...) The implementation my be (though it's only two functions, acquire_semaphore(), release_semaphore()) that can be completely replaced (note, again, the fact that the stdout or stderr fd also serves for locking, is just an implementation detail and
Re: [bug #33138] .PARLLELSYNC enhancement with patch
From: Paul Smith p...@mad-scientist.net Cc: Frank Heckenbach f.heckenb...@fh-soft.de, bug-make@gnu.org Date: Tue, 23 Apr 2013 16:54:33 -0400 Without knowing what kind of resource Windows can take locks on, we can't really know how to help with that. The only resources that don't need their names passed to sub-Make are the standard streams. I will see if locking works on console handles; if not, I will have to introduce a command-line argument for passing the name or the handle of a mutex to a sub-Make. we can do whatever the equivalent of inheriting an open file descriptor is on Windows... Doesn't help unless it's a file descriptor of one of the 3 standard streams. Any other descriptor needs to be passed on the command line, for the same reasons we have --job-server-fds option. If we really want to make this reasonably portable (and without that, I cannot see how Paul's dream about less ifdef's is going to materialize), this code needs IMO to be refactored to make the algorithm know less about the implementation details. Personally I've never had any luck trying to create portable code from scratch, at least not unless I'm very familiar with all the platforms which is certainly not the case here. Once we see the second implementation it will be a lot more obvious what needs to be done to make this more portable. That's one way. Another one is to discuss the design more thoroughly before the patches are accepted. I tried to turn the discussion that way when the issue was first brought up, but my comments were largely ignored (if I compare what was suggested then with what was eventually committed), and most of the discussions were about the details of the Unix implementation anyway. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
