Re: Parallel job instance identifiers?
Wait, it is a dumb question. Never mind. David On Mon, Sep 6, 2021 at 7:58 AM David Boyce wrote: > > One possibility would be for make to write out bytes with values > 0,1,2,3... then when it gets to 255 start over again ... > > I hope this sin't a dumb question, I've never worked with the job server, > but hy not use the value of the file descriptor from the parent side as the > token to send down the pipe? Still limited to 255 of course, same wrapping > problem, but the fd is guaranteed unique and I could imagine cases where > that data might come in handy for debugging via strace or whatever. > > David > > On Mon, Sep 6, 2021 at 7:37 AM Paul Smith wrote: > >> On Fri, 2021-08-13 at 14:14 +0100, Howard Chu via Bug reports and >> discussion for GNU make wrote: >> > I'm looking for a way to expose a job ID number to the individual >> > jobs, in a consecutive range from 1-[number of jobs]. Not just >> > unique numbers, for that we could just use $$ already. The purpose >> > is to e.g. assign non-overlapping network port numbers when firing >> > off a number of client/server test scripts in parallel. >> > >> > Is there anything that could do this now, exposed as a Make variable? >> >> I don't see any way make can provide something like this, at least not >> using the model it does today. The current requirement is that each >> read from the pipe is of exactly 1 byte and there's no way to guarantee >> a unique value in a single byte of data if there are >256 possible >> values of course. >> >> One possibility would be for make to write out bytes with values >> 0,1,2,3... then when it gets to 255 start over again: this would give >> unique values for any -j <= 256 and you'd only have to worry about >> wrapping for very large values of -j. >> >> One reason I'm not excited about this idea is that I've been seriously >> contemplating switching from the current pipe-based jobserver control >> method, to using POSIX named semaphores instead (at least as the >> default). There is support for these in most systems these days, and >> it's already how Windows works (with the Windows equivalent). The >> advantage here is it's just easier in lots of ways to not have to pass >> open file descriptors between make invocations. I could use a named >> pipe but semaphores are more obviously built for this. But of course >> there's no possibility of uniquely identifying a semaphore lock. >> >> The other downside of using a semaphore is I was thinking of >> implementing a "fast fail" feature where as soon as any make in the >> jobserver domain died all the other makes would stop quickly as well. >> This is something people seem to really want. One way to implement >> this is to write "error bytes" into the jobserver pipe when we've hit >> "failure mode"; whenever a make instance reads the "error byte" instead >> of the "normal byte" it would finish any outstanding jobs then stop. >> >> This is pretty straightforward although it's not ideal: make can >> potentially do a lot of work with its single token without trying to >> read from the jobserver for example. >> >> Of course a single semaphore can't help here. If we go with "named X" >> we'll need a directory to hold it; one idea would be to write a >> "failed" file into that directory that all make's could check for. >> >> >>
Re: Parallel job instance identifiers?
> One possibility would be for make to write out bytes with values 0,1,2,3... then when it gets to 255 start over again ... I hope this sin't a dumb question, I've never worked with the job server, but hy not use the value of the file descriptor from the parent side as the token to send down the pipe? Still limited to 255 of course, same wrapping problem, but the fd is guaranteed unique and I could imagine cases where that data might come in handy for debugging via strace or whatever. David On Mon, Sep 6, 2021 at 7:37 AM Paul Smith wrote: > On Fri, 2021-08-13 at 14:14 +0100, Howard Chu via Bug reports and > discussion for GNU make wrote: > > I'm looking for a way to expose a job ID number to the individual > > jobs, in a consecutive range from 1-[number of jobs]. Not just > > unique numbers, for that we could just use $$ already. The purpose > > is to e.g. assign non-overlapping network port numbers when firing > > off a number of client/server test scripts in parallel. > > > > Is there anything that could do this now, exposed as a Make variable? > > I don't see any way make can provide something like this, at least not > using the model it does today. The current requirement is that each > read from the pipe is of exactly 1 byte and there's no way to guarantee > a unique value in a single byte of data if there are >256 possible > values of course. > > One possibility would be for make to write out bytes with values > 0,1,2,3... then when it gets to 255 start over again: this would give > unique values for any -j <= 256 and you'd only have to worry about > wrapping for very large values of -j. > > One reason I'm not excited about this idea is that I've been seriously > contemplating switching from the current pipe-based jobserver control > method, to using POSIX named semaphores instead (at least as the > default). There is support for these in most systems these days, and > it's already how Windows works (with the Windows equivalent). The > advantage here is it's just easier in lots of ways to not have to pass > open file descriptors between make invocations. I could use a named > pipe but semaphores are more obviously built for this. But of course > there's no possibility of uniquely identifying a semaphore lock. > > The other downside of using a semaphore is I was thinking of > implementing a "fast fail" feature where as soon as any make in the > jobserver domain died all the other makes would stop quickly as well. > This is something people seem to really want. One way to implement > this is to write "error bytes" into the jobserver pipe when we've hit > "failure mode"; whenever a make instance reads the "error byte" instead > of the "normal byte" it would finish any outstanding jobs then stop. > > This is pretty straightforward although it's not ideal: make can > potentially do a lot of work with its single token without trying to > read from the jobserver for example. > > Of course a single semaphore can't help here. If we go with "named X" > we'll need a directory to hold it; one idea would be to write a > "failed" file into that directory that all make's could check for. > > >
Re: Parallel job instance identifiers?
Paul Smith wrote: > On Fri, 2021-08-13 at 14:14 +0100, Howard Chu via Bug reports and > discussion for GNU make wrote: >> I'm looking for a way to expose a job ID number to the individual >> jobs, in a consecutive range from 1-[number of jobs]. Not just >> unique numbers, for that we could just use $$ already. The purpose >> is to e.g. assign non-overlapping network port numbers when firing >> off a number of client/server test scripts in parallel. >> >> Is there anything that could do this now, exposed as a Make variable? > > I don't see any way make can provide something like this, at least not > using the model it does today. The current requirement is that each > read from the pipe is of exactly 1 byte and there's no way to guarantee > a unique value in a single byte of data if there are >256 possible > values of course. > > One possibility would be for make to write out bytes with values > 0,1,2,3... then when it gets to 255 start over again: this would give > unique values for any -j <= 256 and you'd only have to worry about > wrapping for very large values of -j. We could change the tokens to be 2 bytes. Individual read/write syscalls are still guaranteed to be atomic, so there's no danger of interleaving. The only problem then is ensuring a large enough pipe buffer to avoid premature blocking. > The other downside of using a semaphore is I was thinking of > implementing a "fast fail" feature where as soon as any make in the > jobserver domain died all the other makes would stop quickly as well. > This is something people seem to really want. One way to implement > this is to write "error bytes" into the jobserver pipe when we've hit > "failure mode"; whenever a make instance reads the "error byte" instead > of the "normal byte" it would finish any outstanding jobs then stop. It'd be easier just to close the write-side of the pipe, and then all subsequent read attempts would fail. > This is pretty straightforward although it's not ideal: make can > potentially do a lot of work with its single token without trying to > read from the jobserver for example. > > Of course a single semaphore can't help here. If we go with "named X" > we'll need a directory to hold it; one idea would be to write a > "failed" file into that directory that all make's could check for. Yes, this is another advantage of using pipes, no special considerations about namespaces, filesystem or otherwise. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: Parallel job instance identifiers?
On Fri, 2021-08-13 at 14:14 +0100, Howard Chu via Bug reports and discussion for GNU make wrote: > I'm looking for a way to expose a job ID number to the individual > jobs, in a consecutive range from 1-[number of jobs]. Not just > unique numbers, for that we could just use $$ already. The purpose > is to e.g. assign non-overlapping network port numbers when firing > off a number of client/server test scripts in parallel. > > Is there anything that could do this now, exposed as a Make variable? I don't see any way make can provide something like this, at least not using the model it does today. The current requirement is that each read from the pipe is of exactly 1 byte and there's no way to guarantee a unique value in a single byte of data if there are >256 possible values of course. One possibility would be for make to write out bytes with values 0,1,2,3... then when it gets to 255 start over again: this would give unique values for any -j <= 256 and you'd only have to worry about wrapping for very large values of -j. One reason I'm not excited about this idea is that I've been seriously contemplating switching from the current pipe-based jobserver control method, to using POSIX named semaphores instead (at least as the default). There is support for these in most systems these days, and it's already how Windows works (with the Windows equivalent). The advantage here is it's just easier in lots of ways to not have to pass open file descriptors between make invocations. I could use a named pipe but semaphores are more obviously built for this. But of course there's no possibility of uniquely identifying a semaphore lock. The other downside of using a semaphore is I was thinking of implementing a "fast fail" feature where as soon as any make in the jobserver domain died all the other makes would stop quickly as well. This is something people seem to really want. One way to implement this is to write "error bytes" into the jobserver pipe when we've hit "failure mode"; whenever a make instance reads the "error byte" instead of the "normal byte" it would finish any outstanding jobs then stop. This is pretty straightforward although it's not ideal: make can potentially do a lot of work with its single token without trying to read from the jobserver for example. Of course a single semaphore can't help here. If we go with "named X" we'll need a directory to hold it; one idea would be to write a "failed" file into that directory that all make's could check for.
Re: Parallel job instance identifiers?
Hi, I'm afraid there's a lot to consider. > That would require starting up a server first, and somehow getting it to report its port# to us. > But the server configs must know port#s in advance because they need to talk to each other. Bootstrapping from leaf servers up the dependency tree should be possible. Cyclic dependencies could get in the way. Cyclic dependencies usually are a bad idea. But that's off-topic. > Currently we use the range of 9000-9020. On a build/test machine we don't worry about conflicts, > a developer has no business running conflicting software at the same time as building/testing > our software. And of course, the baseport 9000 can be moved at will. Well, well. I don't subscribe to the idea that anyone can dictate what a developer would run at the same time and what constitutes such software. IDEs might use ports. The system itself uses ports. Other software might use ports. The developer might want to build/test two version of the software in parallel at the same time to do comparative analysis and comparative debugging. The definition of the port range for ephemeral ports could change. There's just so much that could go wrong. In my experience, assuming the availability of ports leads to flaky tests. Developers can lose a lot of time to get tests running if they first have to resolve port conflicts. I get that you want to use job identifiers for this. But the mechanism is just not made for that. That the job identifier is a number is only by coincidence. Job identifiers are identifiers first, and numbers only coincidentally. If someone decides, for whatever reason, to replace them with UUIDs, so what? I'm not saying that using UUIDs would be a good idea in this case, it's just about making the point about the difference between an identifier and a number. But I totally get that you need something that helps you to get unique numbers. Have you considered using a simple shell script that counts up and writes the count into a named pipe, and then having each job read one line from that named pipe to get "your" number? This should do the job: $ mkfifo counter $ for ((uniqueNumber = 0; uniqueNumber < 1000; uniqueNumber++)) ; do echo $uniqueNumber >counter ; done & You can then get a - for that run - unique number using $ head -n 1 counter It should be easy to do this from a Makefile. If you do this directly from the Makefile, this might require SHELL:=/bin/bash, recursive make, and an exit handler. Here is how you could use an exit handler from within a Makefile: https://stackoverflow.com/questions/28597794/how-can-i-clean-up-after-an-error-in-a-makefile/52159940#52159940 I hope that this helps you and provides you with a pathway solution that allows you even more control without accidental coupling to internal features of make. And while I think that using make is great, this solution would even work if you use something else than make. Q'aplah! On Sat, Aug 14, 2021 at 5:57 PM Howard Chu wrote: > Christian Hujer wrote: > > Hi, > > > > Have you considered using port 0 instead to tap into the ephemeral port > range and communicate that port somehow? > > That would require starting up a server first, and somehow getting it to > report its port# to us. > But the server configs must know port#s in advance because they need to > talk to each other. > > > > From what I understand, you want to use the job id as an offset to a > base port to choose a port from a range, right? That sounds non-portable to > me, spelling > > all sorts of ports conflict trouble. > > Currently we use the range of 9000-9020. On a build/test machine we don't > worry about conflicts, > a developer has no business running conflicting software at the same time > as building/testing > our software. And of course, the baseport 9000 can be moved at will. > > > > Christian > > > > On Fri, Aug 13, 2021, 18:44 Howard Chu via Bug reports and discussion > for GNU make mailto:bug-make@gnu.org>> wrote: > > > > In my original jobserver implementation, I filled the job pipe with > > up to 256 bytes, numbers from 0-255. The idea being that then make > > could distinguish each job with a unique ID number. That idea got > > thrown away when we decided to raise the limit to PIPEBUF (4096 on > > traditional Unix, much larger now on Linux). > > > > I'm looking for a way to expose a job ID number to the individual > > jobs, in a consecutive range from 1-[number of jobs]. Not just unique > > numbers, for that we could just use $$ already. The purpose is to > > e.g. assign non-overlapping network port numbers when firing off a > > number of client/server test scripts in parallel. > > > > Is there anything that could do this now, exposed as a Make variable? > > -- > > -- Howard Chu > > CTO, Symas Corp. http://www.symas.com > > Director, Highland Sun http://highlandsun.com/hyc/ > > Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: Parallel job instance identifiers?
Christian Hujer wrote: > Hi, > > Have you considered using port 0 instead to tap into the ephemeral port range > and communicate that port somehow? That would require starting up a server first, and somehow getting it to report its port# to us. But the server configs must know port#s in advance because they need to talk to each other. > > From what I understand, you want to use the job id as an offset to a base > port to choose a port from a range, right? That sounds non-portable to me, > spelling > all sorts of ports conflict trouble. Currently we use the range of 9000-9020. On a build/test machine we don't worry about conflicts, a developer has no business running conflicting software at the same time as building/testing our software. And of course, the baseport 9000 can be moved at will. > > Christian > > On Fri, Aug 13, 2021, 18:44 Howard Chu via Bug reports and discussion for GNU > make mailto:bug-make@gnu.org>> wrote: > > In my original jobserver implementation, I filled the job pipe with > up to 256 bytes, numbers from 0-255. The idea being that then make > could distinguish each job with a unique ID number. That idea got > thrown away when we decided to raise the limit to PIPEBUF (4096 on > traditional Unix, much larger now on Linux). > > I'm looking for a way to expose a job ID number to the individual > jobs, in a consecutive range from 1-[number of jobs]. Not just unique > numbers, for that we could just use $$ already. The purpose is to > e.g. assign non-overlapping network port numbers when firing off a > number of client/server test scripts in parallel. > > Is there anything that could do this now, exposed as a Make variable? > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: Parallel job instance identifiers?
Hi, Have you considered using port 0 instead to tap into the ephemeral port range and communicate that port somehow? >From what I understand, you want to use the job id as an offset to a base port to choose a port from a range, right? That sounds non-portable to me, spelling all sorts of ports conflict trouble. Christian On Fri, Aug 13, 2021, 18:44 Howard Chu via Bug reports and discussion for GNU make wrote: > In my original jobserver implementation, I filled the job pipe with > up to 256 bytes, numbers from 0-255. The idea being that then make > could distinguish each job with a unique ID number. That idea got > thrown away when we decided to raise the limit to PIPEBUF (4096 on > traditional Unix, much larger now on Linux). > > I'm looking for a way to expose a job ID number to the individual > jobs, in a consecutive range from 1-[number of jobs]. Not just unique > numbers, for that we could just use $$ already. The purpose is to > e.g. assign non-overlapping network port numbers when firing off a > number of client/server test scripts in parallel. > > Is there anything that could do this now, exposed as a Make variable? > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > >
Parallel job instance identifiers?
In my original jobserver implementation, I filled the job pipe with up to 256 bytes, numbers from 0-255. The idea being that then make could distinguish each job with a unique ID number. That idea got thrown away when we decided to raise the limit to PIPEBUF (4096 on traditional Unix, much larger now on Linux). I'm looking for a way to expose a job ID number to the individual jobs, in a consecutive range from 1-[number of jobs]. Not just unique numbers, for that we could just use $$ already. The purpose is to e.g. assign non-overlapping network port numbers when firing off a number of client/server test scripts in parallel. Is there anything that could do this now, exposed as a Make variable? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
