Re: Make losing jobserver tokens on Windows
On Mon, 2022-02-21 at 15:56 +0100, Magnus Ihse Bursie wrote: > I'm trying to built it from source right now (and it does not seem > complicated), but even so, an official binary makes sure I'm not > introducing any issues from my local build environment. There is no such thing as an "official binary" for any GNU/FSF software. The GNU project publishes source code, only. Any binary you find is just someone's unofficial build. "gnuwin32.sourceforge.net" is in no way affiliated with GNU or the FSF and doesn't look like it's been maintained in 11+ years, so I would discourage anyone from using it anymore. > > I believe that the goal of the Cygwin port is to implement a POSIX > > layer on top of Windows, so I'm assuming that when you compile GNU > > make with Cygwin it uses the POSIX implementation of the > > jobserver. This implementation relies on certain POSIX behaviors > > of pipes and signals and it's possible that the Cygwin wrappers > > here have an issue that is leading to this problem. > Is there some way I can verify this hypothesis (e.g. debug output > flags), other than looking at the Cygwin source on how they > configure/build make? I have no idea, since we don't support or maintain the Cygwin port of GNU make. You could go ask the Cygwin folks. However, they definitely provide the source code for all the software they make available so you could also go find that and see if there are any patches applied.
Re: Make losing jobserver tokens on Windows
> Date: Mon, 21 Feb 2022 15:56:37 +0100 > From: Magnus Ihse Bursie > > > I am not aware of any issues like this. However, I recommend that you > > try using the Visual Studio build of GNU make and see if that works > > better. I'm not very familiar with the Cygwin version of GNU make; > > it's not maintained by us but my understanding is that they've needed > > to apply some patches (maybe this understanding is outdated though). > I'd definitely like to try to use the official version! Is there some > pre-built binaries for these? I tried to look around but could not find > anything newer than 3.81 at http://gnuwin32.sourceforge.net (and I'm not > even sure if that's "official"). The latest MinGW build of Make can be found here: https://sourceforge.net/projects/ezwinports/files/ But please beware: this is a native Windows port, so if you need to run complex Makefile's that assume a Posix shell, I don't recommend using that port, as its support for a Posix shell is somewhat limited. In any case, you'd need to have a sh.exe somewhere on Path to run recipes that assume a Posix shell, otherwise that port of Make will call cmd.exe as the shell. I'm quite sure Cygwin has its own official build, so maybe you should try that first.
Re: Make losing jobserver tokens on Windows
>> I believe that the goal of the Cygwin port is to implement a POSIX >> layer on top of Windows, so I'm assuming that when you compile GNU >> make with Cygwin it uses the POSIX implementation of the jobserver. >> This implementation relies on certain POSIX behaviors of pipes and >> signals and it's possible that the Cygwin wrappers here have an issue >> that is leading to this problem. Magnus Ihse Bursie (21 February 2022 15:56) asked: > Is there some way I can verify this hypothesis (e.g. debug output > flags), other than looking at the Cygwin source on how they > configure/build make? Conjecture: run GNU make's own testsuite on the Cybwin build of it. Presumably possible by running make test in make's source tree. At least, worth a try ... Eddy.
Re: Make losing jobserver tokens on Windows
On 2022-02-18 16:04, Paul Smith wrote: On Fri, 2022-02-18 at 10:18 +0100, Magnus Ihse Bursie wrote: make[2]: INTERNAL: Exiting with 1 jobserver tokens available; should be 24! This effectively turns the highly parallelized builds into single-threaded builds, and is absolutely detrimental for performance. :-( On the flip side, this also makes for the perfect testing environment to really get to the bottom of this issue. So, my first question is: Is this a known problem? Are there any workarounds/fixes to get around it? Please be sure to tell us which version of GNU make you're using. Right now it's: GNU Make 4.3 Built for x86_64-pc-cygwin but I've seen this issue as far back as I can remember (3.8x). But it is definitely likely to be a Cygwin-specific issue as you say. I'll try to bring this up with the Cygwin folks as well. I am not aware of any issues like this. However, I recommend that you try using the Visual Studio build of GNU make and see if that works better. I'm not very familiar with the Cygwin version of GNU make; it's not maintained by us but my understanding is that they've needed to apply some patches (maybe this understanding is outdated though). I'd definitely like to try to use the official version! Is there some pre-built binaries for these? I tried to look around but could not find anything newer than 3.81 at http://gnuwin32.sourceforge.net (and I'm not even sure if that's "official"). (In fact, prior to this discussion, I was not aware of any other option than Cygwin to run a decently modern version of GNU make on Windows.) I'm trying to built it from source right now (and it does not seem complicated), but even so, an official binary makes sure I'm not introducing any issues from my local build environment. I believe that the goal of the Cygwin port is to implement a POSIX layer on top of Windows, so I'm assuming that when you compile GNU make with Cygwin it uses the POSIX implementation of the jobserver. This implementation relies on certain POSIX behaviors of pipes and signals and it's possible that the Cygwin wrappers here have an issue that is leading to this problem. Is there some way I can verify this hypothesis (e.g. debug output flags), other than looking at the Cygwin source on how they configure/build make? /Magnus The Windows jobserver uses Windows semaphores: it's a simpler implementation than the POSIX one and may be more reliable since it's using a native W32 implementation. Note, all the above is just my personal noodling based on no actual knowledge, as I'm not really a Windows person. Maybe Eli will have some other thoughts. Otherwise: Any suggestions on how to go on and debug this? I am willing to build and test an instrumented debug build of make, but I will need assistance to find my way around the source and spot likely candidates for the source of the problem. Well, the implementation is in the src/w32/s32os.c file in the make source code. But if the problem occurs reliably with a cygwin build and doesn't happen with a Visual Studio build, then my suspicion is that it's an issue with Cygwin rather than GNU make (since we don't see this kind of problem on other POSIX-based systems). Of course it could be a bug in GNU make where we are implicitly assuming some kind of behavior which isn't actually guaranteed by POSIX.
Re: Make losing jobserver tokens on Windows
> From: Paul Smith > Date: Fri, 18 Feb 2022 10:04:08 -0500 > > I am not aware of any issues like this. However, I recommend that you > try using the Visual Studio build of GNU make and see if that works > better. I'm not very familiar with the Cygwin version of GNU make; > it's not maintained by us but my understanding is that they've needed > to apply some patches (maybe this understanding is outdated though). > > I believe that the goal of the Cygwin port is to implement a POSIX > layer on top of Windows, so I'm assuming that when you compile GNU make > with Cygwin it uses the POSIX implementation of the jobserver. This > implementation relies on certain POSIX behaviors of pipes and signals > and it's possible that the Cygwin wrappers here have an issue that is > leading to this problem. > > The Windows jobserver uses Windows semaphores: it's a simpler > implementation than the POSIX one and may be more reliable since it's > using a native W32 implementation. > > Note, all the above is just my personal noodling based on no actual > knowledge, as I'm not really a Windows person. > > Maybe Eli will have some other thoughts. Oops, I missed the fact that this was about the Cygwin port of Make. If so, the pointer to the sources I gave was the wrong one, sorry. Yes, Cygwin is supposed to emulate a Posix system (via its special DLL). For a Cygwin build of Make, I suggest to post a question to the Cygwin mailing list, with the details you reported here. They might be more familiar with these issues than we are here. It could be some subtle issue with the way they emulate pipes and/or signals.
Re: Make losing jobserver tokens on Windows
On Fri, 2022-02-18 at 10:18 +0100, Magnus Ihse Bursie wrote: > make[2]: INTERNAL: Exiting with 1 jobserver tokens available; should > be 24! > > This effectively turns the highly parallelized builds into > single-threaded builds, and is absolutely detrimental for > performance. :-( On the flip side, this also makes for the perfect > testing environment to really get to the bottom of this issue. > > So, my first question is: Is this a known problem? Are there any > workarounds/fixes to get around it? Please be sure to tell us which version of GNU make you're using. I am not aware of any issues like this. However, I recommend that you try using the Visual Studio build of GNU make and see if that works better. I'm not very familiar with the Cygwin version of GNU make; it's not maintained by us but my understanding is that they've needed to apply some patches (maybe this understanding is outdated though). I believe that the goal of the Cygwin port is to implement a POSIX layer on top of Windows, so I'm assuming that when you compile GNU make with Cygwin it uses the POSIX implementation of the jobserver. This implementation relies on certain POSIX behaviors of pipes and signals and it's possible that the Cygwin wrappers here have an issue that is leading to this problem. The Windows jobserver uses Windows semaphores: it's a simpler implementation than the POSIX one and may be more reliable since it's using a native W32 implementation. Note, all the above is just my personal noodling based on no actual knowledge, as I'm not really a Windows person. Maybe Eli will have some other thoughts. > Otherwise: Any suggestions on how to go on and debug this? I am > willing to build and test an instrumented debug build of make, but I > will need assistance to find my way around the source and spot likely > candidates for the source of the problem. Well, the implementation is in the src/w32/s32os.c file in the make source code. But if the problem occurs reliably with a cygwin build and doesn't happen with a Visual Studio build, then my suspicion is that it's an issue with Cygwin rather than GNU make (since we don't see this kind of problem on other POSIX-based systems). Of course it could be a bug in GNU make where we are implicitly assuming some kind of behavior which isn't actually guaranteed by POSIX.
Re: Make losing jobserver tokens on Windows
> Date: Fri, 18 Feb 2022 10:18:24 +0100 > From: Magnus Ihse Bursie > > So, my first question is: Is this a known problem? Are there any > workarounds/fixes to get around it? Not that I know of, no. Search of the bug database also doesn't seem to bring up anything like that. > Otherwise: Any suggestions on how to go on and debug this? I am willing > to build and test an instrumented debug build of make, but I will need > assistance to find my way around the source and spot likely candidates > for the source of the problem. The MS-Windows implementation of the jobserver is on src/w32/w32os.c, and those functions are called from job.c and from main.c. In particular, the error message you quote comes from main.c:clean_jobserver. If you need further assistance, please ask more specific questions.
Make losing jobserver tokens on Windows
Hi, I'm working for Oracle on the OpenJDK build team. We're using GNU make exclusively to build the JDK on all supported platforms. For Windows, we (typically) use the Cygwin version. We have had a long-standing issue with make losing jobserver tokens on Windows. Most runs end with something like: make[2]: INTERNAL: Exiting with 11 jobserver tokens available; should be 12! Since the build still succeeds, and it just affects performance (and typically not that much), we have not spend too much time getting to the bottom of this. Now, however, I've come across a machine where this happens repeatedly, and on a much worse scale: make[2]: INTERNAL: Exiting with 1 jobserver tokens available; should be 24! This effectively turns the highly parallelized builds into single-threaded builds, and is absolutely detrimental for performance. :-( On the flip side, this also makes for the perfect testing environment to really get to the bottom of this issue. So, my first question is: Is this a known problem? Are there any workarounds/fixes to get around it? Otherwise: Any suggestions on how to go on and debug this? I am willing to build and test an instrumented debug build of make, but I will need assistance to find my way around the source and spot likely candidates for the source of the problem. /Magnus
