Re: [PATCH] src/main.c: Add -J to detect number of job slots based on nproc.
DSB wrote: > Isn't nproc or nproc+1 too much? On systems with hyper-threading, > this will try using too many jobs, and might cause the system be > significantly less responsive. Maybe nproc/2 is a better default? > > > This is an interesting question and I suppose the answer depends on many > factors. At my organization we've actually gone the other direction and > default to -j > (nproc * 2) on the theory that one of the two jobs is liable to be sleeping > on I/O at any given time. But this is very much a YMMV area and the builtin > default, > if there was to be one at all, should be conservative so nproc * 2 would be a > bad choice. I've always used nproc * 1.5. This factor was determined empirically by observing idle time on many builds using nproc * 1 and nproc * 2. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: Add support for limiting CPU pressure
cont...@zhengqiu.net wrote: > Hello, > > While we are doing development with Yocto Project (1), a source- > based Linux distro builder, we found that many instances of making > running in parallel with other work can overload even a large > many-core build machine. Existing mechanisms that are either not > system-wide (-j) or are too slow(-l has 1 minute averaging), so in > order to make optimal use of a multi-core system for some larger > tasks we need a new mechanism. > All the tests are done using an Ubuntu System, as we are not very lucky to > find similar features on macOS/Windows, so if you know anything, please > help us out! Additionally, we also want to gather more data, so if you know > any other large packages that make is often tested with, please let us know > as well! Relying on a new non-portable kernel feature sounds like a pretty bad idea. Especially when you can easily solve the "not system-wide" aspect of "make -j" portably: by using a named pipe. It should be a simple job to patch make to create or reference a named pipe in an initial invocation, and the rest of the existing jobserver machinery will Just Work after that, and all of the relevant OSs have support for named pipes. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: Goodbye to GNU make's "build.sh" ... ?
Herring, Daniel - 0447 - MITLL wrote: > On Jun 26, 2022, at 3:30 PM, Eli Zaretskii wrote: > > How do people bootstrap systems nowadays? Is it only via > cross-compilation? > > > Another way to bootstrap is to have a native tool chain that is less > capable…. For example, an embedded device may come with an old or proprietary > shell and compiler. But it would still include something resembling "make" - embedded systems toolkits have shipped with clones of make since the 1980s. I'm not convinced that there are any notable systems out there capable of supporting gmake that don't already have a make utility. Or that telling anyone so unlucky "first bootstrap with gmake 4.3" is a bad answer. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: Parallel job instance identifiers?
Paul Smith wrote: > On Fri, 2021-08-13 at 14:14 +0100, Howard Chu via Bug reports and > discussion for GNU make wrote: >> I'm looking for a way to expose a job ID number to the individual >> jobs, in a consecutive range from 1-[number of jobs]. Not just >> unique numbers, for that we could just use $$ already. The purpose >> is to e.g. assign non-overlapping network port numbers when firing >> off a number of client/server test scripts in parallel. >> >> Is there anything that could do this now, exposed as a Make variable? > > I don't see any way make can provide something like this, at least not > using the model it does today. The current requirement is that each > read from the pipe is of exactly 1 byte and there's no way to guarantee > a unique value in a single byte of data if there are >256 possible > values of course. > > One possibility would be for make to write out bytes with values > 0,1,2,3... then when it gets to 255 start over again: this would give > unique values for any -j <= 256 and you'd only have to worry about > wrapping for very large values of -j. We could change the tokens to be 2 bytes. Individual read/write syscalls are still guaranteed to be atomic, so there's no danger of interleaving. The only problem then is ensuring a large enough pipe buffer to avoid premature blocking. > The other downside of using a semaphore is I was thinking of > implementing a "fast fail" feature where as soon as any make in the > jobserver domain died all the other makes would stop quickly as well. > This is something people seem to really want. One way to implement > this is to write "error bytes" into the jobserver pipe when we've hit > "failure mode"; whenever a make instance reads the "error byte" instead > of the "normal byte" it would finish any outstanding jobs then stop. It'd be easier just to close the write-side of the pipe, and then all subsequent read attempts would fail. > This is pretty straightforward although it's not ideal: make can > potentially do a lot of work with its single token without trying to > read from the jobserver for example. > > Of course a single semaphore can't help here. If we go with "named X" > we'll need a directory to hold it; one idea would be to write a > "failed" file into that directory that all make's could check for. Yes, this is another advantage of using pipes, no special considerations about namespaces, filesystem or otherwise. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: Parallel job instance identifiers?
Christian Hujer wrote: > Hi, > > Have you considered using port 0 instead to tap into the ephemeral port range > and communicate that port somehow? That would require starting up a server first, and somehow getting it to report its port# to us. But the server configs must know port#s in advance because they need to talk to each other. > > From what I understand, you want to use the job id as an offset to a base > port to choose a port from a range, right? That sounds non-portable to me, > spelling > all sorts of ports conflict trouble. Currently we use the range of 9000-9020. On a build/test machine we don't worry about conflicts, a developer has no business running conflicting software at the same time as building/testing our software. And of course, the baseport 9000 can be moved at will. > > Christian > > On Fri, Aug 13, 2021, 18:44 Howard Chu via Bug reports and discussion for GNU > make mailto:bug-make@gnu.org>> wrote: > > In my original jobserver implementation, I filled the job pipe with > up to 256 bytes, numbers from 0-255. The idea being that then make > could distinguish each job with a unique ID number. That idea got > thrown away when we decided to raise the limit to PIPEBUF (4096 on > traditional Unix, much larger now on Linux). > > I'm looking for a way to expose a job ID number to the individual > jobs, in a consecutive range from 1-[number of jobs]. Not just unique > numbers, for that we could just use $$ already. The purpose is to > e.g. assign non-overlapping network port numbers when firing off a > number of client/server test scripts in parallel. > > Is there anything that could do this now, exposed as a Make variable? > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Parallel job instance identifiers?
In my original jobserver implementation, I filled the job pipe with up to 256 bytes, numbers from 0-255. The idea being that then make could distinguish each job with a unique ID number. That idea got thrown away when we decided to raise the limit to PIPEBUF (4096 on traditional Unix, much larger now on Linux). I'm looking for a way to expose a job ID number to the individual jobs, in a consecutive range from 1-[number of jobs]. Not just unique numbers, for that we could just use $$ already. The purpose is to e.g. assign non-overlapping network port numbers when firing off a number of client/server test scripts in parallel. Is there anything that could do this now, exposed as a Make variable? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]
Bruno Haible wrote: > Continuing this thread from May 2019 > <https://lists.gnu.org/archive/html/bug-make/2019-05/msg00022.html>: > The problem was: > > How can a rule that generates multiple files be formulated so > that it works with parallel make? > > For example, a rule that invokes bison, or a rule that invokes > a different Makefile. For simplicity, here, use a rule that > creates 4 files copy1, copy2, copy3, copy4. > > === > all : copy1 copy2 copy3 copy4 > > copy1 copy2 copy3 copy4: Makefile > install -c -m 644 Makefile copy1 > install -c -m 644 Makefile copy2 > install -c -m 644 Makefile copy3 > install -c -m 644 Makefile copy4 > === > > Unfortunately, with "make -j8", it invokes the rule multiple times. > > It is possible to change this Makefile so that > (A) "rm -f copy?; make" executes the rule once. > (B) "rm -f copy?; make -j8" executes the rule once as well. > (C) After "make", another "make" just prints "Nothing to be done for 'all'." > (D) After removing one of the files copy?, "make" executes the rule once. > (This covers also the case of pressing Ctrl-C during "make", then > doing "make" again.) > (E) After replacing one of the files copy? with a file that is older than > Makefile, "make" executes the rule once. > > There are three possibilities: You're thinking about this the wrong way. Your set of commands is inherently serial, therefore you need to write serial dependencies. === all: copy1 copy2 copy3 copy4 copy1: Makefile install -c -m 644 Makefile $@ copy2: copy1 install -c -m 644 Makefile $@ copy3: copy2 install -c -m 644 Makefile $@ copy4: copy3 install -c -m 644 Makefile $@ ======= This satisfies all your conditions, because it is inherently correct. More on writing proper parallel Makefiles here http://highlandsun.com/hyc/#Make -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: [bug #56701] Do not allow -j without a number
Personally I would have closed this as Not a Bug since it has been clearly documented for 30+ years. Users are responsible for reading the docs and learning how their tools work. Dmitry V. Levin wrote: > Follow-up Comment #2, bug #56701 (project make): > > Let's try to fix this issue and avoid introducing any regressions. > > For example, if -j is given without an argument and no -l option is provided, > behave as if -j -l $(nproc) was specified. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Should make handle non-blocking jobserver and use socket pair instead?
Juha-Matti Tilli wrote: > Hello, > > (Please CC me, I’m not on the list) The jobserver was designed using a pipe because that is the least common denominator for IPC across POSIX systems - it worked across all flavors of Unix in existence in 1991, and continues to work unchanged to this day with no special maintenance requirements. One you start deviating from least common denominator, support and maintenance demands increase. > I’m creating a new build system that is supposed to solve many of the > issues of standard make. Most programmers who create alternative build > systems (SCons, Rake, Shake) do so by selecting an alternative > programming language (Python, Ruby, Haskell), whereas I’m doing it in > C. Most programmers also create the system by incompatibility and > making it wildly different from make, whereas I’m keeping most of the > features in make that are actually good. That includes the jobserver. > > My build system can fork recursive sub-makes, and has support for the > jobserver. While creating the support for jobserver, I noticed a > problem in GNU make handling of jobserver. > > GNU make expects the jobserver to be blocking, whereas a program based > on an event-loop architecture should really be using non-blocking file > descriptors. > > If the jobserver is set to be non-blocking (I’m creating the jobserver > in my build system and passing it to GNU make), GNU make gives this > error: > > make: *** read jobs pipe: Resource temporarily unavailable. Stop. > make: *** Waiting for unfinished jobs > > I’m wondering whether this error exit() is the right thing to do. In > the programming world, there is a lot of variability and different > choices. Some prefer C, some C++, some Rust. Similarly, some may > prefer GNU make and some may prefer alternative build systems. In my > opinion, GNU make, including parallel make, should attempt to be as > compatible as possible with all other build systems, because not every > programmer wants to use blocking file descriptors. The jobserver is > created by the top-level build system always, and it may be something > else than GNU make. Thus, I'm proposing a "be liberal in what you > accept; be conservative in what you do" approach. Now GNU make doesn't > accept a non-blocking jobserver. > > Due to the reasons given in https://cr.yp.to/unix/nonblock.html I > cannot turn on O_NONBLOCK even after dup() because the flag hits the > entire ofile, not just the fd, and because GNU make does not work with > a non-blocking jobserver. I see two solutions: > > 1. Create a socket pair instead of a pipe so that I can use > MSG_DONTWAIT on a blocking socket > > 2. Use a select/poll + read pair, and immediately before the read, > turn on a 10 millisecond interval timer just in case some sub-make won > the race to acquire the token > > Both of these solutions seem to work, and for compatibility with old > GNU make variants, I really have to use either solution for the next > 10 years or so. In case the jobserver was created by make and not my > tool, the MSG_DONTWAIT doesn’t work, so in that case I have to revert > to the second solution. My current approach is to try MSG_DONTWAIT > first and in case of ENOTSOCK use the interval timer with read(). > > However, I think the issue should be fixed in GNU make too. > Specifically, I think GNU make should read(), and if it gives EAGAIN, > do a select()/poll() and read() again, iterating as long as read() > returns EAGAIN. Of course, some systems may not have a functioning > select()/poll(), so the special EAGAIN handling would apply only to > those systems that fully support select()/poll(). > > In fact, I argue that GNU make should too create a SOCK_STREAM socket > pair instead of a pipe, if the socket pair facility is available. > Socket pairs are superior in many aspects, including the possibility > to use MSG_DONTWAIT. I have tested that GNU make works perfectly with > a jobserver that is actually a socket pair instead of a pipe, but then > that jobserver needs to be created externally, as the GNU make > creating the jobserver creates a pipe and not a socket pair. As a > hack, a wrapper program could do it and then call GNU make. > > What are your opinions of this? I can prepare a patch / patches for > GNU make if it would be acceptable. Of course I won't be wasting any > time reading GNU make sources if the improvement proposal is rejected. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: "make -jN" requires mechanical changes to a Makefile
Bruno Haible wrote: > Henrik Carlqvist wrote: >> Example with one rule creating 4 files: >> >> all : copy1 >> >> copy1: Makefile >> install -c -m 644 Makefile copy1 >> install -c -m 644 Makefile copy2 >> install -c -m 644 Makefile copy3 >> install -c -m 644 Makefile copy4 > > I think the "representative" file should be copy4 here, because it's the one > that gets created last. That sort of thing is only true in serial make, you can't rely on it in parallel make. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Parallel Build with GNU make
Martin Dorey wrote: -Original Message- From: bug-make-bounces+martin.dorey=hds@gnu.org [mailto:bug-make-bounces+martin.dorey=hds@gnu.org] On Behalf Of Roland Schwingel Sent: Monday, February 01, 2016 23:40 To: Eli Zaretskii; bug-make@gnu.org Subject: Re: Parallel Build with GNU make Sorry for following up on my own post,but there has been a "bug" in the last sentence... See for correction below Am 02.02.2016 um 08:35 schrieb Roland Schwingel: Hi... Eli Zaretskii wrote on 01.02.2016 20:14:17: > Add "--debug=j" to the make command-line switches and see if it > launches more than one command at a time. > > Anyway, your makefile.mak file can be written in such a way as to > effectively prohibit parallelism, due to how it lays out dependencies. This is something I am struggeling on. I tried this a couple of times but could not make it working in a way that is satisfying for me, as I could not find the correct way to control parallelism. I am aware of .NOTPARALLEL but this might not be enough for me. I have this examplarily layout: subfolder1 - subfolder4 - sub4_1.c sub1_1.c sub4_2.c sub1_2.c ... subfolder2 - sub2_1.c sub2_2.c subfolder3 - sub3_1.c sub3_2.c main1.c main2.c main3.c All subfolders shall be visited recursively first. No parallelism is allowed in visiting folders. But when in a folder and it comes to concrete compiling parallelism should be allowed. This should result in this compileorder for the above example Visit subfolder1 (not in parallel) Visit subfolder4 (not in parallel) Compile sub4_1.c and sub4_2.c in parallel Return to subfolder1 Compile sub1_2.c and sub1_2.c in parallel Return to main Visit subfolder2 (not in parallel) Compile sub2_1.c and sub2_2.c in parallel Return to main Vist subfolder3 (not in parallel) Compile sub3_1.c and sub3_2.c in parallel Return to main Compile main1.c,main2.c and main3.c in parallel When doing a make -j 4 for instance make is visiting all subfolders in parallel which is not desired. When I am using .NOTPARALLEL the whole subfolder is not compiled in parallel which is also not desired. Each folder visit is a concrete submake in my case. Each folder has its own makefile Any clue on how to achieve the flow I want to have? Visiting folders non recursive while compiling itself inside of the folders is recursive? Should read: Visiting folders non PARALLEL while compiling itself inside of the folders is IN PARALLEL. Use a for-loop for anything that must be explicitly serial. Generally, to me this is a sign of a broken Makefiles - when you define your dependencies correctly, everything should naturally run in the correct order, regardless of degree of parallelism. The fact that you want subfolder 2 to be built after subfolder 1 means sub1 *is* a dependency of sub2 and if you haven't stated that in your Makefile then you've made an error in your Makefile. Read this: http://highlandsun.com/hyc/#Make in particular this http://highlandsun.com/hyc/GNUYou.htm -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Dynamic adjustments of build dependencies for the number of available processors
SF Markus Elfring wrote: It seems like you want to automatically expand or contract the number of jobs that make will run in parallel, Yes, exactly. I can not use the parameter -j directly for the start of the main make process in my case so far. based on some processing of the recipe. One of the variables (or make rules) will call the command getconf _NPROCESSORS_ONLN. The corresponding processes should be efficiently started for parallel background execution after a serial data preparation was finished. This makes very little sense. In a properly written Makefile, your serial preparation step will remain serial no matter what -j arguments are used, so it should be safe to just invoke make -j xx at the beginning and let things run serially or in parallel as necessary. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request: parallel builds feature
Jim Michaels wrote: I wasn't digressing. I was explaining the point. the concept I am trying to present as a solution to the problem of making parallel stdin for --jobs in gnu make (which currenty doesn't work and is I guess single-threaded) is to make a separate terminal or command shell for each job, such as via a generated batch file or shell script. this is as simple as I can make it. Who said stdin was a problem? Fundamentally the jobs spawned by make are batch jobs - they should not be requesting input from stdin in the first place. Semantically the effect of running parallel make must be identical to running serial make. You cannot guarantee this to be true if jobs are reading from stdin because stdin's supply of data is inherently serial but the order it gets read is non-deterministic in a parallel build. If the jobs you're spawning from make require input from stdin you need to rewrite those jobs. at the end of the shell script, you can put in whatever you like, such as synchronization stuff saying I am done by creating a semaphore file, a flag file. the problem would then be porting BASH to windows and other platforms in order to handle --jobs. bash has already been ported to Windows. I have learned that on a machine with 12 threads and 64GB of memory, you can have 50+ jobs running. -- *From:* Paul Smith psm...@gnu.org *To:* Jim Michaels jmich...@yahoo.com *Cc:* bug-make@gnu.org *Sent:* Monday, April 22, 2013 10:56 AM *Subject:* Re: feature request: parallel builds feature On Mon, 2013-04-22 at 00:42 -0700, Jim Michaels wrote: it currently has a problem with stdin, because at this point there is only one of those, only 1 of them gets it, and the others starve. so if your build needs stdin or creates files from the commandline using heredocs, you can't use it (check first!). you will get an error. gnu has not yet figured out a solution yet (I have, multiple shells IF you can control them... probably can't without some work creating batch files for the jobs). so there is a solution. even with a batch file, make would need some sort of way of reporting back error conditions. I think there are ways of managing that with files via presence-detect, sort of like semaphores. they should get cleared when the job ends, or when a switch is given to clear the state for that session if the session was broken with ctrl-c. well, I suppose a ctrl-c handler should still kill those terminals or cmd shells and clear up those files. what do you think? if a terminal is opened, it should be created without a window. some OS's have that option. some don't, like freeDOS, which would not have the ability to do terminals, parallel shell windows, or even the --jobs feature (but that's implementation-dependent). Please keep the mailing list CC'd. Thanks. I'm afraid I still don't understand what you're asking for here. You'll need to back up and provide a description of your needs in a clear, orderly way without digressions. Yes, it's true that GNU make only provides its stdin to one job at a time and which job gets it is essentially random. In order to address this we'd need to see a specific use-case or requirement, but my suspicion is that all such possible use-cases are better solved by a change of process at a level above what make can provide. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request: parallel builds feature
Jim Michaels wrote: what if you in your makefile are creating files from scratch using echo, based on system configuration information? I know I have to do that in order to create XML manifest files for resources to compile and link in via resource compiler for windows builds. echo writes to stdout. That has nothing to do with stdin. Looks to me like you have no F'ing idea what you're talking about. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Thoughts on limiting the parallel build load, suggestion for a new -j auto option
Edward Welbourne wrote: Go and shoot the moron who wrote that script. Easy to say, but no use to someone who needs to deploy a package whose build infrastructure includes a badly-written script. If the author of that script has been shot, there's probably no-one alive who understands how to build the package. Quite possibly, the author only knocked the script together for personal use, but made it available to the project as a start-point for others to set up their own builds, only to see it chucked into the distribution because it was more use than nothing. They could have committed the script with either totaly benign settings, or obviously broken settings that force the user to substitute in valid ones. (Since otherwise most users will ignore any documentation.) E.g., changed to make -j PICK_A_NUMBER would have been better. The make -j value should never be encoded in any file; it should only ever be set by the user invoking the actual top level make command. and therein lies the problem with -j and -l; they require the user to have a sophisticated understanding of the system parameters and build load profile in order to guess a half-way decent value for them. Such options are much less useful than ones that have sensible defaults. Make *does* have sensible defaults. The default is to run serially. Sometimes computing is hard. Users *do* need a sophisticated understanding of their systems. That's life. For developers of the package being built, it's possible to learn - by trial and error - what settings tend to work out reasonably sensibly on specific machines. I have heard various ad hoc rules of thumb in use, typically number of CPUs plus either a fixed offset or some fraction of the total number of CPUs, as either -j or -l value. In the end, I know that my old machine grinds to a halt if I let make take load about 4 and my shiny new machine accepts -j 8 -l 12 without breaking a sweat - and gets the build done in a few minutes, instead of half an hour. Yes. If you know how to improve on trial and error, go ahead and post your code. For anyone who has to build a package that isn't the primary business of their work - for example, a distribution maintainer, for whom each package is just one in a horde of many - configuring the right -j and -l flags for each run of make is not practical. It would make sense to provide them with a sensible way to tell make to make good use it can of the CPUs available, without bringing the system to its knees. It might make sense, but it's not the developers' responsibility to know what is sensible for every machine configuration out there. It is the end user's responsibility to know something about the system they're using. You can't expect developers to even have exposure to all the possible parallel configurations on which someone will attempt to build their software. A distro maintainer probably has a farm of machines to build on. That set of machines is already a known quantity to them, because presumably they've already done a lot of builds on those machines. They're the ones with the most knowledge of how the machines behave, not any particular developer. You can't just encode # CPUs x fudge factor into the Make source. The CPUs may be fake to begin with (e.g., SMT with Intel Hyperthreading or Sun Niagara). Putting a database of CPU types/knowledge into Make and maintaining it would be ludicrous. End users should know their own machines. The present unconstrained -j behaviour is, in any case, self-defeating. It starts so many processes that the time the machine spends swapping them all out and back in again swamps the time they spend actually doing anu useful work. The build would complete sooner if fewer processes were started. Pretty sure the documentation already says so too. Anyone using just make -j is a moron. I think R. Diez's suggestions are a constructive step towards designing some half-way sensible heuristics - those with better understanding of what make's internal decision-process looks like can doubtless improve on them, which I'm sure is R. Diez's hope and intent in starting this discussion. We don't need -j auto to achieve perfectly optimal tuning on every machine; it'll be good enough if it can do builds faster than make's default implicit -j 1, without (too often) loading the machine so heavily as to interfere with browsing, reading mail, playing nethack and other things developers like to still be able to do while waiting for a build. Eddy. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #35485] New $(.MFDIR) built-in variable
Paul Smith wrote: On Thu, 2012-02-09 at 17:25 +, Tim Murphy wrote: I also think that it can be expensive to append things onto very long lists in make but that's just suspicion and I really need to take a proper look at it instead of making accusations. Appending is not very expensive. Make doesn't know anything about lists internally. Everything in make is a string. If you append to a variable then make basically does a strlen() of the string, makes sure there's enough memory for the new content in the buffer (or if not, reallocs enough space), then strcat()'s the value. Or the equivalent. So I suppose yes, in some ways the longer the string the longer it takes to do the strlen() but this should be very fast I'd expect. Much of the string manipulation in make (and make does a LOT of string manipulation) is bog-stupid strlen, strcat, strcpy, strdup which gets expensive in some situations. I've considered implementing some kind of fancy string encoding . You've just described an O(n^2) behavior. This is definitely a drag; but it really depends on how heavily you make use of macros. In OpenLDAP 2.0 70% of our execution time was in libc; 35% in malloc/free and 35% in strlen/strcat and friends. I instituted a top-down replacement of all bare char *s in the code to struct bervals (char *ptr; int len) and eliminated all use of strlen in the code and our code base was immediately 60% faster. A number of other profiling and optimization exercises made OpenLDAP 2.1 *20* times faster than OpenLDAP 2.0, but getting rid of moronic uses of libc was the number 1 priority. The thing that DOES get noticeably more complex is manipulating the variable as if it were a list, with functions like firstword, lastword, etc. because those functions must first chop the string up into words, then operate on those, and that happens every time the function is invoked. If that happens a lot, then you should simply construct the lists once and keep them... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Thoughts on limiting the parallel build load, suggestion for a new -j auto option
R. Diez wrote: Hi all: I recently came across a build test script that launched make -j with no limits, which consumed all available RAM and ground my Linux system to a halt. I had to power the computer off, there was nothing else I could do. Go and shoot the moron who wrote that script. After this experience, I believe that the default limitless behaviour of -j is too dangerous. Even if your PC does not end up dying, GNU Make may inadvertently consume too many resources. An explicit -j infinite would be a better option, if anybody really needs something like that. I recently came across option -l , which limits the amount of parallel tasks based on the system's average load. However, this flag does not seem safe enough, as, according to Wikipedia, not all Unix systems include in the average load those processes currently waiting for disk I/O. Besides, the maximum average load one would almost certainly want to use depends on the number of CPUs, so the calling script (or user) has to find out how many CPUs there are. How you find out may also depend on the operating system underneath, so everybody gets a little extra work every time. -l is utterly useless. Load average is computed too slowly; by the time it passes any useful threshold the actual make load will have spiralled out of control. I am writing a build test framework myself, and I have been trying to coordinate all sub-makes from a main makefile. The top-level script decides how many parallel processes are allowed for the entire build and relies on MAKEFLAGS in order to let all sub-makes talk to each other so as to limit the overall load. Because different makefiles may require different GNU Make options, I am filtering out all others and leaving just the parallel build flags in place, like this: export MAKEFLAGS=$(filter --jobserver-fds=%,$(MAKEFLAGS)) $(filter -j,$(MAKEFLAGS)) $(MAKE) ...etc... By the way, option --jobserver-fds is not documented, but I think it should be. As you can see, the user may need to filter it out manually after all. The trouble with this MAKEFLAGS technique is that I often come across some third-party script which insists on calculating and setting its own -j value, rendering my coordination efforts useless. When this happens, I get warnings like this: Go and shoot the morons who wrote those scripts. The make -j value should never be encoded in any file; it should only ever be set by the user invoking the actual top level make command. warning: -jN forced in submake: disabling jobserver mode Needless to say, most heuristics to calculate the -j value are as lame as mine (see below). When writing build scripts, nobody seems to have much time left for finding out how to retrieve the relevant system information in bash/perl/whatever in a portable way and then calculate a good -j value out of it. Nobody writing scripts should ever bother with such a thing. It's for the end-user to control; any other effort will be wrong 99.9% of the time. I have been thinking about the best way to overcome such parallel woes, and I wanted to share this thought with you all. How about adding to GNU Make a new -j parameter like this: make -j auto The behaviour of -j auto could be as follows: 1) If MAKEFLAGS specifies -j and --jobserver-fds , then use those settings (no warning required). 2) Otherwise, calculate the maximum number of parallel tasks with some trivial heuristic based on the number of CPUs and/or the system load. I'm usingCPU count + 1 at the moment, but I'm sure there are better guesses. I could think of several alternative heuristics: make -j auto-system-load # Use -lCPU count + 0.5 make -j auto-processor-count # Use -jCPU count + 1 I guess most people would then end up using some -j auto variant, in order to avoid overloading or underloading the system without having to implement their own heuristics. That way, a top-level script will be much more likely to succeed at setting a global limit when launching third-party sub-makes in parallel. A good value depends entirely on the actual workload. For most compiles I find that CPU count * 1.5 works well. Some people use make for things other than invoking compilers though, and some compilers have different resource requirements than a typical gcc run, so even that isn't a widely applicable estimate. Only the end user can determine the best value. Please copy me on the answers, as I'm not on this list. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: New Feature Submission for GNU Make
Edward Welbourne wrote: Pretty weak. If a few more include paths were added to the project it would still break, regardless of your patch. When you have thousands of object files, shortening the name of each by several bytes when invoking ar can make room for really quite a lot more files before you run up against the command-line length limit. Never understimate the value of a modest shortening of file names - when our ar command-line got to about 230 kibibytes, we had to re-work the way we invoked ar ! Whatever you're doing, you're doing it wrong. ;) If you're tossing a bunch of object files into an ar library simply for grouping purposes, and all of the object files will eventually be used in the final executable, it's faster to use ld -r and combine groups of them all into a single object file. If you profiled your build I'm sure you would find that a large percentage of time is being wasted in ar itself. And if you're really running a build with thousands of object files, on a frequent basis, you're foolish not to profile it and figure out what else you're doing wrong. Giant monolithic command lines also prevent you from leveraging parallel make, so that's another reason not to do things that way... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: New Feature Submission for GNU Make
Ben Robinson wrote: Eli, Thank you for your constructive criticism regarding my submission to GNU Make. I perceive the critiques to fall into three categories (documentation, justification and code improvements) which I will respond to in order. *Documentation*: You are correct, these functions remove only redundant or unnecessary . and .. components. The suggested documentation should instead read: $(trimpath names|...|) For each file name innames,returns a name that does not contain any|unnecessary.|or|..|components, nor any repeated path separators (|/|). Canonical absolute names remain canonical absolute, and relative names remain relative. $(relpath names|...|) For each file name innames,returns the relative path to the file. This relative path does not contain anyunnecessary |.|or|..|components, nor any repeated path separators (|/|). *Justification (trimpath)*: trimpath can be used to shorten the input strings to compilers/linkers, as well as improve readability to the user. Certain compilers have a maximum number of characters which can be passed into a single invocation of their compiler. In my project, I had a dozen or so -includesearch_path which contained many unnecessary . and .. components, which caused the compiler to overflow the input buffer. While it is unfortunate compilers exist with unreasonably small input buffers, trimpath allowed me to only pass in the minimum number of characters required to successfully compile. Pretty weak. If a few more include paths were added to the project it would still break, regardless of your patch. Also, the readability of paths printed to the console is greatly improved by trimpath. I was regularly dealing with paths of the following structure: -I../../../_make_interface/../../../_make_system/source/_make_implementation/../3rdparty/libs/gtest/include which would be reduced by trimpath to: -I../../../../../_make_system/source/3rdparty/libs/gtest/include At first glance this sounds like a good thing, but it seems to miss the possibility of weirdly symlinked build trees, where foo/bar/.. != foo. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Eli Zaretskii wrote: Date: Fri, 15 Apr 2011 10:37:13 -0400 From: David Boycedavid.s.bo...@gmail.com it's more a question of how many parallel jobs a given make process is managing since limits on file handles/descriptors are per process. What about people who use make -j without limits? They're idiots. I've never seen a machine that can handle more than 1.5x number of cores without getting bottlenecked by I/O or system scheduler. (And I've run this stuff on 1024 node Connection Machines, among other things...) Go above a certain threshold and you're only slowing yourself down. It's not like having a 256-core machine is a fantasy that will not happen in a few years. On my day job, we are already using 12 cores and don't consider that living on the edge. We've already seen bug reports from people running on machines with 4096 cores, trying to run with -j8192 or somesuch. (Default pipe buffer is 4K which sets -j4096 as the practical limit without mucking with kernel params.) -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Chiheng Xu wrote: On Tue, Aug 3, 2010 at 2:51 PM, Tim Murphytnmur...@gmail.com wrote: Since some things happen at the same time there is no single serial order. The semaphore mechanism, forces one of the possible orders. I'm not familiar with source code of make, but I believe the serial order of shells is determined by the dependence DAG, it may be unique for a given dependence DAG. Shells can be issued and completed at random order(only need satisfying the dependence relation). But make can print their outputs strictly in their serial order. I'm trying very hard to only provide constructive comments in response to this thread, but frankly this is, in a word, stupid. If you want make's output to be in serial order, then don't use parallel make at all. The point to parallel make is that it allows jobs which have no ordering dependency to run in parallel. If you want their output to be fully serialized, then you will force make to wait for them to complete serially. Which automatically also means that make will have to maintain an arbitrarily large internal queue for all of the output, because given the unpredictable completion times of multiple jobs running concurrently, no output can be emitted until the slowest parallel job completes. In particular, if you have recursive makefiles, no parent make process can output anything at all until all of its submakes have completed, because no individual make process has enough knowledge about what the actual serial order is. Given that this discussion seems to have arisen due to the braindead stdio handling in Cygwin, it seems like any de-mangling of parallel make's output should be directed to the Cygwin libraries. In my experience Cygwin is too slow an environment to be useful anyway, which is why I use MSYS for Windows builds. But I have to admit, I only use it inside a single-core VirtualBox these days so I haven't looked at how parallel make behaves there. But the fact is all I/O in Cygwin is funneled through the Cygwin DLL, so there's no reason that it can't be fixed to not mingle/mangle lines from different processes together. But again, that's not gnu-make's problem, that's a Cygwin issue. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Edward Welbourne wrote: 2x is too much. 1.5x has been the best in my experience, any more than that and you're losing too much CPU to scheduling overhead instead of real work. Any less and you're giving up too much in idle or I/O time. This depends a bit on whether you're using icecc or some similar distributed compilation system. I believe a better approach is to set a generous -j, such as twice the count of CPUs, but impose a load limit using -l, tuned rather more carefully. Scheduling overhead contributes to load, so is taken into account this way. Perhaps in a perfect world -l would be useful. In fact, since load averages are calculated so slowly, by the time your -l limit is reached the actual CPU load will have blown past it and your machine will be thrashing. That's the entire reason I came up with the -j implementation in the first place. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
Chiheng Xu wrote: On Fri, Jul 30, 2010 at 5:35 PM, Eli Zaretskiie...@gnu.org wrote: I asked for an example. Could you please show a messy output and the output you'd like to have after serialization? TIA serially make : execute A, B, C programs, they print: A: Hello, I'm A, I am from Earth. B: The moon is my home. C: Welcome to Mars, It's an amazing planet. parallely make : the output of A, B, C programs interleave : C: Welcome to B: The moon is my A: Hello, I'm A, I am from Earth.home.Mars, It's an amazing planet. This seems like quite an extreme example. stdout is line buffered by default, so individual lines would get written atomically unless the programs you're running are doing weird things with their output. In the common case interleaving like this doesn't happen within lines, it only happens between lines of multi-line output. stderr may skew things since it's usually nonbuffered, but again, that's not the common case. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Chiheng Xu wrote: -- Forwarded message -- From: Chiheng Xuchiheng...@gmail.com Date: Fri, Jul 30, 2010 at 6:02 PM Subject: Re: [RFC]serialize the output of parallel make? To: Tim Murphytnmur...@gmail.com On Fri, Jul 30, 2010 at 5:54 PM, Tim Murphytnmur...@gmail.com wrote: Hi, The cost to the system of even starting a shell to invoke a recipe is so huge compared to the time needed to reserve a semaphore that it is insignificant in comparison. The amount of contention is limited by -j i.e. by how many processes there are ( 2 * CPUs is usually considered reasonable) and by how long 2x is too much. 1.5x has been the best in my experience, any more than that and you're losing too much CPU to scheduling overhead instead of real work. Any less and you're giving up too much in idle or I/O time. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
Paul Smith wrote: On Fri, 2010-07-30 at 09:59 +0800, Chiheng Xu wrote: As parallel make are becoming more and more popular, can make serialize the output of parallel make? Make can redirect every parallelly issued shell's output to an temporary file, and output the stored output serially, as if in a serial make. This would be a good thing, but as always the details are not quite so trivial. Aside from the difficulties outlined below, I just am not fond of having output batched up instead of appearing in realtime. It tends to complicate the polling logic too (though I guess in this case, you just have to cat the appropriate file[s] whenever a child process ends.) The scheme that I grew up with on Alliant Concentrix was just to prefix each output line with its job number |xx|blah blah blah. It obviously requires a pipe for each child process' output, so that lines can be read by the parent make and then the prefix attached. In the original jobserver prototype I used unique bytes for each job token so that the token == the job ID, with an eye toward adding this support later. But that obviously limited it to supporting only 256 concurrent jobs, and these days we already get complaints that it's limited to only 4096. Using a pipe per job would likewise cut make's maximum job count in half (or worse, if using a separate stderr pipe). I still favor this latter approach because it keeps the output flowing in realtime, and its easy enough to use grep if you want to zero in on a single output stream. But the cost in resources will add up... We have to ensure that these temporary files are cleaned up properly, even in the face of users ^C'ing their make invocations. We also need to verify that whatever methods we use will work properly on Windows and VMS and other operating systems make supports (where are their /tmp equivalents?) And, what about stdout vs. stderr? Should we write both to the same file? Then we lose the ability to do things like make -j4 2/dev/null since all output will be written to stdout (presumably). Or should we keep two files per command, one for stdout and one for stderr? But that's even worse since then when we printed it we'd have to print all the stdout first then all the stderr, which could lose important context. Then there's the possibility that some commands will behave differently if they are writing to a TTY, then they will if they're writing to a file. Do we not care about that, or do we try to do something crazy with PTYs or similar (ugh!) And of course we have to have a guaranteed unique naming strategy in case multiple instances of make are running on the same system at the same time, maybe running the same makefiles and even building the same targets. On POSIX systems we can use tmpfile() or mkstemp() or something but other systems might need other measures. These are just some things I thought of off the top of my head. It certainly does not mean that it would not be a good thing to have this ability though. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[bug #22923] option to prevent interspersed output in parallel builds
Follow-up Comment #7, bug #22923 (project make): Reaching back 25 years again, this is basically what Alliant Concentrix parallel make did. It prefixed each output line with the job number |1|, |2| and so on. (This is also why in my original proposal for make -j, I used different numbered job tokens, so that each make invocation could output its own unique number.) Obviously the way that was done was by using a pipe for each spawned jobs' stdout/stderr, so that the invoking make could attach the prefix to any output lines. Of course, in the context of the current make -j server, this would only allow us to handle 255 concurrent jobs. After working on a Connection Machine with 1024 nodes, it became obvious that this wasn't a perfect solution... As a side note - stdout is always linebuffered by default. It's stderr that may present problems, since by default it is unbuffered. Just using each jobs' PID as a prefix isn't quite as pleasant, you get a much noisier output since a single job can spawn a lot of processes and thus cycle thru many PIDs in rapid succession. ___ Reply to this item at: http://savannah.gnu.org/bugs/?22923 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[bug #28134] inconsistent reporting of linux kernel builds with -j2
Follow-up Comment #3, bug #28134 (project make): As a complete digression, and based on 26 years of doing parallel builds (damn it's been a long time), I just wanted to point out that you can generally use 50% more jobs than your number of CPUs/cores. E.g., for a dual-core machine, use make -j3. Compilation tends to be heavily I/O bound; if you only use as many jobs as you have cores, you will tend to always have at least 30% idle CPU. Bump up the jobs by 50% and you can utilize that extra CPU without overloading the machine. Go beyond that and you tend to lose. ___ Reply to this item at: http://savannah.gnu.org/bugs/?28134 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #22923] option to prevent interspersed output in parallel builds
Brian Kustel wrote: Follow-up Comment #3, bug #22923 (project make): Thanks ipjrest and psmith for both replies. They are very appreciated. Responding to ipjrest's comments: = I agree some commands are long-running and that if watching progress is important, the proposed behavior may not be desireable. This is one reason I suggested the proposed new behavior be optional. 1) Line-buffering: While it would help in some cases, not intermixing lines from different processes is only partially helpful. The fact that a single line from process A might be interrupted by a line from process B is my concern here. For example, instead of seeing Process A Fragment 1 was emitted and then fragment 2 was emitted. Process B Message one might see A: Process A Fragment 1 was emitted B: Process B Message A: and then fragment 2 was emitted. if the line for process A is emitted in 2 separate write operations. 2) The VS line prefixing would be OK. In fact I like this idea. But it still has the problem that single lines can be broken if the tool being run sometimes emits lines in pieces. I have noticed that gcc has this problem in some cases (at least the gcc version I am currently using, 4.1.0). Responding to psmith's comments: Thanks for pointing out that make is not multithreaded. I guess I was assuming it was. I agree this complicates matters somewhat. Also, I see your point about my original stdout/stderr marking idea. In both cases I assumed make would have either an auxilliary thread or its main thread doing these tasks but I now realize that this is not how make currently works. But I think something along the lines of your special variable idea could be employed in a generic fashion. Perhaps this is what you intended your second paragraph below to imply? Specifically, the proposed option could cause all commands to have their output piped to a standard tool which stores (its) stdin to a temp file and then writes it synchronously to stdout after stdin is closed. Also, I think it could be reasonable to allow for an option to either combine stdout/stderr here or to pipe them to separate processes (so that stdout/stderr could be atomically emitted to stdout/stderr respectively). While I have nothing against the special variable idea, I'd very much like to see a way to accomplish this via a make command-line option and the variable idea sounded as if it might require a makefile modification. Just my 2 cents, the Alliant parallel make (which inspired my design of the make -j jobserver) also redirected all of the spawned processes' stdio to a set of pipes, so that all of their output was prefixed with |jobnum|. It was quite nice, but I never felt it was important enough to duplicate that behavior for gmake. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Switching from CVS to GIT
Paul Smith wrote: It looks like this discussion may have been premature, perhaps by as little as a few weeks or so, based on the fact that Savannah has Subversion support in beta right now and that there seems to be a lot of action around GIT support on Windows that is being published either now or very shortly. So, let's table this discussion for the time being. We'll revisit it later this year when hopefully some things have shaken out. In the meantime I'm going to play with some of these other tools. If you (esp. non-Linux users) have time to try them out as well I'd really appreciate any suggestions you come up with. IMO the objections to requiring MSYS/Cygwin on Windows made no sense in this discussion. Make is inherently a POSIX command line tool. Anybody using it on Windows needs a POSIX environment already anyway. Whether or not you're building Windows-native code with it doesn't really enter the equation. Aside from that, happy to leave this for later. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: EAGAIN in read jobs pipe
james coleman wrote: not much! also a build making more calls to make can result in jfactor * number of make calls jobs Only if you're using a totally braindead implementation of make. The whole point of the jobserver pipe is to eliminate this fanout problem. So I might regularily use maybe -j 10 (when I know 10 more calls to make are made) On some machines this might be unacceptable and bring them to their knees. On other machines it can work really well. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Martin Dorey wrote: However, your point about programs invoked by make inheriting the setrlimit() is definitely something that seems problematic. Perhaps GNU make could change the stack limit back to what it was after it forks but before it execs its child. I wonder what happens if you change a limit to something too small for the current processes resources? It doesn't look specified by http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html. This test suggests that it works fine on my Linux box. I presume the limit check is normally done on a faulting access off the end of the stack, so setrlimit would need some to have some different and special code to check the current stack usage if it wanted to apply the check immediately. It seems a bit unlikely that anyone would have gone to that effort, thus stopping you from doing something useful in this sort of situation. In Linux (and most other Unix systems) the stack is allocated in page-sized chunks, and an extra page with no access permissions is mapped at the end of it. Accessing that page is what generates the SEGV that occurs when you overrun the stack. The mappings are only established when the stack is grown, and generally the stack never shrinks. So if you already have X pages of stack legitimately in use, lowering the rlimit isn't going to have an immediate effect. I.e., the stack growth occurred before the limit decreased. In general, Unix doesn't impose restrictions retroactively. (E.g., if you open a file and have a valid descriptor on it, and someone else chmods the file, removing your access permissions, you still have your original privileges through the open descriptor.) -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Jon Grant wrote: My view would be that on modern computers switching to allocate from the heap wouldn't make a big difference if it were changed. Modern heaps have pools for small allocations to stop them fragmenting larger allocations anyway. Someone would need to do a compressive test to know for sure, these things often have knock on effects.. I've seen massive slowdowns when someone switched malloc() to calloc() on MS-Windows! Jon Choice of malloc implementation can have a huge effect on execution time. See this data http://highlandsun.com/hyc/#Malloc Some modern mallocs are good, but stack-based allocation is still better a lot of the time. Especially for temporary variables that are just going to be discarded after a few computations. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[bug #17873] .NOTPARALLEL enhancements
Follow-up Comment #3, bug #17873 (project make): Haven't been paying a lot of attention, but when is such a lame thing nas .NOTPARALLEL needed? Why aren't people just writing proper dependencies so that parallel/serial is irrelevant? ___ Reply to this item at: http://savannah.gnu.org/bugs/?17873 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[bug #17881] Better documentation of make rules
URL: http://savannah.gnu.org/bugs/?17881 Summary: Better documentation of make rules Project: make Submitted by: hyc Submitted on: Thursday 09/28/2006 at 20:26 Severity: 3 - Normal Item Group: Enhancement Status: None Privacy: Public Assigned to: None Open/Closed: Open Component Version: None Operating System: None Fixed Release: None ___ Details: The GNU make manual should state explicitly in section 4.2, Rule Syntax, something to this effect: When multiple prerequisites are listed for a rule, their order is insignificant. In particular, you should not assume that prerequisites will be processed in left-to-right order. Rules that list prerequisites that depend on other prerequisites in the same rule are invalid and will break when processed by a parallel make. For example, given a rule targetA: reqOne reqTwo reqThree if reqThree cannot be built successfully until reqOne already exists, then this rule does not specify the actual dependencies of targetA. The correct rules would be: reqThree: reqOne targetA: reqTwo reqThree ___ Reply to this item at: http://savannah.gnu.org/bugs/?17881 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #17881] Better documentation of make rules
Bah. Pretty sure I selected Documentation for this submission, not Enhancement. Howard Chu wrote: URL: http://savannah.gnu.org/bugs/?17881 Summary: Better documentation of make rules Project: make Submitted by: hyc Submitted on: Thursday 09/28/2006 at 20:26 Severity: 3 - Normal Item Group: Enhancement Status: None Privacy: Public Assigned to: None Open/Closed: Open Component Version: None Operating System: None Fixed Release: None ___ Details: The GNU make manual should state explicitly in section 4.2, Rule Syntax, something to this effect: When multiple prerequisites are listed for a rule, their order is insignificant. In particular, you should not assume that prerequisites will be processed in left-to-right order. Rules that list prerequisites that depend on other prerequisites in the same rule are invalid and will break when processed by a parallel make. For example, given a rule targetA: reqOne reqTwo reqThree if reqThree cannot be built successfully until reqOne already exists, then this rule does not specify the actual dependencies of targetA. The correct rules would be: reqThree: reqOne targetA: reqTwo reqThree -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[bug #17881] Better documentation of make rules
Follow-up Comment #1, bug #17881 (project make): It might be worthwhile to add some kind of makefile validation flag, which processes all prerequisites in random or right-to-left order, specifically to expose occurrences of incorrect prerequisites. ___ Reply to this item at: http://savannah.gnu.org/bugs/?17881 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #17873] .NOTPARALLEL enhancements
David Fang wrote: Follow-up Comment #3, bug #17873 (project make): Haven't been paying a lot of attention, but when is such a lame thing nas .NOTPARALLEL needed? Why aren't people just writing proper dependencies so that parallel/serial is irrelevant? Example: In my experience, I frequently build pdf and ps documents from the same latex sources, via latex and pdflatex. Unfortunately, the intermediate auxiliary files they produce are not compatible with each other (latex vs. pdflatex), thus, I use .NOTPARALLEL: foo.pdf foo.dvi, and use a last-build stamp mechanism to clobber the .aux when needed. Everywhere else, I write 'proper' dependencies, and make -j. :) Thanks for that example. (Though for your specific case, since pdf is essentially entropy-coded postscript, you shouldn't need to rerun latex to generate it.) I guess if we're stuck with dumb programs that generate intermediate files whose names can't be changed on the command line, then this sort of approach is the only way out. BTW, this might be off-topic for this thread, but has anyone looked into randomized ordering of dependency-building? (*GASP* why on earth would anyone want to do that!?) I've noticed that SCons has this feature, but it might come in handy for verifying that a makefile is properly written. Yes, I've suggested this a number of times in the past. Never formally in a bug report, but #17881 which I just submitted lists that as well. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #17873] .NOTPARALLEL enhancements
David Fang wrote: Maybe it's time we use the source and submit a patch? :) I remember taking a look at the source where children jobs are dispatched, but don't recall deciding where/how to hack in random ordering. When I wrote the original jobserver code, that was something interesting that I couldn't live without, and it was worth my effort to push it forward. Frankly this is trivial and personally uninteresting, and I can live without it just fine, since I know how to write correct Makefiles. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sunhttp://highlandsun.com/hyc OpenLDAP Core Teamhttp://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Error in processing of :: targets
[EMAIL PROTECTED] wrote: Gnumake maintainers, I've run into a functional difference between the standard unix make and GNUmake (3.80) involving how the :: rules are processed. In the standard unix make (Solaris), the :: targets ( i.e., all::) are process in the order they are found in the Makefile serially, regardless of whether parallelism is used (i.e., -jN, where N is 1). That's funny, my standard Unix (Solaris) make doesn't have a -j switch, and doesn't support parallelism. In fact, -j is a GNU-specific extension, and it Works As Designed. In GNUmake (3.80), the when -j1 is used, the :: targets are executed serially. When -jN is used the :: targets are executed in parallel. Consider the following -- from an IMAKE generated Makefile: This, like most of the dreck that Imake produces, is bogus. http://lists.gnu.org/archive/html/bug-make/2001-09/msg00013.html http://lists.gnu.org/archive/html/bug-make/2004-09/msg1.html all:: all.prologue all:: all.serial all.parallel all:: ; @echo $@; sleep 1; echo [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] all:: all.epilogue all.epilogue all.prologue:: ; @echo $@; sleep 2; echo [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] all.serial:: ; @echo [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]; sleep 1; echo [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] all.serial:: ; @echo [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]; sleep 1; echo [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] all.parallel: aaa bbb ccc aaa bbb ccc: ; @echo $@; sleep 1; echo [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: parallelism: job slots vs. levels
Dan Jacobson wrote: Anyways, $ make x make y wait cannot always be rewritten with -j. $ make -j[whatever number] x y will act differently except for special cases of x and y; probably when both x and y have no dependencies. Anyways, with -j examples added to the manual, we would get on the right track about how to use -j. I'm not quite sure what the point of all this is. I guess the texinfo docs for parallel execution may be somewhat lacking. Here's a summary of Things You Must Know. In a typical Makefile with target and dependencies: foo: bar baz cmds the Make syntax specifies that all of the dependencies are of equal precedence. That means, bar and baz can be made in any order and the result must still be valid. If there is really a dependency such that bar must be created before baz, then the above rule is incorrect. (Or at least, insufficient.) It should be written instead as baz: bar cmds foo: baz cmds In serial make you can get away with the incorrect syntax, because make only fires off one command at a time and it typically runs dependencies in left-to-right order. But that's just an idiosyncracy of the particular implementation. You could run across a make implementation that always processes dependencies right to left. If your Makefile follows correct syntax, it will still work. If it depends on left-to-right serial execution, it breaks, as well it should. In parallel make, all the dependencies of a target can be processed simultaneously. As such, it is imperative that you *explicitly* define all dependency relationships in your Makefile. You can't rely on the *implicit* relationship that says that left-most dependencies will be built before right-most. If in your example of make x make y wait x and y are completely indepedent, then the result will be correct, as would make -j x y. But if you're relying on some implicit behavior to complete x before starting y then your Makefile is broken. The point is that in a correctly written Makefile with all dependency relationships explicitly spelled out, there will be no difference whether make is run serially or in parallel. If there is any difference, then the Makefile rules didn't correctly capture all of the real dependencies of the project. -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support ___ Bug-make mailing list [EMAIL PROTECTED] http://lists.gnu.org/mailman/listinfo/bug-make
Re: parallelism: job slots vs. levels
Dan Jacobson wrote: Anyways, $ make x make y wait cannot always be rewritten with -j. $ make -j[whatever number] x y will act differently except for special cases of x and y; probably when both x and y have no dependencies. make x make y wait will only work correctly if x and y have no dependencies in common. Otherwise, the two separate make's will get in each other's way when they work on whatever is in common. This fact has nothing to do with parallel make or make -j. Anyways, with -j examples added to the manual, we would get on the right track about how to use -j. Since your two examples have nothing to do with each other, I don't see how you can reach this conclusion. -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support ___ Bug-make mailing list [EMAIL PROTECTED] http://lists.gnu.org/mailman/listinfo/bug-make
RE: GNU enhancement -- $(xargs cmd, list)
I would omit the XARGS LEN. Presumably the local platform's xargs command is already properly configured for the native system's line length limitations. -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Reid Madsen Sent: Monday, November 06, 2000 5:42 PM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: GNU enhancement -- $(xargs cmd, list) Date: 6 Nov 2000 16:56:30 -0600 From: Reid Madsen [EMAIL PROTECTED] I also thought about using the _POSIX_ARG_MAX and ARG_MAX symbols to determine when the xargs approach was really needed. If the command line is within limits, then $(xargs cmd, list) would expand to: cmd list Do you believe this is doable/portable? Reid FYI, a quick perusal of our systems shows: POSIX_ARG_MAX ARG_MAX - --- Solaris 40961MB HP 40962MB AIX 409624576 Pathetic! OSF 409638912 Pathetic! NT 409614500 Pathetic ** 2 ! Gotta love those 2-4 byte per arg limits on AIX, OSF, and NT. After experimenting with this on Solaris for about 3 hours, I've come to the conclusion that this is way to complicated. The 'exec' man page states that the error E2BIG is returned when the sum of the command length and environment length exceeds ARG_MAX. So, armed with that information I when out and created a Makefile with enough big arguments to push it right up to the edge. Adding a single character would case make to fail with: make: execvp: num_args: Arg list too long However the 'edge' was nowhere close to the size of the command line + environment. It's off by about 100K. Yuck! So, instead, I've decided that in addtion to the $(XARGS) variable that GNUmake should also define a $(XARGS_LEN) variable. Which will be used to measure the length of the command line. When less the $(XARGS_LEN), then a normal command is launched, otherwise xargs is launched as" xargs -s $(XARGS_LEN) cmd tmpfile In our build process, based on IMAKE, we can set XARGS_LEN differently for the different platforms we build on. IMO, this is probably the most flexible. If you have a better idea, I'd be glad to hear it. Reid -- Reid Madsen [EMAIL PROTECTED] Senior Member, Tech. Staff (972) 536-3261 (Desk) I2 Technologies (214) 850-9613 (Cellular) -- Making incrementally (without 'clean') since May 18, 2000. Long live incremental make! ___ Bug-make mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/bug-make
RE: GNU enhancement -- $(xargs cmd, list)
Your xargs feature sounds useful, but your bug example is flawed. Remember that the shell does all wildcard expansion in Unix. For most problem cases, your echo *.xx | xargs command commandline would fail for the same reason that command *.xx would fail. Just wanted to make sure the actual problem mechanics were understood... -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Reid Madsen Sent: Monday, November 06, 2000 7:46 AM To: [EMAIL PROTECTED] Subject: GNU enhancement -- $(xargs cmd, list) Consider the following: ARGS=$(wildcard *.xx) target: command $(ARGS) The above only works if the length of the resulting command line is within system limits. From the shell, you can solve this by: echo *.xx | xargs command But the above does not work in make because it expands the wildcard, and then launches a command line that is again too long. I propose that GNUmake add the following function: $(xargs cmd, list) Then I could render the intial example as: ARGS=$(wildcard *.xx) target: $(xargs command, $(ARGS)) Which resolves to: command some_args; command more_args; command even_more_args where the number of commands depends on the command line length for the given platform. This would allow us to write Makefiles that work on all platforms regardless of command line length. Since I'm somewhat familiar with the GNUmake source I'd like to take a stab at implementing this. If you can give me any pointers on how I should proceed I would appreciate it. Regards Reid Madsen -- Reid Madsen [EMAIL PROTECTED] Senior Member, Tech. Staff (972) 536-3261 (Desk) I2 Technologies (214) 850-9613 (Cellular) -- Making incrementally (without 'clean') since May 18, 2000. Long live incremental make! ___ Bug-make mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/bug-make
RE: GNU enhancement -- $(xargs cmd, list)
Try "xargs command tmpfile" instead of "cat tmpfile | xargs command" ...
-- Howard Chu
Chief Architect, Symas Corp. Director, Highland Sun
http://www.symas.com http://highlandsun.com/hyc
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of
Reid Madsen
Sent: Monday, November 06, 2000 12:29 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: GNU enhancement -- $(xargs cmd, list)
Well, I've got a prototype up and running and IMHO it's quite cool.
All changes are in function.c. The net of the changes are:
* Added support for:
$(xargs command, list)
which is expanded to:
cat tmpfile | xargs command; rm -f tmpfile
* Added 'func_xargs' as shown below:
static int xargs_file_count = 0;
static char *
func_xargs(o, argv, funcname)
char *o;
char **argv;
const char* funcname;
{
char buff[256];
char filename[256];
FILE* fd;
sprintf (filename, "/tmp/gnumake.%d.%d.xargs", getpid(),
xargs_file_count++);
fd = fopen(filename, "w");
fprintf(fd, "%s\n", argv[1]);
fclose(fd);
sprintf (buff, "cat %s | xargs ", filename);
o = variable_buffer_output(o, buff, strlen(buff));
o = variable_buffer_output(o, argv[0], strlen(argv[0]));
sprintf (buff, "; rm -f %s", filename);
o = variable_buffer_output(o, buff, strlen(buff));
return o;
}
This works perfectly! Well at least it works perfectly within my limited
understanding of GNUmake.
Additional considerations:
* On machines with dynamic command lines (i.e., Solaris), the $(xargs)
function should just expand to:
command list
No need for xargs in those cases.
* Allowing user definition of the XARGS command and options.
Something like:
XARGS = myxargs
XARGS_OPTIONS = -blah -blah...
with the defaults being:
XARGS = xargs
XARGS_OPTIONS =
Let me know if you want to integrate this. Personally, I see
this as a very
useful extension to GNUmake.
Regards,
Reid Madsen
--
Reid Madsen [EMAIL PROTECTED]
Senior Member, Tech. Staff (972) 536-3261 (Desk)
I2 Technologies (214) 850-9613 (Cellular)
--
Making incrementally (without 'clean') since May 18, 2000.
Long live incremental make!
___
Bug-make mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/bug-make
___
Bug-make mailing list
[EMAIL PROTECTED]
http://mail.gnu.org/mailman/listinfo/bug-make
RE: does gnumake 3.77 wildcard work under solaris 2.6?
This is a known bug in 3.77. 3.79 is the current version, you really need to update. -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc -Original Message- From: John Lewis 425-865-3510 [mailto:[EMAIL PROTECTED]] Sent: Thursday, June 15, 2000 10:07 AM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: does gnumake 3.77 wildcard work under solaris 2.6? We're trying to use the wildcard function in gnumake, and find that it produces nothing. The goal is to produce a list of the .o files that should be checked against all the .f files in a directory, as directly from page 18 of the 3.77 manual (and as of some earlier versions of gnumake that worked). "Making" with the attached Makefile in a directory with more than a dozen .f files produces the following output: % gnumake echo echo echo echo The old syntax (from an earlier gnumake) we think omitted the colon in the := assignment. We tried that too with the same results. Here's the Makefile. Any thoughts for us? Thanks, John Lewis cut here - OBJECTS := $(patsubst %.f, %.o, $(wildcard *.f)) TESTOBJ := $(patsubst %.f,%.o,$(wildcard *.f)) test: echo $($(wildcard *.f)) echo $(wildcard *.f) echo $(TESTOBJ) echo $(OBJECTS) -- John G. Lewis | [EMAIL PROTECTED] Technical Fellow | [EMAIL PROTECTED] The Boeing Company | tele: (425) 865-3510 Mail Code 7L-22 | fax : (425) 865-2966 P.O. Box 3707 | Seattle WA 98124-2207 | ---
RE: -j fails on DYNIX/ptx
I can confirm that many filesystem operations can fail with EINTR when operating over NFS. However, someone had to be causing the signals in question - SIGALRM, maybe, or keyboard SIGINT, etc. Protecting stat() is usually the most important remedy, since failure will cause you to think that a file does not exist, even though it does. The big question is what is generating the signals - presumably it's SIGCHLD in this case, and there's probably not much you can do about that. -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc -Original Message- From: Paul D. Smith [mailto:[EMAIL PROTECTED]] Sent: Wednesday, May 31, 2000 12:22 PM To: Michael Sterrett -Mr. Bones.- Cc: [EMAIL PROTECTED] Subject: Re: -j fails on DYNIX/ptx %% "Michael Sterrett -Mr. Bones.-" [EMAIL PROTECTED] writes: ms Do the specifications say that EINTR is not required or that it ms is forbidden? Hmm. They say that a function may have more return error codes than listed in the standards, so you're right: I guess there's nothing technically preventing EINTR. stat() is listed as required to be safe to call within a signal handler, which isn't directly related of course. ms EINTR A signal was caught during the stat() or ms lstat() function. ms Without the code, there's no way for me to know if Solaris will ms actually ever fail with EINTR, but the man pages seems to indicate ms that it *could*. I suspect, but can't prove, that you won't get EINTR from stat(2) for "normal" filesystems. ms I guess I'm still not convinced that this problem couldn't be ms reproduced if sufficiently adverse conditions were ms encountered, even on "normal" UNIX systems. I'm for adding ms the loop on EINTR to the GNU make code base. I just find it hard to believe that there have been no other reported cases of this anywhere else; if it's possible, no matter how obscure, it seems like _someone_ else would have hit it somewhere, by now. Sigh. The problem is, there are a number of places that use stat(2) in GNU make in addition to this one. Do we need to armor them all? What about other system calls? If this is a real (even potential) problem for most/all OS's, then maybe a different solution than wrapping all the system calls in EINTR checks is in order. -- -- - Paul D. Smith [EMAIL PROTECTED] Find some GNU make tips at: http://www.gnu.org http://www.ultranet.com/~pauld/gmake/ "Please remain calm...I may be mad, but I am a professional." --Mad Scientist
