Re: Detecting parallel builds
On Fri, Apr 8, 2011 at 7:58 PM, Philip Prindeville phil...@redfish-solutions.com wrote: Didn't hear back, so I assume there is no easy way to detect (currently) a parallel build. There was a response by Edward Welbourne - didn't you see it? It's not a documented/supported solution but I believe if you find any of -j, --jobs, or --jobserver-fds in $(MAKEFLAGS) the build is parallel. Since the latter two both match --jobs%, this should boil down to two nested findstring function calls on the MAKEFLAGS value. David Boyce ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
patch to support output synchronization under -j
I recently wrote a little program called syncsh (https://github.com/boyski/SYNCSH) to synchronize the output of make -j such that the results of a given job stay together. It's based on ideas from a few posts on this list and an old Ask Mr Make article (http://www.cmcrossroads.com/ask-mr-make/12909-descrambling-parallel-build-logs), and it works fine (in limited testing) so feel free to try it out. However, the point of this email is that after getting syncsh working and seeing how simple the basic mechanism is, I started thinking Why couldn't this be done within GNU make itself?. Because even if syncsh works perfectly, it's another program to port/maintain/document/install plus it requires a shell and an extra fork/exec sequence for each job. So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? It's enabled by a new variable currently called MAKESYNCFILE but I won't go into how it works because that's already done at the URLs above. Caveats: (a) I'm not sure I'm hooking in at the right place. What I have seems to work fine but only on a per-line basis - i.e. if a recipe has multiple lines it will not cohere them. That's to be expected of syncsh, but it seems to me make itself should be able to sync per-recipe rather than per-recipe-line. (b) No consideration has yet been given to non-POSIX platforms. The code requires only the ability to create a pre-removed temp file (as in the tmpfile() API) and to lock a file. WRT to the temp file, it looks to me like on those systems which don't support tmpfile() natively it could be mocked up using the existing open_tempfile() function in GNU make. As for file locking, I used fcntl locking, I'm sure there's a Windows equivalent, and I suspect that any system that can't lock a file probably doesn't support make -j anyway. (c) It doesn't allow for more than one value of MAKESYNCFILE. Right now it opens the first value of MAKESYNCFILE it sees and holds onto that file descriptor. To do this right would require a table mapping pathnames to file descriptors, but I don't see a use case for varying the sync file path. (d) Documentation is TBD but I'll wait and see if the patch is wanted first. Current version attached, with test makefile. David Boyce make-sync.patch Description: Binary data Makefile.synctest Description: Binary data ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, Apr 12, 2011 at 1:46 PM, David Boyce david.s.bo...@gmail.com wrote: So I've made a proof-of-concept patch against 3.82.90 which seems to work without that overhead and my question is, would this be of interest towards 3.83? Ping? The original patch attachment was made by hand using diff -u and I had some trouble applying it myself. So here's a version created with cvs -q diff -uN which should work better. I've done copyright assignment paperwork in the past, BTW. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 1:29 PM, Paul Smith psm...@gnu.org wrote: I've looked at it and as a concept I don't have too many issues with it Thanks. One example: I think saving stdout and stderr to different files and then printing them separately is problematic; consider if your recipe prints lots of information lines, with errors (to stdout) interspersed. If you throw all the errors to the end you lose a lot of context. I actually think this is unavoidable. At least it's unavoidable when using a separate shell wrapper; it may be possible to better internally to make with some extra work. The reason is that the SHELL variable is used not only for recipes but also for the $(shell) function. Intermingling stdout and stderr in the result of $(shell) is just disastrously wrong (as I found in the first iteration of syncsh). I spent some time trying to find a way to determine, from inside a child shell, whether we were forked by a recipe or by $(shell) but could find no reliable way. Thus syncsh was forced to keep them in separate files. Since $(shell) invocations are not jobs according to make's process model there's no need for them to participate in synchronization at all, so it may be that within make there's a way to only sync on recipes which in turn would allow 21. Of course, either way some context is lost. If you put both into one temp file you lose track of which was which; if you keep them in separate files you lose ordering instead. So it becomes a matter of taste, or perhaps an option though that seems like a bit too much to me. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 1:48 PM, Eli Zaretskii e...@gnu.org wrote: David, can you explain why you needed to lock the files? Also, what region(s) of the file you are locking? fcntl with F_WRLCK won't work on Windows, so the question is how to emulate it. I was about to write this up but I see Paul beat me to it ... Finally, I'd suggest that the system-dependent portions of sync_output be factored out into a separate function, so that the Windows port could have its own implementation without infesting job.c with yet another bunch of #ifdef's. Agree. Ifdef bad. Just wanted to see if the feature would be favorably looked on first. I can't actually do the Windows etc ports but I could certainly refactor it with stubs. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 2:10 PM, Paul Smith psm...@gnu.org wrote: There is no specific region of the file that's locked: the lockfile is basically a file-based, system-wide semaphore. Yes, it's conceptually a semaphore. In fact a Windows port might prefer to use real semaphores. The reason I stayed away from them on the POSIX side is that, according to my reading, POSIX semaphores don't go away automatically when the creating process exits; they need to be explicitly released. Which entails a lot of responsibility and complexity in terms of signal handling and so on. An exclusive file lock produces the same result but goes away with its file descriptor. File locking is also older and thus likely a little more portable, though both are pretty old. The entire file is [...] empty and has no content. That's not actually the case with the current implementation. The file can have content, in fact the obvious candidate is often the Makefile as shown in the test case. It must be writable according to fcntl locking semantics but it is never written to. I'm not sure I like the idea of having to define a lockfile to enable this feature though. It feels a little too much like exposing the implementation details to the user. Agree. This needs a little more thought. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 3:00 PM, Eli Zaretskii e...@gnu.org wrote: Yes, but a few words about how is this semaphore supposed to get job done, and in fact what kind of synchronization will this bring to Make, would be appreciated. I don't think you described the feature too much in your original post. No, but what I did say was I won't go into how it works because that's already done at the URLs above. Do you have specific questions not addressed there? They're both pretty short, and I really can't tell what more needs to be said without some context. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 3:08 PM, David Boyce david.s.bo...@gmail.com wrote: On Thu, Apr 14, 2011 at 2:10 PM, Paul Smith psm...@gnu.org wrote: I'm not sure I like the idea of having to define a lockfile to enable this feature though. It feels a little too much like exposing the implementation details to the user. Agree. This needs a little more thought. I don't know why this hasn't occurred to me or the authors of similar programs before, but it appears to be possible to get a lock on any writable file descriptor - for instance stdout or stderr, or one of the jobserver-fds. I just changed syncsh to synchronize around a lock on stdout and it seems to work. This looks much more elegant than specifying a writable file. Or is my understanding of file/pipe/locking semantics flawed? And how would Windows/DOS/VMS/OS2 would do with this? David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, Apr 15, 2011 at 6:45 AM, Eli Zaretskii e...@gnu.org wrote: I lack the higher-level picture. Can you describe in a few words how will this work in a running Make? I don't mean the details of how files are locked and unlocked (I understand that part), I mean the larger picture, like how it will look to the user, how several programs running in parallel will write their output and in what order it will appear eventually on the screen. I'll try to illustrate with a different example. For a test case I've taken an open-source package called jikes which compiles ~40 C++ files. I had to modify the automake-generated makefile to supply illustrative verbosity; here's the compilation pattern rule: %.o: %.cpp @echo BEGINNING $@ ... \ g++ -DHAVE_CONFIG_H -I. -g -O2 -c -o $@ $ \ echo FINISHED $@ ...; and I've attached output logs without.txt and with.txt, both using -j for unlimited jobs. You'll notice that in without.txt the output of recipes is separated; all the BEGINNING lines show up as soon as the job is forked, and all the ENDING lines show up as it finishes. In the with case, the BEGINNING lines are effectively held and released in sync with their matching ENDING lines. The main problem being solved here is that error and warning messages often get separated from their recipes in build logs. It's hard to debug foo: permission denied when you can't figure out which recipe generated that message. Does that help? David BEGINNING case.o ... FINISHED case.o BEGINNING jikes.o ... FINISHED jikes.o BEGINNING tab.o ... FINISHED tab.o BEGINNING lpginput.o ... FINISHED lpginput.o BEGINNING code.o ... FINISHED code.o BEGINNING dump.o ... FINISHED dump.o BEGINNING op.o ... FINISHED op.o BEGINNING unzip.o ... FINISHED unzip.o BEGINNING unparse.o ... FINISHED unparse.o BEGINNING long.o ... FINISHED long.o BEGINNING segment.o ... FINISHED segment.o BEGINNING jikesapi.o ... FINISHED jikesapi.o BEGINNING init.o ... FINISHED init.o BEGINNING modifier.o ... FINISHED modifier.o BEGINNING zip.o ... FINISHED zip.o BEGINNING set.o ... FINISHED set.o BEGINNING parser.o ... FINISHED parser.o BEGINNING platform.o ... FINISHED platform.o BEGINNING scanner.o ... FINISHED scanner.o BEGINNING option.o ... FINISHED option.o BEGINNING depend.o ... FINISHED depend.o BEGINNING incrmnt.o ... FINISHED incrmnt.o BEGINNING stream.o ... FINISHED stream.o BEGINNING system.o ... FINISHED system.o BEGINNING diagnose.o ... FINISHED diagnose.o BEGINNING error.o ... FINISHED error.o BEGINNING lookup.o ... FINISHED lookup.o BEGINNING definite.o ... FINISHED definite.o BEGINNING double.o ... FINISHED double.o BEGINNING ast.o ... FINISHED ast.o BEGINNING javaact.o ... FINISHED javaact.o BEGINNING control.o ... FINISHED control.o BEGINNING body.o ... FINISHED body.o BEGINNING symbol.o ... FINISHED symbol.o BEGINNING decl.o ... FINISHED decl.o BEGINNING class.o ... FINISHED class.o BEGINNING expr.o ... FINISHED expr.o BEGINNING bytecode.o ... FINISHED bytecode.o rm -f jikes g++ -g -O2 -o jikes ast.o body.o bytecode.o case.o class.o code.o control.o decl.o definite.o depend.o diagnose.o double.o dump.o error.o expr.o incrmnt.o init.o javaact.o jikes.o jikesapi.o long.o lookup.o lpginput.o modifier.o op.o option.o parser.o platform.o scanner.o segment.o set.o stream.o symbol.o system.o tab.o unparse.o unzip.o zip.o BEGINNING ast.o ... BEGINNING body.o ... BEGINNING bytecode.o ... BEGINNING case.o ... BEGINNING class.o ... BEGINNING code.o ... BEGINNING decl.o ... BEGINNING depend.o ... BEGINNING diagnose.o ... BEGINNING double.o ... BEGINNING dump.o ... BEGINNING error.o ... BEGINNING expr.o ... BEGINNING incrmnt.o ... BEGINNING init.o ... BEGINNING javaact.o ... BEGINNING jikes.o ... BEGINNING long.o ... BEGINNING jikesapi.o ... BEGINNING lookup.o ... BEGINNING lpginput.o ... BEGINNING op.o ... BEGINNING option.o ... BEGINNING modifier.o ... BEGINNING parser.o ... BEGINNING platform.o ... BEGINNING segment.o ... BEGINNING set.o ... BEGINNING scanner.o ... BEGINNING stream.o ... BEGINNING symbol.o ... BEGINNING system.o ... BEGINNING tab.o ... BEGINNING unparse.o ... BEGINNING unzip.o ... BEGINNING zip.o ... FINISHED tab.o FINISHED case.o FINISHED dump.o FINISHED jikes.o FINISHED op.o FINISHED code.o FINISHED lpginput.o FINISHED unparse.o FINISHED long.o FINISHED unzip.o FINISHED jikesapi.o FINISHED segment.o FINISHED zip.o FINISHED set.o FINISHED parser.o FINISHED modifier.o FINISHED init.o FINISHED platform.o FINISHED scanner.o FINISHED depend.o FINISHED incrmnt.o FINISHED error.o FINISHED option.o FINISHED stream.o FINISHED system.o FINISHED double.o FINISHED diagnose.o FINISHED definite.o FINISHED lookup.o FINISHED javaact.o FINISHED ast.o FINISHED body.o FINISHED control.o FINISHED class.o FINISHED symbol.o FINISHED decl.o FINISHED expr.o FINISHED bytecode.o rm -f jikes g++ -g -O2 -o jikes ast.o body.o bytecode.o case.o class.o code.o control.o decl.o definite.o depend.o diagnose.o double.o dump.o error.o
Re: patch to support output synchronization under -j
On Thu, Apr 14, 2011 at 4:30 PM, David Boyce david.s.bo...@gmail.com wrote: I don't know why this hasn't occurred to me or the authors of similar programs before, but it appears to be possible to get a lock on any writable file descriptor - for instance stdout or stderr, or one of the jobserver-fds. I just changed syncsh to synchronize around a lock on stdout and it seems to work. This looks much more elegant than specifying a writable file. Attached is an updated patch with some improvements: - The feature is turned on by a special target currently called .PARALLELSYNC. No filename required. - It uses stdout as its semaphore. - It combines output into one temp file iff the inherited stdout and stderr are already combined, as suggested by Howard Chu. Very lightly tested, of course. See the Makefile.synctest for a usage example. David make-sync.patch Description: Binary data Makefile.synctest Description: Binary data ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, Apr 15, 2011 at 9:48 AM, Eli Zaretskii e...@gnu.org wrote: So effectively, whenever a job finishes, its parent Make takes the semaphore, outputs all of the output of that job to the screen, then releases the semaphore, is that right? Yes, exactly. And how do you communicate the name of the semaphore file to sub-Make's? Is that by assuming that its file descriptor is inherited? I also don't understand the need in the MAKESYNCFILE variable: why is it important where the semaphore file will live and how will it be named? Addressed by second iteration. There is no sync file anymore. Finally, wouldn't it be a potential problem top inherit so many handles to subordinate processes (2 for each running job)? We could run out of available handles in deeply recursive jobs, couldn't we? Just to be completely clear, we're not talking about a change to default behavior - it's merely a new option. But yes, any feature which consumes additional resources increases the risk of running out of that resource. I don't know that recursion comes into it though, it's more a question of how many parallel jobs a given make process is managing since limits on file handles/descriptors are per process. In other words if using the jobserver you're limited to the number of jobs specified at the top level even in a recursive setup. If not, make tries to work around that by forcing -j1, and if that doesn't work you're in trouble anyway. The strictest file limit I know of is on old Unixy platforms. The tmpfile() API returns a FILE *, not a file descriptor, and older Unices have a limit of 256 of these. But even in that corner case, how many people are going to run make -j200+? How many 64-CPU processors are running really old kernels? So yes, while there is a theoretical limit there are many kinds of resource exhaustion which can be triggered by large values of -j and as far as I can tell this is just another, and optional. Parenthetically, I only use FILE * because that's what tmpfile hands back. I believe it would be possible to dup the file descriptor out of the FILE *, close it right away, and work with an integral file descriptor from there. That would avoid the above limit. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, Apr 15, 2011 at 11:09 AM, Eli Zaretskii e...@gnu.org wrote: But this new option uses up 2 additional files per job, doesn't it? One or two, as discussed elsewhere in this thread. Each sub-Make inherits all the file descriptors of all its parents, grandparents, etc. If a sub-Make was launched when 4 other jobs were running in parallel, the sub-Make will inherit 8 file descriptors that it won't use. The deeper the recursion, the more inherited descriptors. You have a good point. I should probably turn on FD_CLOEXEC for these. What about people who use make -j without limits? As noted, those people are at risk of exhausting many resources. It's not like having a 256-core machine is a fantasy that will not happen in a few years. On my day job, we are already using 12 cores and don't consider that living on the edge. Right, but my point was the limits of which I speak are old. I don't think a 256-core processor will ever run on a kernel from the 1980's. Or in other words, part of scaling an OS to many cores involves removing or at least upping artificial resource constraints. The natural trend toward 64-bit systems is part of that of course. I think the real kernel limitation, if there is one, is on file descriptors, not FILE objects. The latter live in the application. On that you are wrong, at least WRT POSIX machines. The old limit on streams was a function of an 8-bit field in the FILE structure, whereas every system I've ever used can allocate at least 1024 descriptors. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Fri, Apr 15, 2011 at 1:03 PM, Eli Zaretskii e...@gnu.org wrote: Can Make be invoked with its stdout closed by the parent process? If it is, will this still work? I've tried to code it such that if anything goes wrong setting up the sync, of which a closed stdout would be one example, it prints a message and continues on without the sync feature but does not fail the build. Such corner cases have not been tested much. + memset(fl, 0, sizeof(fl)); + fl.l_type = F_WRLCK; + fl.l_whence = SEEK_SET; + fl.l_pid = getpid(); + fl.l_start = fl.l_len = 0; + if (fcntl(syncfd, F_SETLKW, fl) == -1) + perror(fcntl); As discussed, please factor this out into a separate function. As soon as things settle down. + if (c-tempout) + while ((nread = fread(buffer, 1, sizeof(buffer), c-tempout)) 0) + write(fileno(stdout), buffer, nread); Are we sure c-tempout will be zeroed out by initialization? They're explicitly initialized: + child-tempout = child-temperr = NULL; + if (c-tempout) + fclose(c-tempout); + if (c-temperr) + fclose(c-temperr); Should we assign NULL to these once they are closed? Yes. I'm working on a new version which replaces these with descriptors anyway. Same principal applies though. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
Here's an updated version which is much closer to done at least from my POV. I believe most of the requested improvements are there, the semaphore code is factored, there's even a start on documentation. Despite what was said about ifdefs before, the entire patch is currently enclosed in #ifdef POSIX /* PARALLEL-SYNC */ pending any ports. David make-sync3.patch Description: Binary data ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, Apr 19, 2011 at 10:13 AM, Edward Welbourne e...@opera.com wrote: How about if we introduce a variable that can be set for specific targets (using the target-specific variable mechanism). Targets that set this variable to some predefined value will not have their output redirected, so their output will go to the screen. IIUC, the proposal is to make the feature be enabled by a .NAME target (I forget the name); .PARALLELSYNC, as currently coded. perhaps the simplest thing would be to have the feature applied to any target declared to depend on the given .NAME, supplemented with command-line options to turn it on/off everywhere, regardless of such dependencies. Well ... that's no longer the feature I signed up to contribute, and I don't have or see a major use case for it. I think the issue most people have with parallel builds involves debugging scheduled/automated builds with output to a log file. I can't think of a situation where the above feature would be important, but if someone wants to add it as a further enhancement I'm all for it, assuming my enhancement makes it into the codebase in the first place. David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, Apr 19, 2011 at 11:06 AM, David Boyce david.s.bo...@gmail.com wrote: Here's an updated version which is much closer to done Ignore this version, I found a bug :-( David ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[bug #33134] spurious error when stdout is already closed
URL: http://savannah.gnu.org/bugs/?33134 Summary: spurious error when stdout is already closed Project: make Submitted by: boyski Submitted on: Thu 21 Apr 2011 01:03:09 AM GMT Severity: 3 - Normal Item Group: Bug Status: None Privacy: Public Assigned to: None Open/Closed: Open Discussion Lock: Any Component Version: CVS Operating System: None Fixed Release: None Triage Status: None ___ Details: Ironically, make's attempt to be super-duper careful about catching write errors to stdout (in the close_stdout function) results in a spurious error message when the user has already closed stdout: % cat makefile .PHONY: all all:; date % make 1- make: write error Patch attached. ___ File Attachments: --- Date: Thu 21 Apr 2011 01:03:09 AM GMT Name: make-stdout-patch Size: 418B By: boyski http://savannah.gnu.org/bugs/download.php?file_id=23282 ___ Reply to this item at: http://savannah.gnu.org/bugs/?33134 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33134] spurious error when stdout is already closed
On Wed, Apr 20, 2011 at 9:58 PM, Philip Guenther guent...@gmail.com wrote:
Could you explain why you think that's spurious? Make wanted to write
date to stdout and the write failed. Seems legit to me.
That's actually not what generates the error message. I'm not sure why
it doesn't happen the way you say, but it doesn't. Did you look at the
patch? Here it is:
Index: main.c
===
RCS file: /sources/make/make/main.c,v
retrieving revision 1.246
diff -u -r1.246 main.c
--- main.c 29 Aug 2010 23:05:27 - 1.246
+++ main.c 21 Apr 2011 00:53:24 -
@@ -952,7 +959,8 @@
#endif
#ifdef HAVE_ATEXIT
- atexit (close_stdout);
+ if (ftell(stdout) != -1)
+atexit (close_stdout);
#endif
/* Needed for OS/2 */
Basically in a (sensible and nicely documented) attempt to detect all
errors, make does an explicit close of stdout just before exiting in
order to make one last check for failure modes. However, it makes the
mistake of assuming stdout was open to start with. So in fact what the
child process did with stdout has nothing to do with this bug. It's as
simple as closing an already-closed file descriptor and printing the
resulting error, and my patch is as simple as only close it if it's
open.
(I'm not sure why it doesn't say make: write error: Bad file
descriptor like it does if you change the command invoked to :, but
that's a distinct issue.)
I noticed this and found it interesting as well. It gets even stranger
too: looking at close_stdout() there are two possible messages:
if (prev_fail || fclose_fail)
{
if (fclose_fail)
error (NILF, _(write error: %s), strerror (errno));
else
error (NILF, _(write error));
exit (EXIT_FAILURE);
}
If you try the test case with -jN you get the first error but with -j
alone you get the second one.
David
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33134] spurious error when stdout is already closed
On Thu, Apr 21, 2011 at 1:48 AM, Philip Guenther guent...@gmail.com wrote: [...] All quite true and admirably researched but this is not a standards-lawyering exercise, it's a software-engineering issue. I can't remember how many installers I've run which, when cancelled, finish by saying ERROR - the software is not installed!. And I always think That's not an error, I got what I asked for. The only error would be if the product ended up installed. Here we have the same thing. If the user has explicitly closed stdout before handing it to make, does s/he need to be alerted to that fact by make? And even if so, should it take the form of an ERROR message? I'd say no, because when the user gets exactly what s/he asked for that cannot be regarded as an error condition. If make feels the need to say Warning: unable to write to stdout because it's closed that would be reasonable, though a little fussy for my taste, but there's no error condition here. I feel like we're trapped in a classic Internet chest-thumping loop, but my day job requires that I break it. I'd say the bug report exists and this thread provides all the commentary one could wish, so I'm sure Paul will be in a position to make an informed decision when he gets to it, and let's leave it at that. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #33134] spurious error when stdout is already closed
Follow-up Comment #1, bug #33134 (project make): Note that there is a thread on the bug-make mailing list discussing this in some detail: http://lists.gnu.org/archive/html/bug-make/2011-04/msg00077.html ___ Reply to this item at: http://savannah.gnu.org/bugs/?33134 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33134] spurious error when stdout is already closed
On Thu, Apr 21, 2011 at 11:29 AM, Philip Guenther guent...@gmail.com wrote: On Thu, Apr 21, 2011 at 3:50 AM, David Boyce david.s.bo...@gmail.com wrote: All quite true and admirably researched but this is not a standards-lawyering exercise, it's a software-engineering issue. Why are you closing stdout instead of redirecting it to /dev/null? While developing a patch for the .PARALLELSYNC feature being discussed in another thread (#33138), I tested a number of corner cases which included stdout being closed. That helped me find some bugs in my work but I also noticed that even unmodified make had this behavior, so I submitted a standalone patch to fix it. So - in this context at least - I'm not a user doing a dumb thing, I'm a developer doing a careful thing. In real life I've never had a valid reason to close stdout. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
I've filed this as an enhancement request (https://savannah.gnu.org/bugs/index.php?33138) for better tracking. Unfortunately there's a typo in the headline which I can't fix. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #33134] spurious error when stdout is already closed
Follow-up Comment #2, bug #33134 (project make): My original one-line patch had two mistakes, which may be a record! In the course of work for enhancement #33138 I made a macro: #define STREAM_OK(strm) ((fcntl(fileno((strm)), F_GETFD) != -1) || (errno != EBADF)) whose logic would be much better than the original ftell(). Turns out that on many platforms ftell cannot be used on a non-seekable descriptor at all. My development is on Solaris where ftell works as a go/no-go indicator, but that isn't portable. And the EBADF test is required for reasons which are obvious and also discussed in the email thread. ___ Reply to this item at: http://savannah.gnu.org/bugs/?33134 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #33138] .PARLLELSYNC enhancement with patch
URL: http://savannah.gnu.org/bugs/?33138 Summary: .PARLLELSYNC enhancement with patch Project: make Submitted by: boyski Submitted on: Thu 21 Apr 2011 04:17:07 PM GMT Severity: 3 - Normal Item Group: Enhancement Status: None Privacy: Public Assigned to: None Open/Closed: Open Discussion Lock: Any Component Version: CVS Operating System: POSIX-Based Fixed Release: None Triage Status: None ___ Details: A patch will be provided to support a new special target, tentatively called .PARALLELSYNC, which keeps the output of parallel jobs discrete. The background is amply described in a long thread starting at http://lists.gnu.org/archive/html/bug-make/2011-04/msg00018.html My patch will only address POSIX systems but with luck this can be ported to the full set of platforms which also support -j ___ Reply to this item at: http://savannah.gnu.org/bugs/?33138 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #33138] .PARLLELSYNC enhancement with patch
Follow-up Comment #1, bug #33138 (project make): Latest patch attached. This works well in my tests on Solaris and Linux. I know of no reason it wouldn't work the same on any POSIX platform, and it's ifdef-ed so as to not break the build on any non-POSIX platform. A start on documentation is included. (file #23292) ___ Additional Item Attachment: File name: make-sync.patchSize:11 KB ___ Reply to this item at: http://savannah.gnu.org/bugs/?33138 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, May 3, 2011 at 1:33 AM, Paul Smith psm...@gnu.org wrote: I wonder if we can figure out a way to make this work better, as Eli asked. Can we work out a way to handle normal rules (rules with + or $(MAKE)) and sub-make rules differently, so that output from normal rules wasn't collected behind a sub-make? I'm not sure I see exactly how this could work. Thinking, thinking, ... The other thing I was thinking is that this feature might want to be enabled via a command-line argument. All the complex makefiles generated by automake, etc. for example cannot take advantage of this if you have to modify every makefile to add the special target. Hmm, it feels like you've reversed position since last year? When I submitted the patch for .ONESHELL it included a new --one-shell flag and you rejected the flag part saying you didn't want make to end up like GNU tar in the sense of having an overwhelming number of options. Personally I don't see what's so bad about exposing useful features at the command line, and I can't say I ever wished that GNU tar would do less of it, but I figured that was your aesthetic and I'd go with it. That was when you came up with the --eval option, which I found brilliant BTW. So I guess the first-order answer to your point would be there is a way to enable this at the command line and you invented it: --eval=.PARALLELSYNC:. However, I personally like command-line options and if you want a first-class flag you'll get no argument here. [PARALLELSYNC vs OUTPUTSYNC] I'm pretty agnostic on the name. I actually considered .OUTPUTSYNC too but felt the name should expose the fact that it's a no-op unless -j is in use. But I'm happy with it if that's the consensus. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: patch to support output synchronization under -j
On Tue, May 3, 2011 at 1:20 PM, Paul Smith psm...@gnu.org wrote: In that way, SYNC is a feature that the makefile USER selects, or not, and not something the makefile AUTHOR would choose. Does that make sense? It makes perfect sense when you put it that way and I agree wrt to both .ONESHELL and .PARALLELSYNC (or whatever it's called now). Except, I don't remember if --eval is passed down to sub-makes? It is. The mind goes first... Remind me, what were we talking about? David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #33034] Makefile:23: *** mixed implicit and normal rules. Stop. for Linux kernel out of source builds
Follow-up Comment #6, bug #33034 (project make): In fairness, this is a special situation. I have the same problem as tz, and I expect quite a few other people do too. My company makes drivers. The corollary is that we keep dozens if not hundreds of entire Linux kernel source build trees going back 10 years or so in order to build drivers for them. The way Linux kernel drivers are built makes it necessary to use the kernel build system. Thus it's not sufficient for us to fix our own makefiles, or even that Linux kernels which came out in the last year are fixed themselves. As things stand, we'll be unable to upgrade from GNU make 3.81 until all pre-2010 kernels have drained out of our support matrix, which could easily be another 10 years. Of course we also have the option of going back and fixing the makefiles for each old kernel, but once you've changed 2.6.18-128.7.1 then it's no longer 2.6.18-128.7.1. So I at least think this issue is different from other incompatibilities and worthy of special consideration. If the original fix was a complete rewrite of parsing then maybe that's the end of the story, but if it's just a little patch which could be switched at runtime, would it be possible to add a special target to control that switch? I might sign up for the work if you agree it's not an outrageous thing to support. ___ Reply to this item at: http://savannah.gnu.org/bugs/?33034 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: New Feature Submission for GNU Make
On Tue, May 31, 2011 at 8:06 AM, Edward Welbourne e...@opera.com wrote: Never understimate the value of a modest shortening of file names - when our ar command-line got to about 230 kibibytes, we had to re-work the way we invoked ar ! My vote would be favorable as well. There are a number of subtleties which should not be underestimated: - Really long lines in a logfile makes it hard to visually process them and detect errors or warnings. - Long lines cause logfiles themselves to take up more space which can cause them to fill partitions, force an organization to carry fewer old logs, etc. Where I work we have a nightly build report with clickable links to logfiles. It can take quite a while for a browser to load up a big logfile, especially for colleagues working over slow lines. - As noted before, the ability to compare logfiles with relativized paths is helpful, as is the ability to take the failing command line from someone else's build output and copy and paste it into your workspace for debugging. - Objects built for debug tend to have full paths to source files built in. Shortening the paths can thus shrink the objects themselves and make debugging more easily portable. Also some organizations may want to ship debug objects without exposing their internal development infrastructure. Of course it's possible to do some relativizing now with a combination of abspath and subst, but it's not fully robust. I think that canonpath would be a better name than trimpath; more mnemonic, more standard. David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: $(sort) - what is lexical order? (was RE: Follow-up)
On Tue, Jul 19, 2011 at 3:00 PM, Paul Smith psm...@gnu.org wrote: I agree that the manual should document the fact that the sort function does not sort according the current LC_COLLATE value but instead always uses the standard ASCII (or LC_COLLATE=C) order. But I will not say that it doesn't sort lexically, because that's not true: it does. Agree completely, and add a note to the OP that the sort function has an extremely important side effect of removing repeated words. Another reason to keep it, if that was meant seriously. Why not trivially s/lexical/ASCII/ on the affected line in the manual? Lexical may be technically correct but ASCII is more precise. David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [rfc] Colorized output for GNU make?
On Mon, Oct 3, 2011 at 12:40 PM, Tim Murphy tnmur...@gmail.com wrote: I have no decision making abilities so this comment is just an observation Similarly, I'm just a bystander who should be doing real work. Whatever people decide to do, I'm interested in being able to parse the output of make when it's so long that you can't really look at it all with your eyes because it would take you a week. i.e. you need software that can summarise and extract interesting information from a huge quantity of output. I have two reactions to the original post: 1. I hate colorized output in all its forms. If you want to add this feature and can get it in that's fine with me as long as it will never show up as a default in any native build of make. 2. I don't know if you've used Electric Make, which is a commercial make which aims for 100% GNU make compatibility, but they've extended their variant to allow for XML-tagged output. From this they can generate graphs and charts and derive metrics and so on. So I think a more general solution would be to offer XML-style output as a GNU make option, and then it would be trivial to post-process that for colorizing as well as a number of other useful purposes. I can think of a small list of make output categories. Let's see: recipe command lines printed by make verbosity other make output debug the stuff printed with -d dbthe stuff printed by make -p info text from the $(info) function error,warning as above ??? Anything not within one of the tags would be considered regular command output. If you were doing serial build, or parallel and had a synchronization feature such as in https://savannah.gnu.org/bugs/?33138, then output could be nested inside the recipe tag from which it derived which would be more useful. I'm pretty sure ecmake does something like that. Anyway, I think that would have more general utility than colorization per se. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [rfc] Colorized output for GNU make?
On Thu, Oct 20, 2011 at 2:19 PM, Sebastian Pipping sebast...@pipping.org wrote: - Add parameter --color[=(yes|no)] to enable/disable color from the command line. Previously, I suggested XML output piped to a colorizing filter and you indicated a lack of interest in going that way. Which is no problem; it's your contribution, and there are definitely downsides to a pipeline approach. But I continue to think that XML output a la ecmake would enable a lot of useful post-processing, so the only feedback I have on this patch is a suggestion that instead of --color=(yes|no) you use something like --output-format=(color|plain) where of course plain is the default. All your work to do table-driven formatting via a standard output function could easily be applied to XML generation and it could be supported at the UI level without adding another flag by allowing --output-format=xml. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33034] Makefile:23: *** mixed implicit and normal rules. Stop. for Linux kernel out of source builds
Judging from the outpouring of support I received when I made the same suggestion/offer :-( I'm unsurprised at seeing no public response to yours either. But disappointed. I agree that this is a real problem which does not seem to resonate with those not directly affected by it. I think the best approach is probably to follow the CVS commit trail, find the commit which fixed the original bug, and see if it can be made conditional on a special target .ALLOW_MIXED_RULES: or something. This could then be requested without any makefile modification by something like export MAKEFLAGS=--eval .ALLOW_MIXED_RULES: Alternatively, if Paul would allow it, the test could be directly for an EV. The question in my mind, which would probably be answerable by finding the commit, is whether the original fix was a simple patch which could be easily conditionalized or a part of a large-scale parser rewrite. David On Thu, Nov 3, 2011 at 11:54 PM, Jonathan Nieder invalid.nore...@gnu.org wrote: Follow-up Comment #7, bug #33034 (project make): Just for the record, I have been looking for a spare moment to come up with a patch to fix this compatibility problem (which affects many projects other than the Linux kernel, too). If that's a bad idea for some reason, please feel free to let me know why. :) And if anyone else starts before I do, I won't mind --- on the contrary, I'll be very happy. ___ Reply to this item at: http://savannah.gnu.org/bugs/?33034 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Using shell wrapper for descrambling parallel make output
On Mon, Nov 14, 2011 at 4:29 AM, Atte Peltomäki atte.peltom...@iki.fi wrote: Nice work. Your implementation seems much more refined than mine. Only one thing catches my attention; your version doesn't seem to properly preserve the original line ordering between stdout and stderr. I suggest solving this as I did: Atte, Thanks. I assume (without having looked) that your ideas refer to the standalone program and not the patch for make itself? My problem, aside from limited time of course, is that I have no indication yet of whether the patch is likely to be accepted into 3.83 which in turn has an effect on how much work I want to put into the standalone solution. Paul, any chance you can provide a status update or thumbs up/down on http://savannah.gnu.org/bugs/?33138? -David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35063] Suggest conventional $(%XX) syntax for special characters
URL: http://savannah.gnu.org/bugs/?35063 Summary: Suggest conventional $(%XX) syntax for special characters Project: make Submitted by: boyski Submitted on: Wed 14 Dec 2011 02:27:53 AM GMT Severity: 3 - Normal Item Group: Enhancement Status: None Privacy: Public Assigned to: None Open/Closed: Open Discussion Lock: Any Component Version: CVS Operating System: Any Fixed Release: None Triage Status: None ___ Details: This idea seems too obvious to not have been brought up before but I haven't seen it and it seems both helpful and easy to implement so here goes. There's a lot of clever trickery around for inserting special characters in make syntax, e.g. the famous _empty := _space := $(_empty) $(_empty) So why not support a special syntax for predefined variables which uses URL encoding? With this feature, a space would be $(%20) and a backslash $(%5C), etc. The . namespace is already reserved for internal variables, e.g. $(.DEFAULT_GOAL), so the same could be done for the % namespace with a low likelihood of breaking anything. Or, to be perfectly safe, it could be placed in the . namespace as in $(.%20). Just to be clear, what I'm suggesting here would be a simple token replacement which requires no actual use of the symbol table. Whenever the sequence $(%XX) is encountered while doing variable evaluation it's replaced by its character equivalent. The primary use would be for single characters but I see no reason url-encoded strings couldn't be allowed. As long as the variable string starts with % (or .%), the whole thing is replaced by its url-decoded equivalent. ___ Reply to this item at: http://savannah.gnu.org/bugs/?35063 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
On Fri, Dec 16, 2011 at 7:41 AM, Tim Murphy tnmur...@gmail.com wrote: would probably be quite nice and they don't absolutely demand that one use the C library mode flags (w,w+,a etc). This is one thing that Paul didn't like from the previous suggestions. A bit of a sidetrack, but I don't agree with the logic here. The notion that r, w, and a stand for read, write, and append did not originate with the C stdio library. One could make a case against these mnemonics on grounds of English-centrism but realistically English is the lingua franca and GNU make (patsubst, --keep-going) is already thoroughly Anglicized, for better or worse. And + has a truly international mnemonic value. So: I have no particular attitude about which implementation is chosen, but being skeptical of [rwa+] simply because fopen borrowed from the same dictionary makes no sense to me. People familiar with fopen will enjoy the symmetry, others will still see the common sense in it. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make bug with ONESHELL and SHELLFLAGS
I believe the patch below would fix this. It's up to Paul of course
whether he considers it a bug and considers the patch worthy. The only
concern I have with it is that there may be places where the argv is
assumed to be exactly 4 slots wide (shell, flags, recipe, null) and
only the first 3 slots are freed, but I couldn't find any. Also this
over-allocates new_argv a bit in the cause of simplicity.
-David Boyce
Index: job.c
===
RCS file: /sources/make/make/job.c,v
retrieving revision 1.215
diff -u -r1.215 job.c
--- job.c 15 Nov 2011 21:12:54 - 1.215
+++ job.c 22 Dec 2011 21:31:48 -
@@ -2960,12 +2960,21 @@
*t = '\0';
}
- new_argv = xmalloc (4 * sizeof (char *));
- new_argv[0] = xstrdup(shell);
- new_argv[1] = xstrdup(shellflags ? shellflags : );
- new_argv[2] = line;
- new_argv[3] = NULL;
- return new_argv;
+ {
+ char *sfcopy;
+ char *sftok;
+ int n = 0;
+
+ sfcopy = xstrdup(shellflags ? shellflags : );
+ new_argv = xmalloc ((4 + sflags_len/2) * sizeof (char *));
+ new_argv[n++] = xstrdup(shell);
+ for (sftok = strtok(sfcopy, ); sftok; sftok = strtok(NULL, ))
+ new_argv[n++] = xstrdup(sftok);
+ new_argv[n++] = line;
+ new_argv[n] = NULL;
+ free(sfcopy);
+ return new_argv;
+ }
}
new_line = alloca ((shell_len*2) + 1 + sflags_len + 1
On Thu, Dec 22, 2011 at 12:30 PM, michael_rytt...@agilent.com wrote:
Sorry if this has already been reported but I haven’t found anything
searching through the archives.
SHELL = /bin/bash
.SHELLFLAGS = -e –o pipefail -c
Works
.ONESHELL:
SHELL = /bin/bash
.SHELLFLAGS = -e –o pipefail –c
Doesn’t
When the ONESHELL target is set .SHELLFLAGS must be set to a single value,
i.e. –ec.
I am using
Michael Rytting
Agilent Technologies
michael_rytt...@agilent.com
719-590-3708
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35063] Suggest conventional $(%XX) syntax for special characters
Follow-up Comment #4, bug #35063 (project make): I don't agree with this, and I don't know what you mean about altering the basic syntax. The original proposal makes no change to syntax (although I see that I used the word incorrectly in the subject line), it adds a new range of built-in variables. But Paul will make the final call. ___ Reply to this item at: http://savannah.gnu.org/bugs/?35063 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35063] Suggest conventional $(%XX) syntax for special characters
Follow-up Comment #6, bug #35063 (project make): I think the root of our disagreement is in the goals. This is not at all about adding url encode/decode capability to make; it's about a consistent, safe way of getting special characters past the make parser and I suggested url encoding (aka percent-encoding) as a well-known mechanism which is easy to implement. Another encoding would serve just as well. In particular, there's no call for url _encoding_ support as I see it. If you can make a case that there's a reason to support both then maybe your proposal would be better. But that's not the stated goal here. Your question about 'Hello%2C%20world%21' is answered in the original proposal; if the variable reference begins with '$(%' then the string is subjected to standard url decoding, which is well defined to produce Hello, World!. WRT special functions I think the choices are pretty clear. Given that %3A is a colon: $(call %3A) returns : $(value %3A) returns : $(origin %3A) returns default $(flavor %3A) returns simple In essence $(%2C) is defined to behave exactly as if the line %2C := , had been seen (actually not quite, it would be in the list of predefined variables such that --no-builtin-variables would suppress the feature). BTW see http://www.cmcrossroads.com/ask-mr-make/8442-gnu-make-escaping-a-walk-on-the-wild-side for some background on the problem. ___ Reply to this item at: http://savannah.gnu.org/bugs/?35063 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35397] make bug with ONESHELL and SHELLFLAGS
URL:
http://savannah.gnu.org/bugs/?35397
Summary: make bug with ONESHELL and SHELLFLAGS
Project: make
Submitted by: boyski
Submitted on: Sun 29 Jan 2012 08:22:36 PM GMT
Severity: 3 - Normal
Item Group: Bug
Status: None
Privacy: Public
Assigned to: None
Open/Closed: Open
Discussion Lock: Any
Component Version: 3.82
Operating System: Any
Fixed Release: None
Triage Status: None
___
Details:
In a thread from the bug-make mailing list
(http://lists.gnu.org/archive/html/bug-make/2011-12/msg00039.html) a bug was
reported and the following patch was provided and reported to work fine. I 'm
filing this report to ensure the issue gets proper review and disposition
before the upcoming release.
The original report said:
SHELL = /bin/bash
.SHELLFLAGS = -e –o pipefail -c
Works
.ONESHELL:
SHELL = /bin/bash
.SHELLFLAGS = -e –o pipefail –c
Doesn’t.
When the ONESHELL target is set .SHELLFLAGS must be set to a single value,
i.e. –ec.
Index: job.c
===
RCS file: /sources/make/make/job.c,v
retrieving revision 1.215
diff -u -r1.215 job.c
--- job.c 15 Nov 2011 21:12:54 - 1.215
+++ job.c 22 Dec 2011 21:31:48 -
@@ -2960,12 +2960,21 @@
*t = '\0';
}
- new_argv = xmalloc (4 * sizeof (char *));
- new_argv[0] = xstrdup(shell);
- new_argv[1] = xstrdup(shellflags ? shellflags : );
- new_argv[2] = line;
- new_argv[3] = NULL;
- return new_argv;
+ {
+ char *sfcopy;
+ char *sftok;
+ int n = 0;
+
+ sfcopy = xstrdup(shellflags ? shellflags : );
+ new_argv = xmalloc ((4 + sflags_len/2) * sizeof (char *));
+ new_argv[n++] = xstrdup(shell);
+ for (sftok = strtok(sfcopy, ); sftok; sftok = strtok(NULL, ))
+ new_argv[n++] = xstrdup(sftok);
+ new_argv[n++] = line;
+ new_argv[n] = NULL;
+ free(sfcopy);
+ return new_argv;
+ }
}
new_line = alloca ((shell_len*2) + 1 + sflags_len + 1
___
Reply to this item at:
http://savannah.gnu.org/bugs/?35397
___
Message sent via/by Savannah
http://savannah.gnu.org/
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
Thanks, this will make a lot of people happy. One nit: as mentioned earlier I think it would be good to document the relationship of $(file ...) with timestamps. Assuming nothing special is done (I haven't looked at the code) then writing a null string will not update the timestamp. I tried to make a case in favor of doing something special but either way it would be good to get the behavior on the record. David Boyce On Sun, Jan 29, 2012 at 3:02 PM, Paul Smith psm...@gnu.org wrote: On Mon, 2012-01-23 at 09:52 -0800, Lawrence Ibarria wrote: I do like this suggestion, feels quite clean! I implemented the write side of this proposal and committed it to CVS, along with regression tests and documentation. The read side is slightly more work but I can do this one too if people want it. I'm not sure there's much advantage to it over $(shell cat file) as this is unlikely to be a performance bottleneck. Although from a portability and implementation alignment point of view it would be useful. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
On Mon, Jan 30, 2012 at 6:19 PM, Paul Smith psm...@gnu.org wrote: Is there something special you would prefer beyond this? Well ... it's more of a thought than an actual request or preference, but I'm suggesting that make might want to take positive steps in your last case (empty text argument, append to existing file) to force an update of the mod time. Reasons: - Something like $(info touch $@$(file $@)) could become an efficient, portable, in-process pattern for touching a file. - Make is inherently timestamp based and clear rules are important. It could create a new class of mysterious, hard-to-reproduce bugs if this function sometimes updates the timestamp and other times not depending on the value of the text string. - As you've noted, the current semantics just fall out from POSIX behavior. To the degree practical I think it's a good design principle that timestamp semantics are specified by make (and thus portable), not deferred to the platform or filesystem. Again this is not a request per se, just something to think about before semantics are set in stone. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: 'withfile' function implementation
First, I think there may be a bug. Did you intend that appending empty data would always append a newline, such that the file grows by one newline every time? It's consistent with the new documentation but not with what you wrote above (If the file does exist and it's opened for append then the modtime on the file will NOT be updated.). I think it makes more sense to append a trailing newline only when new data was added, and the attached patch implements this (along with the rest) and documents it. On Tue, Jan 31, 2012 at 1:16 AM, Paul Smith psm...@gnu.org wrote: The problem, as I alluded to before, is that there's no way to get the exact semantics of updating a file without actually updating the file. In order to change the modtime on a file without changing the file you have to use utime()... Yes, this is something you and I discussed a while ago in a different context. What I'm thinking of doesn't involve utime and is illustrated by the attached patch. In the special case where $(file) is appending an empty string to an existing file, it reads one byte and writes it back. This gives timestamp semantics identical to an actual write, because it is an actual write. And it's safely atomic because the file has the same contents whether the write succeeds or fails. A proof-of-concept patch is attached along with a demo makefile. -David Boyce Makefile Description: Binary data func_file-dsb.diff Description: Binary data ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35485] New $(.MFDIR) built-in variable
URL: http://savannah.gnu.org/bugs/?35485 Summary: New $(.MFDIR) built-in variable Project: make Submitted by: boyski Submitted on: Wed 08 Feb 2012 06:30:41 PM GMT Severity: 3 - Normal Item Group: Enhancement Status: None Privacy: Public Assigned to: None Open/Closed: Open Discussion Lock: Any Component Version: CVS Operating System: Any Fixed Release: None Triage Status: None ___ Details: The attached one-line (plus docs) patch adds a .MFDIR variable which evaluates to the directory of the containing makefile, more or less. It's actually just shorthand: diff -u -r1.60 default.c --- default.c 16 Jan 2012 02:29:22 - 1.60 +++ default.c 8 Feb 2012 17:45:49 - @@ -527,6 +527,7 @@ #endif #endif /* !VMS */ +.MFDIR, $(abspath $(dir $(lastword $(MAKEFILE_LIST, 0, 0 }; Justification: though this adds no new functionality, I think it's important for support of non-recursive build styles. Given a build tree with many directories containing source files and makefiles, in a recursive model the tendency is to cd into each directory and invoke make locally, and thus the important relative directory is the cwd aka $(CURDIR). But in a flat build model those sub-makefiles tend to be included into a single make process and CURDIR doesn't change. In this case the important relationship is with the directory containing the controlling sub-makefile. Currently the playing field is tilted towards cwd-centric builds by the support for CURDIR. Having .MFDIR would even that out by making it easier to think in terms of a makefile-relative directory. Expecting naive (or even expert) developers to qualify paths with $(abspath $(dir $(lastword $(MAKEFILE_LIST is a nonstarter. This could be made more verbose as MAKEFILE_DIR or less as MWD. The leading . is just my guess as to what would be most compatible. Also, it's arguable whether this is better handled as a regular predefined variable, subject to removal with -R, or a hard-wired variable analogous to CURDIR. ___ File Attachments: --- Date: Wed 08 Feb 2012 06:30:41 PM GMT Name: mfdir.diff Size: 2kB By: boyski http://savannah.gnu.org/bugs/download.php?file_id=25027 ___ Reply to this item at: http://savannah.gnu.org/bugs/?35485 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #35485] New $(.MFDIR) built-in variable
On Thu, Feb 9, 2012 at 5:33 AM, Tim Murphy tnmur...@gmail.com wrote: Hi, I know this is tangential, but: MAKEFILE_LIST itself is a problem for non-recursive builds because it gets very big if you have a lot of include statements (as opposed to a gigantic single makefile file). True, but apparently there's no restriction against modifying it as the makefile below shows, so might it be viable to truncate it at strategic locations as shown? A smaller fix within make would be to normalize all paths in the list relative to CURDIR, which never changes within the make process. Currently pathnames ar recorded as used, which means you can get a mix of absolute and relative and non-canonical names like ../../lib/libfoo/../../cmd/blah.mk. Absolute paths in particular can get quite long. But this is unlikely to happen - have you tried simplifying as below? -David Boyce include MFa include MFb include MFc include MFa include MFb include MFc include MFa include MFb include MFc include MFa include MFb include MFc MAKEFILE_LIST := $(firstword $(MAKEFILE_LIST)) $(lastword $(MAKEFILE_LIST)) include MFa include MFb include MFc $(info MAKEFILE_LIST=$(MAKEFILE_LIST)) all: ; ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35493] Provide access to existing platform type data via a builtin variable
URL: http://savannah.gnu.org/bugs/?35493 Summary: Provide access to existing platform type data via a builtin variable Project: make Submitted by: boyski Submitted on: Thu 09 Feb 2012 05:08:57 PM GMT Severity: 3 - Normal Item Group: Enhancement Status: None Privacy: Public Assigned to: None Open/Closed: Open Discussion Lock: Any Component Version: CVS Operating System: None Fixed Release: None Triage Status: None ___ Details: There are thousands of makefiles containing logic like this uname := $(shell uname -s) # or worse, = ifneq (,$(filter Linux,$(uname))) LINUX_HOST := 1 else ifneq (,$(filter CYGWIN%,$(uname))) CYGWIN_HOST := 1 else ... endif This is complicated, duplicative, hard to read, has portability issues due to relying on a Unix utility, etc. Meanwhile make already contains a variable make_host describing the platform it was built for, and a one-line patch can make it available within makefiles: define_variable_cname (MAKE_HOST_TYPE, make_host, o_default, 0); The attached patch also includes documentation. ___ File Attachments: --- Date: Thu 09 Feb 2012 05:08:57 PM GMT Name: make_host.diff Size: 2kB By: boyski http://savannah.gnu.org/bugs/download.php?file_id=25038 ___ Reply to this item at: http://savannah.gnu.org/bugs/?35493 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35485] New $(.MFDIR) built-in variable
Follow-up Comment #1, bug #35485 (project make): T Murphy says: I think that getting the directory of the current makefile is useful although I am sure my team have had trouble with $(abspath) and the way it works in some situations on windows (unfortunately our stuff has to work on both) - it was something to do with which drive letter got stuck on to a relative path (e.g. off the CWD) to make it absolute and this possibly not always being the drive letter you wanted. This is unsubstantiated since my emails about that are stuck in some old Lotus notes database. To which I say: A. There will be such corner cases. I have a similar one with Cygwin where we must use cygpath -a -m instead of $(abspath) to translate Unix pathnames into Windows style. But with that said, it's no worse having a shorthand. If abspath is broken on Windows you need to hack around it either way, or fix it in make. B. It might be better to use realpath instead of abspath. The differences I know of are that the path must exist and symlinks are resolved. In this case the directory is already known to exist and resolving symlinks might be a feature. C. As noted, it's possible to implement this directly in C code instead which might help work around issues with abspath. D. Of course if abspath itself is really broken WRT drive letters then that should be fixed, but that's outside scope here. ___ Reply to this item at: http://savannah.gnu.org/bugs/?35485 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35485] New $(.MFDIR) built-in variable
Follow-up Comment #3, bug #35485 (project make): I'm not sure how much more specific I can be than saying that it's just shorthand for $(abspath $(dir $(lastword $(MAKEFILE_LIST but let me try to elaborate. First, the variability of .MFDIR would be precisely the same as that of MAKEFILE_LIST so it would be no less useful than that. It's true that someone designing a non-recursive build model which relies on including a tree of sub-makefiles would need to pay careful attention to inclusion sequence but again that's the same as with MAKEFILE_LIST for obvious reasons. I believe this limitation is made clear in the original doc patch but there would be no harm in repeating the same warning that comes with MAKEFILE_LIST. So, though I disagree with the statement this can't possibly be what you want, I agree it could be even more useful if .MFDIR lived on a stack such that it always resolved to the name of the file within which it was evaluated. But as you (Paul) said recently in a different context in make everything is a time-space tradeoff, so here I was going for the simplest, least invasive solution. If you prefer the stack-based approach I could look into it. Bottom line, MAKEFILE_LIST is quite important for figuring out the current makefile in flat/included build models but using it is somewhat obfuscated, and I'm trying to find a way to make it more user friendly. ___ Reply to this item at: http://savannah.gnu.org/bugs/?35485 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35485] New $(.MFDIR) built-in variable
Follow-up Comment #5, bug #35485 (project make): I don't think the stack-based approach is actually more runtime or space intensive ... I don't think so either. I just meant it would be a more intrusive change in terms of diff size than the one-liner proposed here. But if you're that it's not, this is good news. Although, I am not sure what I would like will be what you're looking for, because what I'd like to see is a variable containing the currently parsing makefile. I don't think we disagree that much. First, I don't care so much about whether the path is canonicalized, though I do kind of prefer it aesthetically. Second, it's true that a variable which evaluates to the currently parsing makefile would go a significant way toward addressing my concern. In other words I could live with it. That said, I don't think you've addressed the core of my point which is basically pedagogical. I think the directory of the current makefile is a fundamental concept, just as important to single-DAG systems as the current working directory is to process-recursive build models, and deserves to be treated that way. Yes, MFDIR could be derived easily from the current makefile, but that still leaves it a second class citizen. As noted, though, I could certainly live with the compromise. I can live with the current state of things too, for that matter, I just think it's a bit sub-optimal. ___ Reply to this item at: http://savannah.gnu.org/bugs/?35485 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Patch to allow make to load plugins that add new functions.
A few years ago I suggested a plugin architecture much like this (but I didn't supply a patch - crucial difference), to allow a plugin to make the up-to-date determination, replacing the hardwired timestamp system. That could in theory be quite useful as it would allow an MD5 or similar signature to be used without major mods to make internals which is what's defeated such attempts in the past. Any idea how hard it would be to handle this with a plugin? On an unrelated note, how would you anticipate plugins being loaded in real life? I think expecting people tp type make --plugin=tests/plugins/test-plugin.so ... is too unwieldy for daily use. Environment variable? Auto-discovery of plugins in a conventional location? Is it possible a .PLUGIN special target could be embedded in a makefile that requires a given plugin, or would that be too late? Maybe a MAKE_PLUGIN_PATH env var? -David Boyce On Thu, Apr 5, 2012 at 1:12 PM, Tim Murphy tnmur...@gmail.com wrote: Hi, On 5 April 2012 20:52, Lawrence Ibarria libar...@nvidia.com wrote: I like this idea quite a bit. I see this as still work in progress to define what type of functions the plugins can have. Maybe they can even create or change make variables. At this point, plugins are naughty and are kind of able to access anything in make that they have the prototype for (using -rdynamic to make all the symbols available). This is definitely dodgy since there's nothing stopping a plugin from trying to access something that might be intended as fairly volatile within make. In the case of variable_buffer_output, I'd suggest that each plugin has an 'initialization function'. This is a handshake where make provides the plugin with function pointers (avoids the extern function) and the plugin can provide its name, versions, and the function table. That simplifies the code that right now tries to get every single variable from the dl_handle. -- Lawrence Ok, good idea - I'll add something to look for an initialiser and call it and as you say, pass it a table. Returning values is critical and I think that looking up variable values at the least is also probably important. Another thing I've wanted for ages is to be able to know if some target was declared already - I want plugins to have the ability to look at make's target database if possible although not necessarily modify it. Regards, Tim -Original Message- From: bug-make-bounces+libarria=nvidia@gnu.org [mailto:bug-make-bounces+libarria=nvidia@gnu.org] On Behalf Of Tim Murphy Sent: Thursday, April 05, 2012 2:51 AM To: bug-make@gnu.org Subject: Patch to allow make to load plugins that add new functions. Hi, I am between jobs which made me realise that I am absolutely free to contribute to make for about 10 days :-) The one thing I have wanted the most and the longest is a way to add new functions without having to rebuild and look after a custom version of make. Essentially this should allow people to extend make without necessarily having to maintain a custom version of it. So here is an attempt at a plugin system for your interest. Currently it ought to work on systems with dlopen. LoadLibrary on windows should do the job admirably but I haven't had time to have a go at that yet. So the feature is optional (configure --with-plugins). I am looking for feedback indicating interest or disagreement or whatever and mostly I just want to put some code into the open so that it's there. My test plugin adds the $(equal ($X),$(Y)) function which returns $(X) if the variables X and Y match as strings and returns the empty string if not. I have often wanted this for use with $(if) and have a horrible and inefficient macro for doing it now but it's something I feel really ought to be built in. Here's how you tell make to use a plugin: ./make --plugin=tests/plugins/test-plugin.so -f tests/plugins/plugintest1.mk Here's the code for this simple test plugin: #include plugin.h #include string.h /* The next line is a cheat to save from having to include all the make headers and possibly end up linking to all sorts of make object files which defeats the purpose. Don't like it and will have to think of something better: */ extern char *variable_buffer_output (char *ptr, const char *string, unsigned int length); int gnumake_plugin_api_version = 1; /* what api the plugin expects make to provide - just a way for make to know if it's loading a plugin that should work */ int gnumake_plugin_major_version = 1; /* The version of this plugin - might be useful for problem reports */ char gnumake_plugin_name[] = test_plugin; /* also the name of the feature added to .FEATURES */ /* $(equal astring1,astring2) returns blank, $(equal astring1,astring1) returns astring1 * The purpose of this function is to allow equality tests in $(if) functions e.g. $(if $(equal $(X),$(Y)),something,else) * */ static char
Re: Patch to allow make to load plugins that add new functions.
On Fri, Apr 6, 2012 at 2:59 AM, Tim Murphy tnmur...@gmail.com wrote: I was thinking of marking this feature as experimental in the first release (in the documentation), just to be more clear on expectations. Very very much so - there are many platforms to support anyhow and when someone eventually tries plugins on them we might see new problems so better to let it take time to mature. Apropos of this, the Apache Portable Runtime has a nice abstraction over dlopen and LoadLibrary. Assuming they're license-compatible, it might be worth looking into borrowing that logic for this feature. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Patch to allow make to load plugins that add new functions.
On Fri, Apr 6, 2012 at 12:40 PM, Eli Zaretskii e...@gnu.org wrote: Date: Fri, 6 Apr 2012 12:08:33 -0400 From: David Boyce david.s.bo...@gmail.com Cc: bug-make@gnu.org bug-make@gnu.org Apropos of this, the Apache Portable Runtime has a nice abstraction over dlopen and LoadLibrary. Is it significantly better than what libltdl provides? The advantage of the latter is that it's readily available for both Posix and MinGW builds. Sorry, I've never used libltdl. Maybe it would have been better just to say libraries exist to paper over the differences between various platforms dynamic linking APIs; consider not reinventing the wheel. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Patch to allow make to load plugins that add new functions.
On Fri, Apr 6, 2012 at 5:36 PM, Tim Murphy tnmur...@gmail.com wrote: I had a little look at libtdl. To be brutal I thought that using dlopen/LoadLibrary directly was *much* easier. There isn't really anything madly complicated about what's being done. To clarify: when I originally spoke about APR, I was not suggesting that GNU make actually incorporate and use APR but just that useful bits of library-loading code could be stolen from it. And I assume the same could go for libltdl. It might be worth at least looking at these just to make sure you don't paint yourself into a corner API-wise vis-a-vis HPUX or OS X or whatever. David B ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Feature request: silently overriding existing rules
Even with GNU make as it stands, couldn't you emit your rules in the form of variables, override them at will, and eval() them at the end? E.g. define ruleA version 1 endef then later... define ruleA version 2 endef $(eval $(call ruleA,...)) -David Boyce On Fri, Aug 10, 2012 at 12:36 PM, Stefano Lattarini stefano.lattar...@gmail.com wrote: In some situations, it would be very useful to be able to override already-defined rules for a target without having GNU make complaining about the override. For example, when writing a library of makefiles recipes, organized as a set of makefile fragments to be included by a master (user-written) makefile, it is useful to be able to seamlessly override some of the recipes defined in those fragments from the master makefile. My Automake fork Automake-NG (whose generated Makefiles only target GNU make) could certainly take great advantage of such an ability in the future (near future in fact). I'm not sure what be the best way to implement my proposed feature. Some random ideas: - an all-encompassing option --no-warn-rule-override, that would suppress *all* warnings about overridden rules; - a new special target .OVERRIDABLE-RECIPES:, that would suppress the override warnings only for the rules specified as its prerequisites; - a new built-in to explicitly instruct make to clear the rule(s) so far associated to a target: $(clear-rules TARGET) This approach has the advantage of being very explicit, and to allow clearing of not only for normal rules, but also double-colon ones. - a new syntax to define special kind of rule for a given target -- a rule which can be silently overridden later; e.g., TARGET ?: commands that will be overridden ... TARGET : actual commands Note that I include this latter approach only for completeness; I rather dislike it, because it doesn't give enough power to the user -- the writer of the original makefile must decide statically and beforehand which rules to make overridable. Thoughts, objections, feedback? Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Feature request: silently overriding existing rules
On Fri, Aug 10, 2012 at 2:38 PM, Stefano Lattarini stefano.lattar...@gmail.com wrote: I have no answer for that, lacking any knowledge about GNU make internals; I guess the make developers here will be in a better position to answer my question. Yes, and I hope you get your feature. But consider that auto-tools are traditionally targeted at the lowest common denominator. You've made an explicit exception for Automake-NG that it will require GNU make, which is reasonable. But do you really want to require a not-yet-even-released version? That might not become generally available for a decade or so, depending how portable you want to be. It seems to me that targeting 3.81 or so would be better. IMHO. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #33138] .PARLLELSYNC enhancement with patch
Follow-up Comment #4, bug #33138 (project make): This is the original author. I've become very busy at my day job in the last year or two so I've lost track of this. Thanks for picking it up and improving it. I haven't had time to look at your new patch yet, and not sure when I will, but here are a few responses to your comments: Target vs flag I have to agree that a flag is more correct. It's more convenient to use .PARALLELSYNC because getting users to use a flag can be difficult (in fact the major insight of Feldman's original make was that a naive user should be able to just type make and the right thing will happen), but a flag is more flexible. To the best of my memory (not so good) the reasons I went with a special target are: 1. Given the clever --eval flag Paul added in 3.82 any special target can be used as a flag, e.g. --eval .PARALLELSYNC: (BTW I tried to argue that --eval was worthy of a scarce single-letter alias (-E) on the grounds that it was a flag-to-end-all-flags but I don't think that happened). 2. Paul has some concerns about make becoming like GNU tar which apparently has too many options. Personally I'm always willing to trade a few more minutes of RTFM for more capability, and have no problem with GNU tar, but tastes differ. Bottom line, I'm more or less agnostic on flag-vs-target. tmpfile vs mkstemp IIRC the reason I used tmpfile() instead of mkstemp() is that mkstemp doesn't unlink the file automatically, which leads to a number of risk factors such as filling up /tmp and so on. Possibly the best/most portable approach would be to refactor the existing open_tmpfile() function, which returns a FILE *, into an open_tmpfd() which returns a descriptor wrapped by open_tmpfile() which just converts the descriptor into a FILE * using fdopen(). This is already what happens, it's just a matter of splitting existing logic into two functions. Now open_tmpfd() is logically just mkstemp-with-porting-hacks and we could enhance it with an unlink option for this feature. make[1]: Leaving directory 'bar' make[1]: Leaving directory 'bar' The problem with doubling these is that it becomes non-deterministic. What if there was a directory structure foo/bar/bar/baz? Additionally, would it be possible to implement the enhancement proposed in the email discussion? See http://lists.gnu.org/archive/html/bug-make/2011-04/msg00041.html for context? Or was that already done? I don't remember. Last, Paul has been pretty clear lately that new patches should be accompanied by associated regression tests. That might be the remaining hurdle. ___ Reply to this item at: http://savannah.gnu.org/bugs/?33138 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #33138] .PARLLELSYNC enhancement with patch
Follow-up Comment #7, bug #33138 (project make): I agree this seems useful. However, since it would involve changes to existing code which are not strictly needed... Not functional changes, just a little refactoring, and adding a new reliance on tmpfile() may cause more stress to ports than enhancing the existing use of mkstemp(). But I agree, this patch has reached the point where it needs an up-or-down vote from Paul et al before any such final finish work is done. David ___ Reply to this item at: http://savannah.gnu.org/bugs/?33138 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make goes into an infinite loop
Ian, You don't mention a platform so I will assume Linux. GNU make 3.81 dates from 2006, and 3.82 was released in 2010. That's a lot of bugfixes, so the first thing is to try at least 3.82 and ideally the CVS version which has ~3 more years of fixes. Building the CVS versions is not hard if you have a fairly modern system which either has or allows you to add a couple of required packages (see the README.cvs file). I suggest turning off any -O flags and adding -g; not only does this improve debugging and stack dumps but there's a class of subtle bug which can be triggered by optimization. If you still see the problem with the CVS version (which identifies as 3.82.90 under --version) then you could try looking at what the process is doing. The way I often start in situations like this is to echo $$ in the window where you plan to run make, to get what will be its parent pid. Then run make, and in another window run something like pstree -a -p -l pid with that parent pid you collected. This should clarify the process tree situation; either make will be stuck internally (no new children) or will be waiting on some subprocess which doesn't return. Though in this case it seems fairly clear there are no new processes. Assuming the hang is within make, you could try forcing a core dump. This can be done with kill -BUS (or there may be a better way, Linux is not my native country). Something else to try first would be strace -p make-pid. This may generate a lot of data but there may be a cycle which provides a clue. Or make may just be sleeping, waiting for some resource, in which case strace should help figure out what that resource is. These may provide useful clues. However, based on the data you provided I'd bet on a bug in make and your best hope is that it's been fixed in the last ~7 years. David Boyce On Mon, Jan 7, 2013 at 10:17 AM, Ian Lynagh ig...@earth.li wrote: Hi all, I've got a makefile for a large project which appears to send make into an infinite loop. I wonder if anyone could help me work out what is going wrong, and how to fix it, please? Unfortunately, it does so after about 30mins of compiling with parallel make (using e.g. -j5), and making a small testcase is unlikely to be easy. I've run make with the -d flag, and put the part of the log around where it starts to go wrong here: http://lambda.haskell.org/~igloo/make-loop.txt (14MB). My understand is: * around line 1, there are 2 child processes building things, one of which finishes * around line 2607, there is 1 child process building something, and another child is started * around line 18477 there are 2 child processes building things, one of which finishes * around line 36876 there is 1 process building something, and it finishes * on line 55282 an iteration of the infinite loop starts * on line 73678 the next iteration of the infinite loop starts * on line 92074 the next iteration of the infinite loop starts * on line 110470 the next iteration of the infinite loop starts * on line 128866 the next iteration of the infinite loop starts The end of one loop and start of the next looks like this: The prerequisites of `libraries/dist-haddock/index.html' are being made. Finished prerequisites of target file `all_library_doc_index'. The prerequisites of `all_library_doc_index' are being made. Finished prerequisites of target file `all'. The prerequisites of `all' are being made. Considering target file `all'. File `all' does not exist. Considering target file `check_packages'. File `check_packages' was considered already. Considering target file `all_utils/mkdirhier'. Note that there are no Live child lines or anything. If I ^C it, and run make again, then compilation continues and succeeds. Is this a bug in make, or in my makefiles? If the former, is there a workaround? If the latter, have you got any advice on how to diagnose it please? I have GNU Make 3.81. Thanks Ian ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make goes into an infinite loop
On Tue, Jan 8, 2013 at 8:37 AM, Ian Lynagh ig...@earth.li wrote: Well, ideally I'd like to find a solution that doesn't require all the users to install a newer make. But I'll do some more diagnosis and see what I find. Yes. If it's a fixed bug then a little binary searching or a walk through the Changes file should find the details of the specific bug which may with luck lead to a workaround. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [Question] How can i stop build immediately in build fail?
It may be a reasonable feature request but it also may not be that
hard to implement with existing functionality. Here's one approach:
% cat makefile
SHELL := $(abspath shx)
.PHONY: job
job: job1 job2
.PHONY: job1
job1:
sleep 3
exit 1
.PHONY: job2
job2:
while echo ok; do sleep 1; done
% cat shx
#!/usr/bin/env python
import optparse
import os
import subprocess
import sys
if '__main__' == __name__:
parser = optparse.OptionParser()
parser.add_option('-c', '--command', type='string',
help='Command to execute')
(opts, args) = parser.parse_args(sys.argv[1:])
rc = subprocess.call(opts.command, shell=True)
if rc != 0:
print 'stopping make!'
os.killpg(0, 15)
sys.exit(rc)
% make -j2
sleep 3
while echo ok; do sleep 1; done
ok
ok
ok
ok
exit 1
stopping make!
make: *** [job2] Terminated
make: *** [job1] Terminated
Terminated
I implemented the replacement shell in Python but it could use any
language. You just need to kill the process group once the first
recipe fails.
-David Boyce
On Fri, Jan 11, 2013 at 9:38 AM, Oleksandr Gavenko gaven...@gmail.com wrote:
On 2013-01-10, jungsoo.son wrote:
I always run 'make' with -j8. In this case, when there are a fail it is too
hard to check the fail.
I want to kill the all sub-make process immediately when the error occurred.
How can i stop build immediately in build fail?
I make such example:
.PHONY: job
job: job1 job2
.PHONY: job1
job1:
sleep 3
exit 1
.PHONY: job2
job2:
while echo ok; do sleep 1; done
Ir you run like:
$ make -j2
you get:
sleep 3
while echo ok; do sleep 1; done
ok
ok
ok
exit 1
make: *** [job1] Error 1
make: *** Waiting for unfinished jobs
ok
ok
...
Seems to be reasonable have an option to kill jobs after some timeout if one
job fail (or make receive signal) and enabling .DELETE_ON_ERROR in this
case...
--
Best regards!
___
Help-make mailing list
help-m...@gnu.org
https://lists.gnu.org/mailman/listinfo/help-make
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: tasklist item still relevant?
I don't know the answer except to point out that someone, not sure who, has worked out how to solve this problem within the existing capabilities of GNU make. I believe this clever technique is now used in the Linux kernel makefiles and a number of other places. This presumably lessens the pressure to implement it with new GNU make syntax and makes the new feature less likely to happen. However, it would still be best if GNU make had (optional) native support. David Boyce On Wed, Jan 23, 2013 at 10:01 PM, toby cabot t...@caboteria.org wrote: Hi Folks, Greetings from the GNU Volunteer Coordinators[0]. There's an item in the GNU task list[1] that reads Add features to GNU Make to record the precise rule with which each file was last recompiled; then recompile any file if its rule in the makefile has changed. This task has been in the task list for longer than I've been involved with GNU (i.e., more than a decade), is it still needed? Has someone accomplished this yet? Do you still think it's something that you want to do? I'd be happy to keep it; I just want to check if it's still relevant. Thanks, Toby [0] http://www.gnu.org/software/tasklist/howto-volunteer.html [1] https://savannah.gnu.org/people/viewjob.php?group_id=3972job_id=200 ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #29074] -include target fails to issue Error in 3.81
The INSTALL document is for people who just download the standard source tarball. I suspect there's a README.git for people doing a git clone; at least there used to be a README.cvs for CVS users. -David Boyce On Fri, Feb 15, 2013 at 4:15 PM, dave kerns invalid.nore...@gnu.org wrote: Follow-up Comment #9, bug #29074 (project make): I thought I'd take a look at the source, so I did a git clone, the INSTALL doc says to do the standard ./configure; make ... there is no file configure, there is a configure.bat, what am I missing? ___ Reply to this item at: http://savannah.gnu.org/bugs/?29074 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #38433] Example for eval in documentation contains error with define
On Wed, Feb 27, 2013 at 5:02 PM, Philip Guenther guent...@gmail.com wrote: Side note: I should make clear that I am not committer to GNU make and do not speak for the project. I'm just a contributer to the lists, answering questions where I can. Likewise. IMO, the suggestion that was proposed would reduce the overall usability of the manual and increase the confusion. I think you and the others in the nay camp may be being a bit unfair. As far as I can see, nobody has proposed (in this thread) that the entire manual be reworked to note the version in which each feature appeared. You're absolutely right that that would stink but it's a straw man. The proposal, as I understand it, is to add a note about this particular incompatibility because of the mysterious, silent way it fails. I think any of us who've been reading this list for a long time can attest that it's one of the more common problems that people stumble over. It's a golden oldie, so I don't know why it would be so bad to add a sentence for it. There may even be a couple of other cases that could get similar treatment. It doesn't have to become a slippery slope. If the GNU website were to require you to select the version of make you wanted to see the documentation for, I think that would be a reasonable 'solution'. Perhaps a layout like http://gcc.gnu.org/onlinedocs/ could be done without too much complexity. Another nice example is the Python docs ( http://docs.python.org/2/index.html). Notice the dropdown at the top left where you get to pick the version you care about. David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Wed, Apr 17, 2013 at 1:00 PM, Eli Zaretskii e...@gnu.org wrote:
I believe the thinking is that some implementations may have a much
smaller number of open streams (FILE*) allowed, than open file
descriptors. The POSIX standard, for example, allows this:
Some implementations may limit {STREAM_MAX} to 20 but allow {OPEN_MAX}
to be considerably larger.
I'd be surprised if this were a real problem nowadays. E.g., the
Windows C runtime is documented to allow up to 512 FILE streams, which
can be enlarged to 2048 by calling a function. The max number of file
descriptors is also 2048.
That's not the way it is on Unix. My understanding, and I believe this
applies to all Unix variants, is that because the original stdio FILE
structure used an 8-bit int to hold the file descriptor, the number of
available streams is 256 always and forever (in 32-bit variants - 64-bit
builds fix lots of limits). I don't know of any Unix variants which don't
allow at least 1024 descriptors, and some allow the limit to be raised
dynamically without bound, but the limit on descriptors *associated with
streams* is fixed at 256.
I believe this was all documented in comments in the original patch I sent
in.
-David Boyce
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Thu, Apr 18, 2013 at 10:09 AM, Frank Heckenbach f.heckenb...@fh-soft.dewrote: FILE is an opaque structure which should never be allocated by user code, so its layout can be implementation specific. Of course it's an opaque structure. The problem is that the implementation can't change its size without breaking binary compatibility with every previously-compiled program that uses stdio. This is a well-known problem, easily googled for. Here are a couple of references for Solaris: https://groups.google.com/forum/?fromgroups=#!topic/comp.unix.solaris/yky1fsJ3PMI http://www.oracle.com/technetwork/server-storage/solaris10/stdio-256-136698.html In particular, on Linux the field is an int ... Linux is a pretty elastic term, covering both 32- and 64-bit variants plus many kernels and distros. All 64-bit builds of any system will expand the original unsigned char; the opportunity to break binary compatibility allows for a lot of fixes. For all I know Linux even offers a clever fix in 32-bit versions, but there remain plenty of other 32-bit platforms with the problem. Bottom line, in 32-bit code you have to program for the likelihood that no more than 256 streams may be open at once. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Thu, Apr 18, 2013 at 2:39 PM, Frank Heckenbach f.heckenb...@fh-soft.dewrote This might be right (I was just objecting to your claim that it was necessarily so on any 32-bit Unix), and I'd prefer to use fd's anyway. Well ... Linux is not Unix, as partisans will proudly tell you, so technically what I said is unobjectionable. I won't pretend that that's what I meant though. Presumably the real distinction is between statically and dynamically linked binaries; anything linked statically with libc would need to contain an actual FILE struct, not just a pointer to one. But that aside, we're all in violent agreement so we can let this sub-thread drop. -David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
Since you asked basic questions I'm going to start this at a basic level. Apologies if it covers some stuff you already know or if I misinterpreted the questions. Note that I haven't actually looked at the patch that went in so this is generally wrt the original. The first thing is get the word lock out of your mind because we aren't really locking anything. Yes, that API is in use but it's only to create a semaphore or baton. Nobody is ever prevented from doing anything. It just happens that on Unix the most portable (i.e. oldest) way of implementing a semaphore is with the advisory locking API. All cooperating processes agree not to proceed unless and until they are able to acquire the exclusive lock on a shared file descriptor, but it's not necessary to ever actually write anything to that descriptor. Second, the original implementation (not sure if I ever sent that one one in though) actually created a temp file to use as the semaphore fd. But then I discovered that stdout can be locked in the same way, which is simpler. But applying the lock to stdout is just a frill; it could be a temp file, especially if some platform turned out to need it that way. I just figured that stdout is always available, or at least if it's closed you don't have to worry about synchronizing output. Third, yes, nothing is locked while the child runs. If a shared resource was locked during child runs it would have the effect of re-serializing the build as each supposedly parallel child waited on the lock. So what happens here is really very simple: each child (aka recipe) runs asynchronously, assuming -j of course, and dumps its output to one or two temp files. Only when the child has finished and wants to report results does it enter the queue waiting for the baton. When it gets it, it holds it just long enough to copy its output from the temp files to stdout/stderr and then lets the next guy have his turn. Thus, assuming the average job runs for a significant amount of time (multiples of a write() system call anyway) there will not be much contention on the semaphore and it won't be a bottleneck. You're right that simply writing to temp files and dumping everything at once when the job finished would be likely to reduce the incidence of garbling even without the semaphore, but not to zero. It may be that the locking of stdout is only useful on Unix due to the fact that it's inherited into child processes. I don't know what Paul or Frank is thinking, and as mentioned I haven't looked at the current version, but my thinking originally was that Windows could easily handle this using its own far richer set of semaphore/locking APIs. I'd actually expect this to be easier and more natural on Windows than Unix. All that's required is to choose a semaphore to synchronize on, dump output to temp files, and copy it to stdout/stderr only after acquiring the semaphore. And remove the temp files of course. -David Boyce On Tue, Apr 23, 2013 at 10:50 AM, Eli Zaretskii e...@gnu.org wrote: Date: Fri, 19 Apr 2013 11:54:05 +0200 Cc: bo...@kolpackov.net, bug-make@gnu.org From: Frank Heckenbach f.heckenb...@fh-soft.de Eli Zaretskii wrote: Initial investigation indicates that tmpfile should do the job just fine: the file is deleted only when the last descriptor for it is closed. That includes any duplicated descriptors. Great. As for fcntl, F_SETLKW, and F_GETFD, they will need to be emulated. In particular, it looks like LockFileEx with LOCKFILE_EXCLUSIVE_LOCK flag set and LOCKFILE_FAIL_IMMEDIATELY flag cleared should do the job. I will need to see how it works in reality, though. OK. Upon a second look, I'm not sure I understand how this feature works, exactly, and why you-all thought making it work on Windows is a matter of a few functions. I sincerely hope I'm missing something, please bear with me. First, most of the meat of OUTPUT_SYNC code, which sets up the stage when running child jobs, is in a branch that isn't compiled on Windows (#if !defined(__MSDOS__) !defined(_AMIGA) !defined(WINDOWS32) on line 1482 of job.c). So currently that part is not even run on Windows. Please tell me that nothing in this feature relies on 'fork', with its copying of handles and other data structures. Because if it does, we have no hope of making it work on Windows, at least not using the same algorithms as on Unix. More importantly, how exactly locking the (redirected) stdout/stderr of the child is supposed to cause synchronization, and why do we need it at all? Isn't synchronization already achieved by redirecting child's output to a file, and only dumping it to screen when the child exits? What does lock add to this? Who else will be writing what to where, that we want to prevent by holding the lock/semaphore? In an old thread, Paul explained something similar: David, can you explain why you needed to lock the files? Also, what region(s) of the file you
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Tue, Apr 23, 2013 at 12:04 PM, Paul Smith psm...@gnu.org wrote: I'm not sure if the lock locks the FD (so that if you dup'd the FD but it still pointed to the same output, you could take exclusive locks on both), or if it locks the underlying resource. I'm pretty sure it's the underlying resource that gets locked. The man page is not quite explicit but implicitly fairly clear. The opening paragraph says: Apply, test or remove a POSIX lock on a section of an open file. The file is specified by fd, a file descriptor open for writing David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Wed, Apr 24, 2013 at 10:26 AM, Tim Murphy tnmur...@gmail.com wrote: I would suggest pretending that one has semaphores first with some nice little abstraction and then implementing them in the best way the platform has to offer. How about we introduce functions called acquire_semaphore() and release_semaphore()? Oh, wait, we did. Platforms without semaphores might use a file and locking or if that doesn't work then they can't implement the feature and you configure it out. This kind of (re)invention is not known to be needed at this time. I have the impression Eli has the Windows port in hand, he's just skeptical about a few details. And what's there now is already the LCD for most other platforms as far as I know. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Sun, Apr 28, 2013 at 1:34 AM, Paul Smith psm...@gnu.org wrote:
I'm not excited about that term (job); it's kind of accurate, but in
the documentation for example we're really mushy about exactly what a
job is, vs. a recipe or a command line etc. I'd like to pick some
terms for this, define them in a solid way, then clean up the
references. It would be best to do this before the release to avoid
changing things later.
For example, we currently use target as the name; maybe recipe is
better?
If anyone has opinions I'm listening.
This seems quite wrong to me. I think it's inherent from the existence
of the --jobs option that job must be synonymous with recipe, even
if that's not what was intended at the time. Therefore my HO is that
not only does this one need a new name but -O target would be better
renamed -O job.
What's wrong with line for the line-by-line case? It's exactly the
way make is documented, that each line of the recipe is fed to the
shell in turn, and the documentation you just wrote for the new
argument is all about lines; I see the word line 4 times in 2
sentences. The whole thing seems clear and well written to me, but
would be even more clear if the new option was called -O line.
So I'd argue for:
-O line (new)
-O job (current -O target)
-O make
And document the notion that job and recipe are equivalent terms
(more precisely, one is the instantiation of the other: job is to
recipe what process is to program).
I also think it wouldn't hurt for the documentation of the default to
contain a forward reference. Pending potential rename:
--- a/doc/make.texi
+++ b/doc/make.texi
@@ -4136,7 +4136,7 @@ specified by giving an argument to the option
(e.g., @samp{-Ojob} or
@table @code
@item none
The is the default: all output is sent directly as it is generated and
-no synchronization is performed.
+no synchronization is performed. It is identical to @samp{--output-sync=target}
@item job
Output from each individual line of the recipe is grouped and printed
David
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: dynamic object searching (was: Re: Dynamic objects)
On Mon, Apr 29, 2013 at 4:34 PM, Paul Smith psm...@gnu.org wrote: Plus on UNIX any extension is acceptable since we're using dlopen() (even with the normal linker you can give any library name you want, it's only the -l flag that makes assumptions). Maybe someone wants to write pattern rules to build their GNU make loadable objects with a suffix .gmkso to distinguish it (and use a different rule) from building normal .so shared objects. I want to be sure the benefits outweight the loss of flexibility before we go down that path. Why not try opening the pathname as given, and if that fails append the platform-standard extension and try again? It's probably a good idea to have make predefine a variable containing the host architecture, to avoid the need for uname. We currently have an internal variable make_host which is the GNU autoconf --host value on systems where configure runs, and the various config.h templates have hardcoded values. Maybe we could do something with this (just using the --host value might be too arbitrary, I'd have to look at the options). I've twice submitted a patch to provide make_host as a make variable, but they've gone into the ether. It's a one-line patch and hugely useful IMHO considering the number of people who've had to reinvent the $(if $(shell uname)) etc wheel. It's impossible to determine how a given user or community will define platform, so no variable will work for everybody, but I think make_host is pretty hard to improve upon. -David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: dynamic object searching (was: Re: Dynamic objects)
On Mon, Apr 29, 2013 at 7:53 PM, Paul Smith psm...@gnu.org wrote: Well, David, when you suggested it I wasn't so sure. But now that I've thought of it myself... brilliant!! :-p :-) But now I'm having second thoughts ... -David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: dynamic object searching (was: Re: Dynamic objects)
On Tue, Apr 30, 2013 at 3:57 AM, Tim Murphy tnmur...@gmail.com wrote: Since you can't (in my recent experience) load a 64-bit DLL into a 32-bit program, the real issue is what architecture was make itself built with. It's sort of a matter of make identifying itself rather than telling you that you're running on ubuntu 13.04 or solaris 10. Which is exactly what make_host tells you. It provides the architecture make was built for in the form of e.g. x86_64-unknown-linux-gnu or whatever configure comes up with. On Mon, Apr 29, 2013 at 8:03 PM, David Boyce david.s.bo...@gmail.com wrote: On Mon, Apr 29, 2013 at 7:53 PM, Paul Smith psm...@gnu.org wrote: Well, David, when you suggested it I wasn't so sure. But now that I've thought of it myself... brilliant!! :-p :-) But now I'm having second thoughts ... That was, I hope it's clear, a feeble joke. I find Paul's insight to be a stroke of pure genius. -David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse
On Sun, May 5, 2013 at 2:49 PM, Stefano Lattarini stefano.lattar...@gmail.com wrote: On 05/05/2013 05:30 PM, Paul Smith wrote: On Sun, 2013-05-05 at 11:11 +0200, Stefano Lattarini wrote: Sorry to add this only now, but I realized the failure is only reproducible if I run the testsuite with make -j, as in make -j8 check; and even in that case, the failure is racy. With a bare make check, things work for me as well. On the other hand, increasing the parallelism even more, other tests start to fail as well: The test suite definitely cannot be run in parallel. I realize that (and it's no issue, considering how fast the make testsuite is); but I often run a make -j8 check just after running ./configure (requires less typing than make -j8 make check), and that's where things broke down. Remember that GNU make has a .NOTPARALLEL special target to suppress parallelism. Maybe a makefile in the test suite would benefit from one of those? -David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: *** INTERNAL: readdir: Invalid argument error
On Wed, May 15, 2013 at 7:54 AM, Eli Zaretskii e...@gnu.org wrote: because of the need to convert the file names from UTF-16 to something the rest of the code can grok. Well, if I understand correctly there's no need to actually grok the UTF-16 filenames, just to get past them with a more graceful failure condition, and I was hoping these other APIs would fail differently. But if you say there's no joy here I believe it. In any case, there's no reason to use any of this just to check whether a file by a specific name exists in a directory. Are you saying you'd just walk along PATH doing a stat() or access() or equivalent on dir/sh.exe? David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
David.S.Boyce
http://256.gr/whzo/tmil.bsdiajld ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] Use spawn() in GNU Make on Cygwin, updated
Slight change of subject ... All this arguing has been about the value of spawn as a speed enhancement: Pro: spawn is way faster! Con: but it may destabilize things - we'll stick with slow-but-stable fork. Which ignores the fact that Cygwin fork is *not* that stable. In our experience make -jN results in frequent failed builds due to fork failures. Thus it seems to me that Pro: spawn doesn't suffer from the documented problems which result from trying to emulate fork on Windows. should be part of the conversation too. -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] Use spawn() in GNU Make on Cygwin, updated
On Mon, Sep 2, 2013 at 4:19 PM, Chris Faylor wrote: *There is one know issue with fork where dlls in forked processes don't load at the right address and cause cygwin errors. That is usually fixed by running the autorebase program. And it is that issue to which I refer. One of my co-workers claims to have an STC which, using make -j, can reliably reproduce fork failures even in 1.7.24 (or at least some very recent version employing autorebase). Possibly he's wrong, or possibly it's BLODA, but even if it's BLODA a spawn-based version would likely not suffer from the same problem. If your claim is that fork should be expected to work 100% of the time in a fully rebased install (which is what I thought and what I told him when this first came up) I'll be happy to try and get ahold of the test case and send it in. And if a command-line flag has been agreed upon as Eli says, that's good enough for me. In fact any answer is good enough for me as long as the resulting make.exe works reliably with -j at large scale factors. David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #39943] Add an alternative parsing mode that regards space and tab as identical tokens
On Wed, Sep 4, 2013 at 4:42 PM, Paul D. Smith invalid.nore...@gnu.orgwrote: Follow-up Comment #1, bug #39943 (project make): IMO _any_ editor which automatically replaces TABs with spaces should never be considered to be a useful programming tool. But YMMV of course. You know what they say about standards and how great it is that there's so many? The Python coding standard calls for no use of tabs and thus many people who do Python coding have their editors configured to turn tabs into spaces. I do, for example. Of course it's usually possible to configure the editor to use different profiles with different languages but that gets harder since makefiles have no standard extension. Anyway, the fault doesn't necessarily lie with the editor per se[*]. [*] To be precise, the fault lies with Stu Feldman and his unwillingness to break an installed base of around a dozen users :-)( http://www.faqs.org/docs/artu/ch15s04.html). -David Boyce ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make 3.99.91 release candidate is available
Solaris has a really old, really broken, /bin/sh which they've kept around
forever for compatibility reasons. Maybe there are some tests which assume
features that aren't supported in Solaris /bin/sh?
David
On Tue, Sep 17, 2013 at 7:38 AM, Paul Smith psm...@gnu.org wrote:
On Tue, 2013-09-17 at 10:23 +0200, Dagobert Michelsen wrote:
Unfortunately I don't have a copy of Sun Studio and I don't have any
SPARC hardware. I'll need someone with access to these to assist.
That is no problem. There are already a number of GNU projects that use
our buildfarm to ensure portability to Solaris:
http://www.opencsw.org/extend-it/signup/to-upstream-maintainers/
The farm is equipped with Solaris 8, 9, 10, 11 both Sparc and x86
and both Sun Studio compilers and gcc. Just let me know if you want
an account and send me your ssh public key.
OK, will do.
The first thing to try is re-running configure with the
--without-guile option then rebuild, to see if that helps. If it
does, then I think we'll need to reconvene this discussion on the
guile-users list.
The compile works with --without-guile, but I get two possibly harmless
warnings:
Hm. I'll check with the Guile folks.
I think these warnings are harmless but I'll look.
load.c, line 85: warning: assignment type mismatch:
pointer to function(pointer to const struct {pointer to const
char filenm, unsigned long lineno}) returning int = pointer to void
load.c, line 90: warning: assignment type mismatch:
pointer to function(pointer to const struct {pointer to const
char filenm, unsigned long lineno}) returning int = pointer to void
Additionally, there are a number of tests failing:
features/archives ... FAILED (4/10
passed)
features/escape . FAILED (6/8
passed)
features/output-sync FAILED (8/9
passed)
targets/ONESHELL Error
running
/home/dam/mgar/pkg/gmake/trunk/work/solaris10-sparc/build-isa-sparcv8plus/make-3.99.91/tests/../make
(expected 0; got 512):
/home/dam/mgar/pkg/gmake/trunk/work/solaris10-sparc/build-isa-sparcv8plus/make-3.99.91/tests/../make
-f work/targets/ONESHELL.mk.1
FAILED (5/6 passed)
variables/SHELL . Error
running
/home/dam/mgar/pkg/gmake/trunk/work/solaris10-sparc/build-isa-sparcv8plus/make-3.99.91/tests/../make
(expected 0; got 512):
/home/dam/mgar/pkg/gmake/trunk/work/solaris10-sparc/build-isa-sparcv8plus/make-3.99.91/tests/../make
-f work/variables/SHELL.mk.6
FAILED (7/8 passed)
This is much more concerning.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make 3.99.92 release candidate is available
I'm not sure if Pavel is on this list so adding him explicitly. Looks like there's a strong risk that no version of the spawn patch will get into 4.0 which would be a shame IMHO. -David On Tue, Sep 24, 2013 at 7:37 AM, Frank Heckenbach f.heckenb...@fh-soft.dewrote: Try this patch. I don't use EMX myself, but I noticed this when reading through the lastest changes, and it seems to fit your bug description. --- function.c.orig 2013-09-22 07:53:27.0 +0200 +++ function.c 2013-09-24 16:21:37.0 +0200 @@ -1710,7 +1710,7 @@ CLOSE_ON_EXEC(pipedes[1]); CLOSE_ON_EXEC(pipedes[0]); /* Never use fork()/exec() here! Use spawn() instead in exec_command() */ - pid = child_execute_job (FD_STDIN, pipedes[1], FD_STDOUT, command_argv, envp); + pid = child_execute_job (FD_STDIN, pipedes[1], FD_STDERR, command_argv, envp); if (pid 0) perror_with_name (error_prefix, spawn); # else /* ! __EMX__ */ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Mon, Sep 30, 2013 at 3:12 PM, Frank Heckenbach f.heckenb...@fh-soft.dewrote: I tested the new version and found no new issues, so as far as I'm concerned this feature can now be considered finished. Thanks for the initial patch, David, and the integration into GNU make, Paul! And special thanks to Frank and Eli for pushing it over the goal line! More than anything I'll be happy to see this thread die of natural causes, since it's going on ~2.5 years with an embarrassing misspelling by me in the subject line. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #33034] Makefile:23: *** mixed implicit and normal rules. Stop. for Linux kernel out of source builds
Follow-up Comment #12, bug #33034 (project make): I'm uploading a patch that adds an --allow-mixed-rules option which converts the fatal error to a warning. This is the only testing it's been subjected to: % cat Makefile all %.o: @echo Making $@ % make-4.0 Makefile:1: *** mixed implicit and normal rules. Stop. % make-4.0 --allow-mixed-rules Makefile:1: warning: mixed implicit and normal rules Making all I don't claim to understand all the logic in read.c but from a quick examination of the git log this seemed to have a good chance of being all that's required. I'm not in a position to test it thoroughly at this time. (file #29357) ___ Additional Item Attachment: File name: allow.patchSize:2 KB ___ Reply to this item at: http://savannah.gnu.org/bugs/?33034 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #33034] Makefile:23: *** mixed implicit and normal rules. Stop. for Linux kernel out of source builds
Follow-up Comment #14, bug #33034 (project make): Yes, I realize it's a bit of a wing and a prayer but from the git log it looks like this fix got into the same commit as a rewrite of major sections of read.c. Without reading and thinking through all of it, I figured there was a chance that adding the fatal error had been the only thing done for this particular fix. ___ Reply to this item at: http://savannah.gnu.org/bugs/?33034 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33034] Makefile:23: *** mixed implicit and normal rules. Stop. for Linux kernel out of source builds
Paul. Thank you very much! This means I'll be able to make professional use the many features and bugfixes which have arrived post-3.81 at some point. Given the flurry of other fit-and-finish fixes lately, would it be safe to assume there will be a 4.01 or equivalent upcoming in the foreseeable future? David On Sun, Oct 20, 2013 at 10:09 AM, Paul D. Smith invalid.nore...@gnu.org wrote: Update of bug #33034 (project make): Status: Not A Bug = Fixed Assigned to:None = psmith Fixed Release:None = SCM ___ Follow-up Comment #21: I have pushed a change to the Git repository. ___ Reply to this item at: http://savannah.gnu.org/bugs/?33034 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #40639] GNU Make with profiling information
Sorry for the side-track but for future reference when discussing the hypothetical rigorous output format: On Sun, Dec 15, 2013 at 5:38 AM, Tim Murphy tnmur...@gmail.com wrote: I'm not an XML fan really Agree. JSON might be an alternative. IMHO, YAML is to JSON what JSON is to XML (oh, and vim emacs!). David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] Fix output-sync option on EMX
On Tue, Jan 14, 2014 at 12:41 PM, Eli Zaretskii e...@gnu.org wrote: Thanks, but does EMX support output-sync? Didn't he just say The purpose of this piece is to add missing support for output-sync option to [...] EMX? David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Sat, Jan 18, 2014 at 4:17 PM, Paul D. Smith invalid.nore...@gnu.org wrote: There are limits to the size of command lines that can be invoked. On newer systems like Linux that limit is pretty large (I think it's 16K or so on Linux) but it exists, and on other systems it's much less. It's actually quite a bit bigger than that. On RHEL6 (2.6.32 kernel), getconf ARG_MAX reports 2621440. This, minus the size of the current environment, is the number of bytes available for command lines. It's conceivable that bash imposes its own limit due to fixed buffer sizes; I haven't checked. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: $(file) function bug or not?
On Tue, Jan 28, 2014 at 7:31 AM, Gisle Vanem gva...@yahoo.no wrote: Paul Smith psm...@gnu.org wrote: Then make invokes the shell with the results of the expansion, which are the rm and echo commands (and some empty strings which are ignored). So, the rm is running AFTER the file is created, and deleting it. Thanks. That makes sense when I think about it. I think the headline here is that $(file) is analogous to $(shell) in that it's intended specifically for use _outside_ of recipes. If you find yourself using either one in a recipe it's probably a sign you're on the wrong track. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: $(file) function bug or not?
On Tue, Jan 28, 2014 at 1:20 PM, Paul Smith psm...@gnu.org wrote: I'm not sure I'd go that far. $(shell ...) really _is_ useless in a recipe because make will invoke a shell to run the recipe anyway, so why have it invoke two shells? It's just redundant. However, $(file ...) can be useful in a recipe especially on systems which have limited command line lengths (Windows for example)... in fact I'd say that this is one of the main reasons people wanted $(file ...). I'll grant you your point about $(file ...); sorry for overstating. But (and here we go completely away from the original topic, sorry again) I think $(shell ...) in a recipe should be more firmly deprecated than you do here. It's not just redundant, it creates an order-of-evaluation error. See the third date in this example: % cat Makefile all: @echo The time is now $$(date) @echo The time is now $(shell date) sleep 4 @echo The time is now $(shell date) @echo The time is now $$(date) % make The time is now Tue Jan 28 13:31:08 EST 2014 The time is now Tue Jan 28 13:31:08 EST 2014 sleep 4 The time is now Tue Jan 28 13:31:08 EST 2014 The time is now Tue Jan 28 13:31:12 EST 2014 David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: $(file) function bug or not?
Paul has already demonstrated later in the thread that I was wrong. Nothing to see here, move along. David On Wed, Feb 5, 2014 at 7:42 AM, wpmcnamara wpmcnam...@yahoo.com wrote: David Boyce-3 wrote I think the headline here is that $(file) is analogous to $(shell) in that it's intended specifically for use _outside_ of recipes. Then why does the example given in the GNU Make manual show $(file) being used in a recipe? Not arguing whether it is intended to be used there are not as I'm not expert enough to know one way or another. Patrick -- View this message in context: http://gnu-make.2324884.n4.nabble.com/file-function-bug-or-not-tp15161p15213.html Sent from the Gnu - Make - Bugs mailing list archive at Nabble.com. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Add makefile.mak to list of default makefiles
I believe it's theoretically possible to have a case-sensitive
filesystem on Windows though it would primarily benefit masochists.
On Mon, Jul 7, 2014 at 5:00 PM, Paul Smith psm...@gnu.org wrote:
On Mon, 2014-07-07 at 21:15 +0100, Jonny Grant wrote:
+#ifdef WINDOWS32
+ { GNUmakefile, makefile, Makefile, makefile.mak, 0 };
+#else
This is OK with me. I definitely don't want to add Makefile.mk to the
UNIX/POSIX side; that's not something anyone ever uses anyway.
I do wonder, though, why we have both makefile and Makefile above.
Does that actually ever do anything on Windows, other than waste a bit
of time checking for the same file twice?
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. On Thu, Aug 21, 2014 at 9:13 AM, Paul Eggert egg...@cs.ucla.edu wrote: Eric Blake wrote: You may want to check out http://austingroupbugs.net/view.php?id=857 and add comments and/or change GNU make behavior accordingly. Let's leave GNU 'make' alone. Its behavior is better for rules like this: copy: original cp -p original copy I've added a comment to the Austin Group page. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, Aug 21, 2014 at 12:27 PM, Ray Donnelly mingw.andr...@gmail.com wrote: On Thu, Aug 21, 2014 at 8:03 PM, David Boyce david.s.bo...@gmail.com wrote: The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. Sounds pragmatic; the repeatable builds people would probably like a solution that doesn't require changing Makefiles though, either an env. var or a command line option? The command line is always possible for a special target given Paul's clever --eval option: make --eval .POSIX: ... And for an EV approach it should be possible to add that to an exported MAKEFLAGS. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
should assigning to .SHELLFLAGS suppress fast path?
I just noticed that while assigning to SHELL suppresses the fast-path algorithm, assigning to .SHELLFLAGS does not. Feature or bug? David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: should assigning to .SHELLFLAGS suppress fast path?
On Wed, Feb 11, 2015 at 12:22 PM, Paul Smith psm...@gnu.org wrote: On Wed, 2015-02-11 at 10:44 -0800, David Boyce wrote: I just noticed that while assigning to SHELL suppresses the fast-path algorithm, assigning to .SHELLFLAGS does not. Feature or bug? That's a good question. I'm sort of agnostic on it. On one hand, it's hard to think of a flag added to SHELLFLAGS (given the default value of SHELL) that would matter for the fast path Actually it gets weirder. Some values of .SHELLFLAGS suppress fast path, others do not: % cat makefile foo:; @echo making $@ % strace -e process -q -f -- make-4.1 .SHELLFLAGS=-ec 21 | grep 'execve.* = 0' [pid 23605] execve(/bin/echo, [echo, making, foo], [/* 46 vars */]) = 0 % strace -e process -q -f -- make-4.1 .SHELLFLAGS=-xc 21 | grep 'execve.* = 0' [pid 23745] execve(/bin/sh, [/bin/sh, -xc, echo making foo], [/* 46 vars */]) = 0 The two straced invocations differ only in that one adds -e to .SHELLFLAGS and the other adds -x. Wait … ChangeLog.3 says: (construct_command_argv_internal): If .SHELLFLAGS is non-standard use the slow path. Use that value instead of hard-coded -c”. But apparently -c and -ec are both considered standard so I guess it works as designed. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: disparity in Paul’s Rules
For the record, I didn’t ever suggest it was a problem with GNU make, just that the wording in your “rules” doesn’t map perfectly to reality. It’s your call, of course, whether that wording needs to be updated or whether fixing it would muddy the message. Separately: this situation is is not described explicitly in the manual (unless I missed a reference). Also, the line which says This means the first thing in the makefile seems to be part of a recipe” is also not entirely correct. How about “This means an apparent recipe line was encountered in a context where it could not be assigned to a recipe”. David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #45275] 4.0+ core dumps with long .PHONY targets via variables
Follow-up Comment #1, bug #45275 (project make): Sorry, ignore the via variables in the title string. ___ Reply to this item at: http://savannah.gnu.org/bugs/?45275 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #45275] 4.0+ core dumps with long .PHONY targets via variables
URL: http://savannah.gnu.org/bugs/?45275 Summary: 4.0+ core dumps with long .PHONY targets via variables Project: make Submitted by: boyski Submitted on: Mon 08 Jun 2015 01:45:44 PM GMT Severity: 3 - Normal Item Group: Bug Status: None Privacy: Public Assigned to: None Open/Closed: Open Discussion Lock: Any Component Version: 4.1 Operating System: POSIX-Based Fixed Release: None Triage Status: None ___ Details: I've isolated a bug in which make 4.x segfaults when a phony target name is longer than 8162 characters. I'll attach the test case. I haven't tracked down the coding problem but I do have some useful data: 1. This is a regression: it does not occur in 3.82 but does in 4.0 and 4.1. 2. At least in my environment it occurs when the target is 8163 characters or longer. This number is not a factor of 2 though it's fairly close to 8192. 3. Interestingly, if the phony target is converted to a real target e.g. long string: ; date Then the result is make: stat long string: File name too long (with exit status of 0 which may be an ancillary bug). Technically a phony target shouldn't be subject to the same length limits as an actual target since nothing is imposed by the OS/filesystem. It may be acceptable to impose those limits anyway but the core dump shouldn't occur. 4. Running under gdb I was able to get a stack trace showing the eventual seg fault occurring in xcalloc(). The trace is below. Program received signal SIGSEGV, Segmentation fault. 0x003a17a799b3 in _int_malloc () from /lib64/libc.so.6 (gdb) bt #0 0x003a17a799b3 in _int_malloc () from /lib64/libc.so.6 #1 0x003a17a7a5a6 in calloc () from /lib64/libc.so.6 #2 0x0041ca6a in xcalloc (size=136) at misc.c:231 #3 0x0040aea5 in enter_file (name=0x65a4b1 'X' repeats 2048 times...) at file.c:185 #4 0x0040bdaf in enter_prereqs (deps=0x654e50, stem=0x0) at file.c:538 #5 0x00422072 in record_files (filenames=0x663920, pattern=0x0, pattern_percent=0x0, depstr=0x6659c0 h\365\330\027:, cmds_started=1, commands=0x663850 \030\366\330\027:, commands_idx=0, two_colon=0, prefix=9 '\t', flocp=0x7fff6880) at read.c:2000 #6 0x0041fc22 in eval (ebuf=0x7fff6a60, set_default=1) at read.c:1013 #7 0x0041e958 in eval_makefile (filename=0x65a47d Makefile, flags=0) at read.c:445 #8 0x0041e3b7 in read_all_makefiles (makefiles=0x0) at read.c:262 #9 0x00419434 in main (argc=2, argv=0x7fffc688, envp=0x7fffc6a0) at main.c:1895 #10 0x003a17a1ecdd in __libc_start_main () from /lib64/libc.so.6 #11 0x00405f49 in _start () ___ File Attachments: --- Date: Mon 08 Jun 2015 01:45:44 PM GMT Name: Makefile.gz Size: 80B By: boyski http://savannah.gnu.org/bugs/download.php?file_id=34178 ___ Reply to this item at: http://savannah.gnu.org/bugs/?45275 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #45311] makes only one of two dependencies that depend on the same subdir-pattern rule
Follow-up Comment #2, bug #45311 (project make): I think the confusion here has to do with the value of $@. One might reasonably expect, in a multi-target pattern rule, that $@ would be the list of all matching out-of-date targets but in fact, according to the documentation: In a pattern rule that has multiple targets (see Introduction to Pattern Rules), ‘$@’ is the name of whichever target caused the rule’s recipe to be run. Which is a bit unclear wrt the case where neither target exists, but does imply strongly that $@ will refer to just one target. ___ Reply to this item at: http://savannah.gnu.org/bugs/?45311 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
