Re: POSIX ruling on up-to-date vs. identical timestamps
On Sat, 2014-08-23 at 18:33 -0700, Paul Eggert wrote: Paul Smith wrote: It needs to be considered carefully. How about having GNU 'make' do what GNU 'cp -u' does? The idea is to infer filesystem timestamp resolution by looking at every file timestamp that crosses your desk. When you see a file timestamp whose tv_nsec is nonzero modulo 100, for example, you know that its filesystem's resolution is finer-grained than 1 millisecond. When computation starts, you are conservative and assume that filesystems have 1-second resolution, but as computation goes on you gain more information about each filesystem and can become less and less conservative. It concerns me that this could cause the build to be non-deterministic in certain pathological situations, but it could be managed I'm sure. The immediate issue is that today, make doesn't track filesystems. It has no idea which files live on which filesystems, so we can't really keep track of the resolution on a per-filesystem basis. Of course the ability to track filesystems could be added without too much effort. It's trivial to determine the filesystem in POSIX via the device ID available from stat(), of course, but I'm not sure what facilities are available on the other ports. Although, I don't think either Windows or VMS support high-resolution timestamps right now (I don't know why Windows doesn't, because NTFS does support millisecond resolution timestamps I believe) so perhaps this is irrelevant. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Tue, 2014-08-26 at 18:04 +0300, Eli Zaretskii wrote: (I don't know why Windows doesn't, because NTFS does support millisecond resolution timestamps I believe) Because no one wrote the code, of course. Ah, the oldest reason in free software :-). The main problem is that this requires to write a replacement 'stat' (not rocket science). Can't we just #define stat(_p,_b) _stat(_p,_b)? Not sure if that's sufficient: I'm not overly familiar with the limitations on the POSIX emulation functions in Windows. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: POSIX ruling on up-to-date vs. identical timestamps
On Thu, 2014-08-21 at 13:57 -0700, Paul Eggert wrote: David Boyce wrote: The obvious compromise would be to change the behavior only in the presence of the .POSIX: special target. We should limit .POSIX to what POSIX requires. Even if the ruling stands POSIX won't require the HP-UX behavior, so .POSIX should be independent of this issue. I pretty much agree with everything Paul says in this thread. First, I can't remember getting bug reports on GNU make that the current way it handles identical timestamps is a problem. I'm not saying that such a thing has never happened (my memory is not what it was for one thing) but certainly there have not been enough complaints to make this a known issue. And since this is just the kind of seemingly small change that will end up annoying and frustrating many people, I'm not excited about it. Similarly, unless POSIX mandates this change in behavior I'm not that excited about having the .POSIX target imply this change either: that seems to be mixing up too many obscure differences in a single flag. It'd be OK to introduce a new special target to enable the HP-UX behavior. .EQUAL_TIMES_ARE_OUT-OF-DATE, say. We could document the new target next to .LOW_RESOLUTION_TIME, since they're related issues. The new target could act like .LOW_RESOLUTION_TIME, except that it imposes HP-UX rather than low-res behavior. I think something like this may be the way to go, but it might need to be more comprehensive than this even. Consider the Savannah bug https://savannah.gnu.org/bugs/index.php?40056 which complains about builds where some of the files live on filesystems that do support hires timestamps and some files do not. Despite the interesting review of the 10th grade concept of significant digits (*rolleyes*) I don't particularly care for the solution provided there: assuming that a subsecond value of 0 always means there is no subsecond resolution seems to me to be problematic. However, it needs to be considered carefully. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH v2 1/6] * expand.c (variable_name_extract): extract variable name and strip prefix.
On Mon, 2014-08-18 at 21:27 +0800, Macpaul Lin wrote:
Variables used in conditional lines usually has '$',
'(', and ')' prefix, and etc.
We can use vairable_name_extract() to extract pure variable
name without these prefix.
Hello. Thanks for your work on GNU make!
Can you provide some sort of summary of the feature you've created, what
its goals are, examples, etc.?
It can be helpful to contact me or someone on the development team
before (or while) doing feature work for GNU make. As a GNU project,
for example, we'll need to get copyright assignment paperwork for any
contribution of significant size and this takes some time.
Also, we may have suggestions or alternative implementations or concerns
that should be considered. Of course these can always be addressed
later, but it can save some effort to think about them up-front.
And finally, note that fully-formed changes need to have at least a stab
at changes to the manual (I usually rework these but it's helpful to
have a starting point) and some additions to the regression test suite,
if possible.
Cheers!
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: this is bug?
On Wed, 2014-08-13 at 17:45 +0400, Рушан Секаев wrote: GNU Make 3.81 Ubuntu 12.04 3.8.0-44-generic makefile hello: main.c hello.c gcc -o hello main.o hello.o main.o: main.c gcc -c -o main.o main.c hello.o: hello.c gcc -c -o hello.o hello.c if i update 'main.c' and run make hello then 'hello' file is not updated. Although the documentation is written on the contrary. When you say not update, I assume you mean that hello is recompiled but it still has the old behavior, not the changes you made to hello.c. This is because your rule says to rebuild hello whenever any of the .c files change, but in the command line you link the object files: gcc -o hello main.o hello.o Since your target hello doesn't depend on the object files, make doesn't rebuild them and so it always uses the original object files. If you want to link with the object files you should have your hello depend on the object files, not the source files: hello: main.o hello.o gcc -o hello main.o hello.o ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: bug report : make --help
On Tue, 2014-08-12 at 14:48 +0800, clo...@gmail.com wrote: [root@localhost ~]# make --help -C DIRECTORY, --directory=DIRECTORY “在执行钱先切换到 DIRECTORY 目录。 ” should be “在执行前先切换到 DIRECTORY 目录。 ” Hi; Translations for GNU projects, including GNU make, are handled by the Translation Project. Information on how to reach the translators can be found here: http://translationproject.org/domain/make.html Thanks for your effort to improve GNU make! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: strange issue with make cgit
On Mon, 2014-08-04 at 11:43 -0700, Gregory Fong wrote: When I select any of the other commits, the commit info is shown as expected. Not sure why these two have mysteriously decided not to work. Maybe the cgit cache got into a bad state? Hopefully someone can help get this sorted out. I saw it not working this past weekend, but it seems to be fixed now, magically... ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Make does not throw an error for target without a recipe?
On Thu, 2014-06-26 at 16:19 -0400, Patrick Donnelly wrote: Updating makefiles Considering target file 'test.mk'. Looking for an implicit rule for 'test.mk'. [...] Why is it trying to build target test.mk...??? See http://www.gnu.org/software/make/manual/html_node/Remaking-Makefiles.html File 'foo' does not exist. Looking for an implicit rule for 'foo'. [...] No implicit rule found for 'foo'. Considering target file '/etc/passwd'. Looking for an implicit rule for '/etc/passwd'. [...] No implicit rule found for '/etc/passwd'. Finished prerequisites of target file '/etc/passwd'. No need to remake target '/etc/passwd'. Finished prerequisites of target file 'foo'. Must remake target 'foo'. Successfully remade target file 'foo'. make: Nothing to be done for 'foo'. So the interesting thing here is that Make decides it needs to remake `foo' but it doesn't do anything (based on strace output). Then it decides it was successful? See: http://www.gnu.org/software/make/manual/html_node/Force-Targets.html That doesn't make any sense... How do I get Make to fail if it can't make the target? You have to give make a recipe to run. Then if the recipe fails (exits with a non-zero error code), make will fail. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Make does not throw an error for target without a recipe?
On Thu, 2014-06-26 at 16:19 -0400, Patrick Donnelly wrote: I'm bringing this problem up because targets are not being created (as expected) by implicit rules but I don't know this because make claims success. This part doesn't make sense to me; maybe you can provide more info here. An implicit rule ALWAYS has a recipe, so if a target matches an implicit rule, and does not have a recipe of its own, that recipe will ALWAYS be run. So, I don't understand the situation you're trying to describe. Having a target with no recipe will absolutely not keep targets from being created by implicit rules. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Make does not throw an error for target without a recipe?
On Thu, 2014-06-26 at 17:00 -0400, Patrick Donnelly wrote: This is a little inconvenient when you're relying on implicit rules. e.g. %.o: %.c cc -o $@ $ %: %.o ld $@ $^ foo: bar.a Even if foo.c is missing, Make still succeeds. This is actually what I'm dealing with and I'd like a method of catching a mistake like a missing .c file in the future. Aha. Now that we have a complete example we can discuss what's going on here. The problem--as you can actually see from the debug output you provided: No implicit rule found for 'foo'. is that because the .c file is not present, it means that make decides that these implicit rules DO NOT MATCH. Make checks the implicit rule chain, but the chain fails because foo.c doesn't exist, so the %.o:%.c rule cannot match, which means the %:%.o cannot match, so no implicit rule matches. Thus, there is no rule at all to build the target foo and make falls back to the no recipe provided behavior. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make -j 4 throws long list of compiler errors
On Tue, 2014-06-24 at 17:24 -0700, Renewal Computer Services wrote: make -j 4 (or whatever number, I use -j 12) is broken, with mingw-w64 a *nix-based project will cause a long list of compilation errors whereas without -j project will compile fine. No it's not. Your makefile is wrong. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Problem with GNU Make (3.81, probably newer) with stacking --include-dir=...
On Tue, 2014-06-10 at 16:01 -0700, Corey Brenner wrote: I've run into a situation where I want to control the include dirs in a recursive make. I am adding include paths to recursive invocations via --include-dir=, when I find one which matches my criteria. However, GNU Make seems to be holding on to these directories from high level invocations to lower ones. The construction of include paths, via --include-dir, should be per-invocation, and not shared with children, except by providing --include-dir=... on the recursive command line. I don't think you'll find too many people who agree with your assessment that GNU make's behavior in this situation is wrong. The typical use for the -I (--include-dir) flag is to provide a general location to search for included makefiles and most people would expect and intend that recursive invocations of make would get the same paths as the initial invocation so they can all include the same makefiles, without having to specify them to every sub-make. Is there a way to suppress this behavior? The only way I can think of is to modify the MAKEFLAGS variable in your makefile to remove the flags. In GNU make 4.0, this is pretty simple because the flag and the option are concatenated into a single argument: $ echo 'a:;: $(MAKEFLAGS)' | make-4.0 -f- -I/usr/include -I/bin : -I/usr/include -I/bin You can easily use $(filter-out ...) for example, to remove all the -I flags. Unfortunately in older versions of make the option and argument are left as two separate words: $ echo 'a:;: $(MAKEFLAGS)' | make-3.81 -f- -I/usr/include -I/bin : I /usr/include -I /bin which makes it much more difficult to remove them. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Problem with GNU Make (3.81, probably newer) with stacking --include-dir=...
On Wed, 2014-06-11 at 13:45 -0400, Paul Smith wrote: Unfortunately in older versions of make the option and argument are left as two separate words: $ echo 'a:;: $(MAKEFLAGS)' | make-3.81 -f- -I/usr/include -I/bin : I /usr/include -I /bin which makes it much more difficult to remove them. Sorry, pushed send too soon. One idea would be something like this: MAKEFLAGS := $(filter-out -,$(filter-out -I%,$(subst -I ,-I,-$(MAKEFLAGS))) which is gross for sure but should work. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU Make 3.80 : Problems and Bugs
On Wed, 2014-05-28 at 14:29 +0530, chandrababu nallani wrote: Hi, Could you send me the link to get the known bugs and issues for GNU Make 3.80. Sorry, but we don't spend the effort to keep detailed information on this. You can look at the bugs that were listed as fixed in the GNU make 3.81 release: https://savannah.gnu.org/bugs/index.php?go_report=Applygroup=makefunc=browseset=custommsort=0report_id=111advsrch=0fix_release_id=103severity=0bug_id=history_search=0history_field=0history_event=modifiedhistory_date_dayfd=28history_date_monthfd=5history_date_yearfd=2014chunksz=50spamscore=5boxoptionwanted=1#options This shows 108 bugs fixed. However, some of those will be fixes to new features introduced in 3.81, so they won't have been bugs that appeared in 3.80. Also, if you look at the bugs fixed in GNU make 3.82 some subset of those will have also existed in 3.80, but there's no good way to know which without reading the description and/or testing. And, of course, there were some fixes that were made where a bug was never filed in Savannah; finding those would require searching the ChangeLog. You can also look at the NEWS file for user-visible changes; these are not bugs per se but they give you an idea of things that were added to GNU make after 3.80 (and so weren't available then): http://git.savannah.gnu.org/cgit/make.git/tree/NEWS ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: -j on Tandem
On Wed, 2014-05-21 at 11:12 +, Van der Zaag, Paul wrote: We use your make utility on Tandem. Does the –j option have any effect, or is does it not on Tandem? I'm sorry but I have no experience with Tandem. The -j option will basically work on all systems. However, it works better on some operating systems than on others. You can find out if your system is fully supported by running this command (this is a UNIX shell command; not sure what kind of shell you might have): echo 'all: ;@echo $(.FEATURES)' | make -f- If the output contains the word jobserver, then your version of GNU make has full support. Otherwise it will have limited support. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Bug, or not? Pointer, please.
On Thu, 2014-05-01 at 17:41 +0200, Houder wrote: Below, besides the specified target (bla), the output of make also outputs the name of the makefile (M) ... AS IF THE MAKEFILE IS A TARGET. Can anybody explain this to me? Thank you. http://www.gnu.org/software/make/manual/html_node/Remaking-Makefiles.html Cheers! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make 4.0: archive rebuild resulted in 73 packages broken (help wanted)
On Wed, 2014-04-30 at 18:19 +0200, Guillem Jover wrote: build-stamp: echo $@ build-arch: build-stamp $ make --version | head -n1 GNU Make 4.0 $ make -f detect.mk -qn build-arch; echo $? 2 This is definitely a bug in GNU make 4.0 in handling -q (note the -n is not relevant: you can leave it out and get the same behavior). The docs are clear on what the exit codes should be, and with -q make should exit with 1 if something needs to be updated and no error was detected. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: error reporting
On Mon, 2014-04-07 at 21:26 -0700, Philip Guenther wrote: I am unable to reproduce this: Based on the offer of MSVC project files I would guess Rob is running on Windows. I expect this is a result of the buggy snprintf()/vsnprintf() implementations in the Windows MSVC compiler (well, by buggy I mean not conforming to the ISO C99 standard, which is 15 years old now...) I actually think that MSVC 2013 is supposed to fix this, because the C++11 standard has moved up to incorporate a number of the C99 features (C++03 only relied on C89) which is forcing Microsoft to finally implement them. As John mentions, I rewrote all that code since the 4.0 release to rely only on features available in the C89 standard, which likely fixes this issue as well. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: error reporting
On Tue, 2014-04-08 at 20:15 +, Rob Juergens wrote: Note that in Unix, vsnprintf() returns the TOTAL number of chars needed (add 1 for the null). If the output would overflow the buffer, then you would get a return value larger than the specified buffer size. In Windoze, vsnprintf() will return -1 if the buffer would be overflowed, and there is no indication of what length the buffer must be. Yes, I'm well aware of the difference in behavior, unfortunately :-/. Microsoft is *not* going to change this, since that would break who-knows how many existing programs that depend on that behavior. Well, that's a shame: if true MSVC will never be a conforming compiler implementation for C++11 or any newer C++ standard. The C++11 standard clearly states that the return value of the (standard) vsnprintf() function must be the number of characters that would have been written if [the buffer size] had been sufficiently large, not counting the terminating null character. This is basically the exact text for the C99 standard, imported into the C++11 standard. Microsoft is on the C++ standards committee and they certainly were aware of this, so my hope is they have a plan to allow for both legacy implementations and conforming implementations. Attached are 2013 files and updated other files I'm really not excited about the prospect of continuing to add new project files every year for each new version of Visual Studio. Isn't there any sort of backward-compatibility that allows the older files to work in newer Visual Studio releases? Also, is there any way to get these project files out of the root directory? I'd be a lot more sanguine about them if we could put them into the w32 subdirectory, or in an msvc subdirectory or something and get them out of the way. To my mind the only reason to ship Visual Studio project files with GNU make is if there are people who want to develop and enhance GNU make itself, and who want to use Visual Studio to do it. For people who just want to build GNU make on Windows and use the result for other projects, surely it's easier to just run a .bat build file or maybe use an nmake file to build make.exe; that's all you need. Visual Studio seems like real overkill for that. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: error reporting
On Tue, 2014-04-08 at 21:01 +, Rob Juergens wrote: Attached is a rewrite of the method vfmtconcat() in output.c. It seems to fix the problem. Thanks, but as Philip mentioned earlier I've completely rewritten the output.c file and callers of it so they use only C89 compliant functions (so no vsnprintf(), only sprintf() and vsprintf()). The latest code is available in the Git repository, available through Savannah: https://savannah.gnu.org/projects/make/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Test harness on VMS, running 511 tests, 60 tests failing.
On Sun, 2014-03-16 at 21:43 -0500, John E. Malmberg wrote: With that correction, I am back to 60 tests failing in 29 categories failing if I use the -keep option. You should never use -keep when invoking the full test suite. Using -keep will cause other tests to fail, on all platforms: it's an invalid mode of operation. -keep is only if you want to run one specific test to reproduce a problem, and keep the results for further testing (re-running the test with a debugger or similar). Cheers! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Thu, 2014-02-27 at 10:53 +0900, Mike Hommey wrote: I have no problems with your original patch. I think I said that right there and then. If Paul agrees, I will commit it. Paul? I'll reiterate my position that (a) I've seen nothing showing that it's inherently impossible for make to figure out for itself what the right thing to do is, instead requiring the user to specify it (if there is such a case it should be simple to show an example and explain why it's impossible for make to decide), and (b) given that (a) is true, I'm not excited about adding a makefile flag to force the user to make that decision... what happens, for example, if someone does decide to fix this the right way in the future so the user option is not needed... now we have this make setting which doesn't make any sense but which we have to maintain for backward-compatibility. HOWEVER. I don't know nearly enough about all the variations and different possibilities on Windows to make sound judgements on the specific situation, so if Eli is OK with it then I'm OK with it. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Bug-make Digest, Vol 135, Issue 17
On Mon, 2014-02-24 at 18:50 +0100, Bjoern Michaelsen wrote: Yes. But of course for any bigger C/C++ project, although a rather specific usecase, it makes up the majority of the source to parse. _If_ LibreOffice wouldnt already do some tricks, parsing the 13GB of generated dependencies would dwarf anything else in makes performance. And even with the tricks we do to reduce the deps to parse to some 350MB it still eats half of the time[1]. Yes, I definitely understand the issue and the desire for a solution. I actually spent some time considering it this weekend, and it was a fun thought-exercise. I imagine something like Python's auto-compilation mode, generating a Makefile.mkc (or something) for compiled makefile whenever it parsed a makefile. The big issue is obviously detecting whether the compiled version is appropriate and can be used, or the real makefile must be used instead. I was trying to think about what the inputs to a makefile would be, that would cause a pre-compiled version to be invalid. Unfortunately, there are a lot of them. But I have to say it's an intruiging problem. The best approach I think would be to provide a framework for compiled makefiles, but solve the problem in stages: initially it would only compile the very simplest makefiles. If there were anything in the makefile beyond the most basic syntax, it would simply not generate a compiled output and always read in the real makefile. Then over time, more sophistication could be added to the compiler as we understood what that might mean. I would definitely want this to be totally invisible to the user and not require any magic in makefiles (so no special include operator, etc.) Basically it should either be so safe that there's no way to tell the difference between using the compiled version or not (other than performance), or it should not be generated at all. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Bug-make Digest, Vol 135, Issue 17
On Mon, 2014-02-24 at 18:51 +, Tim Murphy wrote: On 24 February 2014 18:33, Paul Smith psm...@gnu.org wrote: I would definitely want this to be totally invisible to the user and not require any magic in makefiles (so no special include operator, etc.) Basically it should either be so safe that there's no way to tell the difference between using the compiled version or not (other than performance), or it should not be generated at all. Just a word of caution - there is a way in which this kind of thing can be a mistake, if you take a fairly simple request and expand it into a general solution for all things, then hit snags and abandon the whole effort when the initial subcase would have worked. That's as may be, but I'm not prepared to add new user-visible syntax like includedepcache to the GNU make language, which then needs to be maintained forever, just because we're not sure if anything better will come along or not. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] output.c: Fix memory stomp when need==fmtbuf.size
On Tue, 2014-02-04 at 10:33 +, Ray Donnelly wrote: I can't see it in the git repository yet. .. am I being too impatient? Sorry, it's committed in my local repo at home but I haven't pushed. I'll do that tonight. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Tue, 2014-02-04 at 18:54 +0200, Eli Zaretskii wrote: Another issue is with backslashes in paths. For example: $ cat EOF foo.mk foo: grep foo foo\\bar EOF (Note the is just there to trigger sh -c) This executes sh -c grep foo foo\\bar, which fails with: /usr/bin/sh: foobar: No such file or directory But in batch mode shell, this executes sh maken.sh with a content of grep foo foo\\bar and that fails with: maken.sh: line 1: foo\bar: No such file or directory (Note how in one case the backslash is eaten and not in the other case) Why are you using backslashes in file names when your shell is a Unixy shell? That makes little sense to me, and I don't see why Make on Windows should support such use. Unixy shells are supposed to get Unixy file names with forward slashes, including in redirection. I agree about using backslashes as directory separators, that obviously cannot work in /bin/sh, even on Windows. But I do see a problem above; what if the literal file 'foo\bar' (a file with a backslash in the name) existed? Then this would work on UNIX but fail on Windows, because (Mike shows) too many backslashes are eaten. On UNIX, grep foo foo\\bar would do as Mike shows the batch mode shell to do, and look for the literal file 'foo\bar', but note his example above where BOTH backslashes are dropped in non-batch mode. That seems wrong to me... ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] output.c: Fix memory stomp when need==fmtbuf.size
On Sun, 2014-01-26 at 16:35 +, Ray Donnelly wrote: I missed a few assert cases in the previous patch. Please find a fixed version attached. I applied this change. Thanks! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time
On Fri, 2014-01-31 at 16:55 +0200, Eli Zaretskii wrote: An option as command line argument, or as a special target as the original patch did? The former, as Paul objected to the latter. I didn't object, per se. I just prefer my tools to DTRT in all cases without me having to use any magical incantations to make it happen :-). My question, or _challenge_ if you like, is whether we can find a way to know, without any hints from the user, whether a given command line will work properly for /bin/sh -c in Windows, or if it needs to use a batch file. It seems, from my lofty vantige point of knowing virtually nothing about it, that this is a solveable problem. If we can try to write down (in text) the rules for when we would or would not be able to use /bin/sh -c, I would think it would quickly become clear if we can solve it or not. As for whether we use a command line option or makefile target, I think probably the makefile target is probably more appropriate. In general I like to use command line options to control user preferences which do not have direct impacts on the build, so if you ran the same build with different flags it would still work, just with different behaviors. But for things which will cause the makefile to not work properly if not specified, those things should be embedded into the makefile itself: I don't want it to be the case that you always have to remember to give a particular command line option or your build will break! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] Refactor and merge child_execute_job() code
On Thu, 2014-01-30 at 19:29 +0200, Eli Zaretskii wrote: I will review the patch some more in a day or two. (And I hope Paul will as well.) Yes, definitely, but it won't be until the weekend I expect. Life is intruding on hacking this month. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: $(file) function bug or not?
On Tue, 2014-01-28 at 09:52 -0800, David Boyce wrote: I think the headline here is that $(file) is analogous to $(shell) in that it's intended specifically for use _outside_ of recipes. If you find yourself using either one in a recipe it's probably a sign you're on the wrong track. I'm not sure I'd go that far. $(shell ...) really _is_ useless in a recipe because make will invoke a shell to run the recipe anyway, so why have it invoke two shells? It's just redundant. However, $(file ...) can be useful in a recipe especially on systems which have limited command line lengths (Windows for example)... in fact I'd say that this is one of the main reasons people wanted $(file ...). You can use it in a recipe to create an @-file, for example, for input to a program where just using $^ directly would be far too large for the command line. But there's no real point in deleting the file first with rm, since the operator will truncate it anyway. I guess there might be _some_ small reason to try to delete it with rm -rf if you suspect it might already exist as a directory. Now that I think about it, I just did come up with a valid reason to use $(shell ...) in a recipe! foo: $(shell rm -rf biz) $(file biz, hello there) :-p :-) ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] output.c: Fix memory stomp when need==fmtbuf.size
On Sun, 2014-01-26 at 16:35 +, Ray Donnelly wrote: I missed a few assert cases in the previous patch. Please find a fixed version attached. Thanks Ray; I'm utterly swamped for the last week or so with real life but I should have a bit more free time later this week; I'll check out your fix. Cheers! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: win32 compilation of make 4.0 source code
On Sun, 2014-01-26 at 16:22 -0800, Mark Brown wrote: That example method is a device to perform this .FEATURES test without inserting it into an existing Makefile. The syntax errors I was seeing were occurring when attempting to insert this test into an existing Makefile, full of Targets and command sequences. Well, since you've provided no details about exactly what you tried and what errors you received, or even details about exactly what you're trying to do, there's not much else we can say. You said echo the data, and I showed an example of echoing the data. If that's not what you want, you need to give more information. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: win32 compilation of make 4.0 source code
On Fri, 2014-01-24 at 11:03 -0800, Mark Brown wrote: I had a make.exe 3.80 and it had problems with else ifeq constructs, so that forced me to seek a more recent version for win32. Yes that version didn't support it. You can look at the latest NEWS file for info on what appeared when: http://git.savannah.gnu.org/cgit/make.git/tree/NEWS As a separate related issue, I was try to echo the .FEATURES information, since one of the its it claims to display mentions its Else If capabilities. However, every time I tried to echo the data I received a syntax error. What would be an example of a Makefile contents which would display the .FEATURES information ? There are a number of ways. Here's one that works with newer versions of GNU make (this is POSIX shell syntax): echo 'all: ; @echo FEATURES is $(.FEATURES)' | make -f- ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Missing po files in GIT
On Tue, 2014-01-14 at 11:00 +0400, Pavel Fedin wrote: Hello! I am trying to rebuild GIT version of Make, however .po files are missing in the repository. Is this intentional ? I have copied them over from my 4.0-2 archive. But where are they originally stored ? The PO files are dynamically downloaded from the translation site during builds when building from Git. This ensures we always have the latest versions. If you want to build from a Git checkout please be sure to read the README.git file. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #40639] GNU Make with profiling information
On Tue, 2014-01-14 at 11:58 +0200, Eddy Petrișor wrote: I understand the interest in the amount of time a given job takes to run, but I guess I don't understand the need for a start time offset at all. Isn't it sufficient to record the start time of a job, then when it's complete show the elapsed time for that job? Or recipe? Or both? The resulted information when using absolute time stamps is almost meaningless until they are further processed in an external tool because is hard to identify with a glance which job finished last, which first and so on. If a relative time stamp is provided one can waste less time on the analysis when some target is clearly the wasteful one. Also the relative time stamps generate very readable graphs directly after insertion in a tool such as Oocalc, Gnumeric or Excel. With absolute time stamps the very first thing I found myself doing was to generate relative time stamps. Sorry, I was not clear: I wasn't suggesting that it would be better to display the absolute time for the start time. I was wondering why we display the start time at all. Why not just show the elapsed time, and nothing else? That would avoid all of these issues. However Tim makes a reasonable point in his response so if it can be done without too much difficulty it would be good to show a relative start time. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make doesn't complain if target cannot be built
On Tue, 2014-01-14 at 06:56 +0100, Christian Eggers wrote: Am Montag, 13. Januar 2014, 17:20:43 schrieb Paul Smith: On Mon, 2014-01-13 at 22:23 +0100, Christian Eggers wrote: In Makefile 2 my intention was to state that foo.o depends on some generated header which must be generated first (might be in another rule). But I didn't want to change the be behaviour if foo.o cannot be built because e.g. there's no foo.c. Unfortunately, this behavior is correct, and even required by the POSIX standard (and is implemented by every version of make). Is there a workaround for this? Using explicit rules seems to be difficult in my case because some objects are built from .c sources, other from .cpp. Can you add the prerequisite to the pattern rules? %.o : %.c generated.h $(COMPILE.c) ... %.o : %.cpp generated.h $(COMPILE.cpp) ... This has the definite potential downside that if generated.h changes then _every_ object file will rebuild, even if they do not use it. But maybe they all use it anyway. Alternatively you could use order-only prerequisites: %.o : %.c | generated.h $(COMPILE.c) ... %.o : %.cpp | generated.h $(COMPILE.cpp) ... Now you have the opposite problem: generated.h will be rebuilt BUT none the object files will be recompiled (if the only thing that's changed is the generated.h file). Really, I'm not sure I see the ultimate problem. First, it seems that you will always need to define header files, etc., so you will already have plenty of rules of the form: foo.o: bar.h biz.h and so this issue already exists. Second, if all you're worried about is someone adding a bogus file to the makefile list of objects, that doesn't seem like a real issue; sure, it won't be caught here but the build will still fail (presumably you're using that list of objects to construct something else and THAT will not work if one of the object files is missing). If you really want to catch it early you can just do a specific check to make sure all the listed files exist: SOURCES := foo.c foo.cpp foo.asm ... MISSING := $(filter-out $(wildcard $(SOURCES)),$(SOURCES)) ifneq (,$(MISSING)) $(error Missing files: $(MISSING)) endif It's definitely true that you won't be able to get the traditional make error message; I just don't think it's worth the hoops you'll need to jump through to make that happen. But maybe you have some other requirement. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: win32 compilation of make 4.0 source code
On Mon, 2014-01-13 at 18:21 +0200, Eli Zaretskii wrote: From: Mark Brown mkbrown_...@hotmail.com Date: Mon, 13 Jan 2014 06:04:24 -0800 I was able to compile the make 4.0 source code downloaded from the gnu make site using Visual C++ 2005 under Windows 7 64 (generated 0 errors, 259 warnings) but executing the resulting make command file from the Windows 7 DOS Command Prompt yields a series of warnings/errors: == process_begin: CreateProcess(NULL, uname, ...) failed. make: process_begin: CreateProcess(NULL, uname -a, ...) failed. make: process_begin: CreateProcess(NULL, cygpath C:\zzz_13.12.1_general\tools, ...) failed. make: process_begin: CreateProcess(NULL, pwd, ...) failed. == Is the resulting file supposed to executed under Cygwin ? From what I can see, you're not using Cygwin at all either to build or to run GNU make, so there's no need to consider that. On Windows, GNU make can be compiled in a quite a number of different ways. It would be helpful if you gave us an idea of which method you used. But it doesn't seem like there's any need to worry about Cygwin. In any case, the released version of Make 4.0 had a few bugs, so you may wish to try building the current development code. This is true; I want to get a followup release out sooner rather than later but it will still be a little while yet. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: win32 compilation of make 4.0 source code
On Mon, 2014-01-13 at 19:37 +0200, Eli Zaretskii wrote: On Windows, GNU make can be compiled in a quite a number of different ways. It would be helpful if you gave us an idea of which method you used. He said that: he used Microsoft Visual C++ version 2005. But I meant, how? Through the Visual Studio project files? Or via the build_w32.bat script? Maybe that doesn't matter; I'm definitely no expert on this :-). ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make doesn't complain if target cannot be built
On Mon, 2014-01-13 at 22:23 +0100, Christian Eggers wrote: In Makefile 2 my intention was to state that foo.o depends on some generated header which must be generated first (might be in another rule). But I didn't want to change the be behaviour if foo.o cannot be built because e.g. there's no foo.c. Unfortunately, this behavior is correct, and even required by the POSIX standard (and is implemented by every version of make). Once a target is listed in the makefile it becomes known to 'make'. When make wants to build a target it will try to find an implicit rule for it. However, if there is no implicit rule found it just means that there were no commands available for that target, not that the target couldn't be built. According to the POSIX spec: If there are no commands listed for the target, the target shall be treated as up-to-date. This is actually useful behavior and is definitely taken advantage of in real makefiles. Cheers! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: output from $(error) lost with output sync
I fixed this one locally a couple of days ago; sorry for not pushing.
I'll do that shortly.
I don't think this change is sufficient because if output_sync !=
make_sync then make_sync is never dumped with the change below.
On Tue, 2014-01-14 at 06:21 +0100, Frank Heckenbach wrote:
Oliver Kiddle wrote:
Given the following Makefile, the output from the error function is
being lost when the gmake 4 output-sync is enabled:
[...]
With assertions active I even get this error:
% make -O
make: main.c:3409: die: Assertion `output_context == make_sync'
failed.
Aborted
I tried inserting OUTPUT_UNSET (); in func_error(), case 'e'. This
fixes the above problem, but still fails with fatal() called from
other functions, e.g.:
foo:
echo $(wordlist 0, 0, foo)
So the problem is more general. ISTM the assumption in die() that
output_context is either NULL or make_sync is just not valid, since
fatal() can be called from basically anywhere. So I made this change
which seems to fix the problem.
--- main.c.orig 2014-01-14 05:55:19.0 +0100
+++ make/main.c 2014-01-14 06:14:00.0 +0100
@@ -3406,9 +3406,8 @@
if (output_context)
{
- assert (output_context == make_sync);
+ output_close (output_context);
OUTPUT_UNSET ();
- output_close (make_sync);
}
output_close (NULL);
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #40639] GNU Make with profiling information
Sorry, I've been mostly away from my systems recently. On Wed, 2013-12-18 at 13:28 +0200, Eddy Petrișor wrote: Thanks for clarifying this. Could you please confirm if the general direction of the the is OK in the latest patch I sent? I will take a look. What it is in scope and what I would need help with is adding relative time stamp support in the profiling info instead of absolute time stamps. When analyzing the 'simple' output I realised the graphs looked awful because there was such such a scale difference between the time stamp and duration. The absolute time stamp also doesn't fit well worth the scope of the 'simple' output. Does it have to be relative to the start of the entire build (user's invocation of make)? I understand the interest in the amount of time a given job takes to run, but I guess I don't understand the need for a start time offset at all. Isn't it sufficient to record the start time of a job, then when it's complete show the elapsed time for that job? Or recipe? Or both? I tried to pass down a reference time stamp through an environment variable, but I am missing something from the processing since submakes don't see the variable I defined. I'll take a look. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #40639] GNU Make with profiling information
On Wed, 2013-12-18 at 13:28 +0200, Eddy Petrișor wrote: Could you please confirm if the general direction of the the is OK in the latest patch I sent? Conceptually it seems OK. I'm still not jazzed about having any more than one output format, and I'd prefer that format to be in a more-or-less readable form, more like the long form than the others. I think the output should go in the standard make output format, so something like: make[LVL]: target: details... Or, alternatively: target[LVL]: details Also I think it's enough to show the start offset and the elapsed time. End offset is not necessary IMO. I'm unsure about the PID. This is the pid of the make process so I'm not sure what the goal is. Is it to be able to collect all the times together maybe? Is it necessary to dump all the output times at the end? Doing so requires that we increase the size of the file structure to hold the information, and this is already large AND the most common structure in memory; there's one for every single target which for non-recursive builds can get really big. I'm trying to keep memory usage under control. If instead of that we print the information after each target is complete we can shift the storage of this information out of the file structure and into the commands structure or similar. To me it seems more useful to keep the elapsed time info right next to the command output rather than dumping it all at the end. Some other comments: 1. In general remember that GNU make code must conform to ANSI C89 / ISO C90. We shouldn't be using newer features of the language or runtime library unless we need to, and most of those require some kind of autoconf test. 2. Let's avoid float and double (and struct timeval). There's no reason why we can't fit enough precision in a uint32 to count elapsed time in milliseconds for a build: that gives 50 days or so. GNU make still supports running on systems where there is no floating point support (see the NO_FLOAT #define). Although I haven't tested it in a while. 3. The use of $ tokens in printf() statements is likely problematic from a portability standpoint. It seems like this should be relatively easy to avoid. 4. If the printed string contains text then it needs to be marked for translation (with the _(...) macro). 5. We don't want to be using fprintf() here. All output needs to go through the output.c module, so that it's properly managed via output sync. 6. gettimeofday is not portable. Also, it's not really the best option for timing things because (due to NTP etc.) it can change (by that I mean that if it reports 10s elapsed it might not be that 10s really elapsed). Using a monotonic clock is better, although that's also not portable. But if we have to be non-portable maybe we should try to get an accurate accounting. On the gripping hand maybe it's not that important to be absolutely accurate. You mentioned something about trying to send the start time through the environment but I don't see any code to that effect here; how were you doing that? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #40639] GNU Make with profiling information
On Sun, 2013-12-15 at 13:38 +, Tim Murphy wrote: I suppose I'm skirting around saying that I think gnu make needs an output format in the same way that valgrind has --xml=yes. I'm not an XML fan really - JSON might be an alternative. It isn't your problem to provide such a mechanism and I realise it's unfair of me to give you any sort of hard time about it. This feature is just a small example of how gnu make will evolve an irregular output format that's not easy to change once its finalised because it's not designed to be extendable. I'll quote my comment from the bug report in Savannah: Lastly, and this is where we may need to have more conversation, I'm not so excited about adding the formatting capability, at least not this way. I think that it could be a very useful thing to allow for specially-formatted output from GNU make. For example, perhaps an output format in XML that could be easily sucked into tools like Eclipse or whatever for further parsing (I'm not a huge fan of XML but it is relatively universal). Now that we have the output sync capability it would be straightforward to combine these and format the output of recipes for proper XML encoding as well. But I don't want to add multiple different formatting options, for different types of output. I'd prefer to have one comprehensive formatting capability. In other words, I prefer to take a page from Git, GDB, and other projects where the default output is human readable but probably not easily parsed by tools, and then provide a different output format option that provides machine-parse-able formats. I'm not interested in trying to create one output format that solves both of those problems. And, I think that this issue is completely separate from profiling and we shouldn't bundle them. On Sun, 2013-12-15 at 05:50:49 -0800, David Boyce wrote: On Sun, Dec 15, 2013 at 5:38 AM, Tim Murphy tnmur...@gmail.com wrote: I'm not an XML fan really Agree. JSON might be an alternative. IMHO, YAML is to JSON what JSON is to XML My concern with both YAML and JSON is they don't seemed well designed (especially JSON) to handle large blocks of formatted text. Any output format from make would need to be able to represent both the the recipe that was run as well as the complete output generated by that recipe. That could be quite a lot of text indeed, and preserving the formatting verbatim is crucial. In fact, I would suspect that the output format from make would often be much closer to a text markup format than to a data exchange format. That said, I'm not much of a fan of XML either so I'm certainly open to alternatives. (oh, and vim emacs!). Well, now you've just lost all credibility! :-p :-) ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: mingw-w64 build breaks and warnings
On Mon, 2013-11-25 at 19:39 -0800, Stephan T. Lavavej wrote: #1: OSN (fatal, NILF, ^ Fixed, thanks. #2: w32err.c: In function 'map_windows32_error_to_string': w32err.c:70:3: warning: passing argument 2 of 'fatal' makes integer from pointer without a cast [enabled by default] fatal(NILF, szMessageBuffer); ^ Fixed, thanks. Then there is a severe warning: #3: main.c: In function 'prepare_mutex_handle_string': main.c:796:7: warning: format '%x' expects argument of type 'unsigned int', but argument 3 has type 'sync_handle_t' [-Wformat=] sprintf (sync_mutex, 0x%x, handle); ^ Note that this sprintf() is guarded by #ifdef WINDOWS32 above, so we don't need to worry about other platforms. Using the MS length modifier, %Ix will work for both 32-bit and 64-bit builds. Fixed, thanks. #4: function.c: In function 'func_shell_base': function.c:1625:7: warning: variable 'errfd' set but not used [-Wunused-but-set-variable] int errfd; ^ I didn't change anything for this one. I'll look at it later. #5: getopt.c: In function '_getopt_internal': getopt.c:679:8: warning: suggest explicit braces to avoid ambiguous 'else' [-Wparentheses] if (opterr) ^ The getopt() code is taken from glibc, so I don't like to make GNU make-specific changes there. #6: job.c: In function 'reap_children': job.c:666:9: warning: label 'remote_status_lose' defined but not used [-Wunused-label] remote_status_lose: ^ I didn't change anything for this one. I'll look at it later. #7: job.c: In function 'construct_command_argv_internal': job.c:2667:9: warning: variable 'end' set but not used [-Wunused-but-set-variable] char *end; ^ I didn't change anything for this one. I'll look at it later. Cheers! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: mingw-w64 build breaks and warnings
On Tue, 2013-11-26 at 12:21 +, Ray Donnelly wrote: Instead of adding the MS-specific %Ix, could you not add (in the batch file) the define of __MINGW_USE_ANSI_STDIO=1, otherwise I suspect you'd be breaking people who prefer the stdio a bit more ansi (mingw-builds for example). I went with the easy fix, since this is already used in other places. I don't have Windows systems to test with so I don't want to make major changes. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: VMS port
On Mon, 2013-11-25 at 13:48 +0400, Pavel Fedin wrote: I am restarting work on spawn-patch for Cygwin. Actually, i have the very first version working, but want to try to do some face-lift and get rid of some #ifdef's. My first question is: is VMS port maintained, or dead long ago ? The VMS port is actively and capably maintained by Hartmut Becker. The ChangeLog shows he provided VMS fixes for 4.0 as recently as September. It's easier if patches are targeted for specific results, so it's best not to include major refactorings, like removing ports, in the same patch. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #40226] Weird failure on Windows with OUTPUT_SYNC_TARGET
On Sun, 2013-11-24 at 12:35 +0200, Eddy Petrișor wrote: Since you just worked on this, and the new type overwrites in case of repetition, does it make sense to overwrite instead of error? I think that for profiling this type of behaviour would be better. Sorry, but I didn't understand that. For the current options, I definitely want to allow for multiple instances without an error. Consider if some setting for -O was defined in the MAKEFLAGS environment variable, and another setting was defined on the command line, for example, overriding it. This is a common and expected situation and we don't want to throw an error. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make-4.0 allows empty ARFLAGS to be passed to ar
On Mon, 2013-10-21 at 17:36 +0100, Ken Moffat wrote: I haven't even figured out where quiet-command is located. I'm only an integration monkey, checking that everything in our distro (BLFS) will build with make-4.0. Will take a look. Please see my second email, from earlier today. Based on the parts of the email you sent, adding V=1 to your make invocation will likely cause your quiet commands to not be quiet anymore. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make-4.0 allows empty ARFLAGS to be passed to ar
On Mon, 2013-10-21 at 18:03 +0100, Ken Moffat wrote: 2. Unfortunately, rules.mak *is* adding -rR to the MAKEFLAGS - # Don't use implicit rules or variables # we have explicit rules for everything MAKEFLAGS += -rR I'll attach rules.mak. Is this a qemu bug which just happened to work in make-3.82, or a make problem ? It's a qemu bug, that just happened to work with make-3.82. By adding the -R flag they are stating they don't want to use ANY default variable values. However they are relying on the default variable value for ARFLAGS to be set. If they want to use -R they should set ARFLAGS to an acceptable value in their makefile. In previous versions of GNU make adding those flags wouldn't take effect in the current instance of make; it would only be in effect in a sub-make; that bug/mis-feature was fixed in GNU make 4.0. Now adding those flags in a makefile takes effect immediately, within that same instance of make. From the NEWS for GNU make 4.0: * Setting the -r and -R options in MAKEFLAGS inside a makefile now works as expected, removing all built-in rules and variables, respectively. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make-4.0 allows empty ARFLAGS to be passed to ar
On Mon, 2013-10-21 at 16:02 +0100, Ken Moffat wrote: # command line ARFLAGS = So this tells us that the empty value is being set from the command line. I then ran it in the directory where the build fails (dtc/) - # makefile (from 'Makefile', line 51) ARFLAGS = rc This is not interesting; don't bother trying to run make from within this subdirectory. It's the top-level directory makefile that's causing the problem. The error is happening there, not in the subdirectory. subdir-dtc:dtc/libfdt dtc/tests $(call quiet-command,$(MAKE) $(DTC_MAKE_ARGS) CPPFLAGS=$(DTC_CPPFLAGS) CFLAGS=$(DTC_CFLAGS) LDFLAGS=$(LDFLAGS) ARFLAGS=$(ARFLAGS) CC=$(CC) AR=$(AR) LD=$(LD) $(SUBDIR_MAKEFLAGS) libfdt/libfdt.a,) This is why the value of ARFLAGS is marked as coming from the command line; this rule re-runs make with ARFLAGS=$(ARFLAGS) and somehow this is empty. Unfortunately due to this recursion it's masking the initial location where ARFLAGS was set to empty in the makefile. If you look for ARFLAGS in the top-level makefile, is it being set to the empty value? That seems the most likely cause of the problem to me. If that's the case then we'll need to look at the Makefile.in file and the configure script and the config.log file to figure out why it's being set to the empty value. Also, you can run make V=1 to turn on verbose mode and see the actual commands that make is running; that can be helpful in some cases. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make-4.0 allows empty ARFLAGS to be passed to ar
On Mon, 2013-10-21 at 18:48 +0100, Ken Moffat wrote: Thanks. Is it OK if I quote this whole reply to the qemu devs ? Of course. This mailing list is publicly archived; quote away. Or point them to the thread: http://lists.gnu.org/archive/html/bug-make/2013-10/msg00151.html ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33034] Makefile:23: *** mixed implicit and normal rules. Stop. for Linux kernel out of source builds
On Sun, 2013-10-20 at 20:15 -0700, David Boyce wrote: Paul. Thank you very much! This means I'll be able to make professional use the many features and bugfixes which have arrived post-3.81 at some point. Given the flurry of other fit-and-finish fixes lately, would it be safe to assume there will be a 4.01 or equivalent upcoming in the foreseeable future? Yes, before too long. I don't plan on any major features in the next release, just cleanup and bug fixing. I don't plan on a release right away (like this month) though. I have to admit I still just don't understand the problem here. Surely no one is building kernels that old (pre-2.6.34) without any patches at all applied; that sounds inconceivably insecure. Why not just add one more patch that changes the 3 places (at most) in the makefiles that have this problem to the suite of patches that are already applied to those old kernels when you build them? It seems insane to me to avoid updating tools merely because of a few lines of makefile change in kernels that are almost 4 years old. Anyway. It will work for now, apparently. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make-4.0 allows empty ARFLAGS to be passed to ar
On Mon, 2013-10-21 at 04:05 +0100, Ken Moffat wrote:
With GNU Make 4.0 on x86_64-unknown-linux-gnu I am unable to build
qemu 1.6.{0,1} because it fails with
AR libfdt/libfdt.a
ar: two different operation options specified
Makefile:234: recipe for target 'libfdt/libfdt.a' failed
If I try to build it with make V=1 the output shows it has
ARFLAGS=
It builds if I export ARFLAGS=rv. If I have understood
the reply on the qemu-devel list correctly, qemu deliberately passes
empty ARFLAGS and relies on the following in 'info make' :
`ARFLAGS'
Flags to give the archive-maintaining program; default `rv'.
There must be something else going on in your makefile. The built-in
value for ARFLAGS is still present; you can see it set:
$ make -p -f /dev/null | grep ARFLAGS
make: *** No targets. Stop.
ARFLAGS = rv
$(AR) $(ARFLAGS) $@ $
Something in your makefile is overriding this value.
Perhaps you are setting MAKEFLAGS to add the -R flag (which disables
built-in variables)? Or, you're overriding it somehow.
If you run:
$ make -p -n
and search the output for ARFLAGS, the comment before should tell you
where that variable is being assigned.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: load on Windows
On Sat, 2013-10-05 at 16:34 +0300, Eli Zaretskii wrote:
EXPORT int mk_test_gmk_setup (const gmk_floc *flocp)
{
gmk_add_function (hello_world, hello_world, 0, 255, 0);
^^^
Make functions cannot have the '_' character in their names, so it
seems. Here's why:
/* Look up a function by name. */
static const struct function_table_entry *
lookup_function (const char *s)
{
const char *e = s;
while (*e ( (*e = 'a' *e = 'z') || *e == '-'))
e++;
So if you name your function hello-world instead, it will work.
Paul, if this limitation is deliberate, I suggest to document it where
we explain the arguments of gmk_add_function.
It's not so much a deliberate restriction, as it was a performance
improvement I added in 2002 along with the switch to hash table lookups,
and completely forgot about afterward :-). All the existing make
functions were lowercase and contained only '-', so we avoided the
lookup effort if we found a name which could not be a function.
Now that users can define their own functions, I don't think that
restriction is appropriate anymore. I'll fix this, thanks.
(I can make these changes in documentation, if you agree.)
I have a vivid memory of adding documentation regarding this so let me
look around for it.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: make check under Cygwin
On Wed, 2013-09-25 at 11:30 +0200, Denis Excoffier wrote: Still experimenting 'make check' with Cygwin (without the spawn-patch, with --disable-load): 1) In test_driver.pl, line 486 (look for Test returned), a comparison of $code against the value -1 is performed. However, 3 lines above the same test is done. This looks strange. It's not beautiful code, but it's not wrong. However I rewrote this function to be more clear. 2) In connection with (1) above, the category 'default_names' produces '*** Test returned 0' and this is considered to be $suite_passed = 0 (no deletion of files, no incrementation of categories_passed etc.), however the category seems to pass since the ok string is printed. If i apply (1) above, all is ok. I don't understand why this Test returned 0 does not show up on other (eg Linux) platforms. There is a bug in default_names; it's missing the final 1;. That means that the result of the last thing in that file will be used as the return. For case-sensitive filesystem versions of make that's an unlink() which succeeds. For case-insensitive filesystem versions of make it's the if-statement test, which is false. I've fixed default_names. 3) If the HOME env var is not set, one of the jobserver tests fails. Perhaps this test should not be launched if HOME is not set. I changed this test to not require $HOME. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make check under Cygwin (3.99.93)
On Tue, 2013-10-01 at 11:04 +0200, Denis Excoffier wrote:
Hello,
Thank you for this new RC.
I have tested make-3.99.93 under cygwin 32 bits with --disable-load
(and without the spawn-patch).
Several items that i had reported in
http://lists.gnu.org/archive/html/bug-make/2013-09/msg00110.html
(and for which i didn't get any feedback) are still there.
Yes, sorry, this one got lost.
I'm especially asking for
- item 1 (duplication of $code != -1) and
- item 3 (jobserver test fails if HOME is not set)
- and, if possible, implementation of ${abspath /foo} == /foo
I'll look at the first two. Eli said yesterday he will look at the last
one.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make 3.99.93 release candidate is available
On Tue, 2013-10-01 at 10:21 +0200, Stephan Beal wrote: On Tue, Oct 1, 2013 at 7:06 AM, Paul Smith psm...@gnu.org wrote: Note, no changes to translatable strings were introduced in this release. i'm not sure if this counts as a translatable string, but i'm seeing differences from 3.81: I meant between 3.99.92 and 3.99.93; this was a note to translators that there were no new changes since the last release candidate. I guess that wasn't very clear, sorry. 2 errors generated. make: *** [obj/fsl_vfile.o] Error 1 vs 2 errors generated. Makefile:75: recipe for target 'obj/fsl_vfile.o' failed Hm. That doesn't seem right. The *** should still be there, I thought. I'll take a look at that. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make check on darwin
On Sun, 2013-09-22 at 22:16 +0200, Denis Excoffier wrote: On 2013-09-22 06:50, Paul Smith wrote: Just curious: did you build with --disable-load because something failed otherwise? Or just because you wanted to? I didn't investigate the errors produced with --enable-load. Here they are: I believe these problems are because the test suite doesn't have the correct linker command line for creating shared objects on MacOSX. I suspect that if someone were to actually create a shared object using the right commands, it would work to load that into GNU make on MacOSX. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
GNU make 3.99.93 release candidate is available
Hi all. The fourth release candidate, 3.99.93, for the next version of GNU make is now available on alpha: ftp://alpha.gnu.org/gnu/make/make-3.99.93.tar.bz2 81e1f8693c50bf485382c31f0e827d91 ftp://alpha.gnu.org/gnu/make/make-3.99.93.tar.gz fc854c46fb60d8ffb1066dc46026aa4c There are a number of bug fixes since the last RC. Barring terrible bugs this will be the final release candidate for the next release of GNU make. Please test and verify. Note, no changes to translatable strings were introduced in this release. signature.asc Description: This is a digitally signed message part ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
As always thanks for your thorough testing Frank. On Tue, 2013-09-24 at 20:41 +0200, Frank Heckenbach wrote: Paul Smith wrote: On Thu, 2013-09-19 at 14:47 +0200, Frank Heckenbach wrote: Paul Smith wrote: I didn't fix it this way. Instead I used the existing MAKE_RESTART environment variable to communicate from the current make to the restarted make that the enter message was already printed (if it was) so it wouldn't be printed again. This ensures we don't get a stream of enter/leave statements as we re-exec. This works now (in my repo). It works for me too. However, since MAKE_RESTARTS is a documented variable, couldn't this change confuse user Makefiles? That's a good point. I rearranged this to ensure we set the make variable to the integer value, so the user can't see the special token. If so, the use of output_context might be slightly irritating (though not wrong) -- at first I wondered where the log_working_directory() output after the pump_from_tmp() calls was going to and whether it didn't need pumping too if it was going to the temp file, but apparently this never happens. ... which apparently does lead to a problem here (non-deterministic like many -j problems): AIUI, it dumps out to stdout/stderr, but prints Enter/Leave to output_context (which might be dumped much later), so out's contents are not properly enclosed. Since we dump out to stdout/stderr (and we got the semaphore for writing to those), it seems logical to me to print Enter/Leave there as well, so this seems to fix it for me (and again would obviate the need for the first parameter to log_working_directory()): I agree with this change. 8. Like job.c, I think function.c should check output_context-err = 0, to avoid redirecting to -1 when no temp file for stderr was set up. Yes, sounds right. 9. - if (! output_context || output_sync == OUTPUT_SYNC_RECURSE) + if (! output_sync || output_sync == OUTPUT_SYNC_RECURSE) This looks right to me as well. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make 3.99.92 release candidate is available
On Wed, 2013-09-25 at 13:51 +0200, Boris Kolpackov wrote: Paul D. Smith psm...@gnu.org writes: I'm not sure I fully understood the situation. This comment makes it sound like same version of make (same code) is 50% slower on the new system. Is that what you meant? Yes, the same make binary is 50% faster on 2-generations old Xeon compared to the current one. On the old system 3.99 is quite a bit faster than 3.82 (don't remember the actual numbers, i think it was about 30-40%). On the new box this difference is completely wiped-out; both versions take about the same amount of time. Nice! Are they running the same distro? Same kernel and libc versions anyway (that's about all make uses)? If you run make -d do you get essentially the same output for both? Maybe something in the environment or something in one of them is causing make to do a lot more work. The only really variable thing in a do-nothing make build is the amount of time it takes to stat all the files. Well, make also has to do a lot of memory-intensive processing (entering files into caches, creating all the dependency structures, pattern matching, etc). It could be some bad CPU cache interaction. That was my first thought since everything on this machine is faster, CPU, disks, memory. Maybe... you'd need to use perf to find out stuff like that I expect. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make 3.99.92 release candidate is available
On Mon, 2013-09-23 at 11:02 +0300, Eli Zaretskii wrote: There's also another problem: you added a test script dash-w, where we already had a dash-W. On Windows, these two map to the same file, so git overwrites the same file, and the file is always marked as modified. Please rename one of the files to some other name. Gah. Sorry about that. I've pushed a change for this. I had this problem on MacOSX on another project and it can be annoying to get out of it. Let me know if you get stuck and I'll dig up my email about how to fix your workspace. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Sat, 2013-09-21 at 07:28 +, Edward Welbourne wrote: I've never understood why someone would use $(shell ...) in a recipe... I mean, the recipe will be run in the shell!! I remember we once had a library where the command-line to the archiver was too long (about a quarter megabyte, IIRC). We worked round this by having a temporary scratch dir, hard-linking every .o file into it, then running the archiver in that directory, so as to trim all the paths off .o files and get the command-line short enough. We *would* have populated the scratch-dir by generating rules with define and eval and having those do the job, but some project managers weren't happy to let build machines take any software changes, not even the make upgrade to get define/eval working properly, so we hacked it by having the archiver command start by populating the scratch-dir. Of course, it couldn't do that by running a single command (the command-line would have been too long ...) so we did it with a $(shell ...) per object file via $(foreach ...). It was kinda ugly but it worked ;^ Yeah, I guess that's a legit reason. The next release of GNU make will have the $(file ...) function which will make that a LOT more efficient. Maybe you can convince TPTB to finally take a change :-). (Separate infrastructure for auto-creating directories looked after the .exists target, creating its directory.) I think we could, with hind-sight, have used the foreach to generate one command per object file, all separated with semicolons, so that no single command was too big for the shell, No, that wouldn't work. It's not the individual command (between simicolons) that's too long, the problem is that make can't invoke the shell itself because the command line + environment is too large. The only way to work around this limitation is to avoid invoking a single command that's too long. All but one of those I've looked into is manifestly less powerful than make (and I'm still waiting for my former colleagues to get the exception released to open source). I have no illusions that there aren't very serious issues with make and I have nothing bad to say about people who choose a different tool, or even write a different tool. Make is weighed down by standardization and history, so starting over from scratch has a lot of appeal. On the other hand starting over from scratch with an entirely different syntax means you have a significant disadvantage in terms of familiarity. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: make check on darwin
On Tue, 2013-09-17 at 23:46 +0200, Denis Excoffier wrote: I have configured with --disable-load. I had to apply the patch below (self explanatory i think) for 'make check' to return with no error. Moreover: 1) when configured with --disable-job-server, the tests in features/parallelism are however all activated and 'make check' obviously fails (#7 and #10) 2) when some errors are produced, a message is printed to direct the user to look for files ending in '.diff'; but many diff-files are actually named .diff.X, where X is a small number 3) and more importantly, 'make check' begins to write its results on stdout (eg features/archives..), then seems to switche to stderr for the results of the features/archives and all the following text. This is weird. I've addressed these issues in my repo (will be pushed shortly). Thanks for the testing. Just curious: did you build with --disable-load because something failed otherwise? Or just because you wanted to? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make 3.99.91 release candidate is available
On Tue, 2013-09-17 at 10:23 +0200, Dagobert Michelsen wrote:
Unfortunately I don't have a copy of Sun Studio and I don't have any
SPARC hardware. I'll need someone with access to these to assist.
That is no problem. There are already a number of GNU projects that use
our buildfarm to ensure portability to Solaris:
http://www.opencsw.org/extend-it/signup/to-upstream-maintainers/
The farm is equipped with Solaris 8, 9, 10, 11 both Sparc and x86
and both Sun Studio compilers and gcc. Just let me know if you want
an account and send me your ssh public key.
OK, will do.
The first thing to try is re-running configure with the
--without-guile option then rebuild, to see if that helps. If it
does, then I think we'll need to reconvene this discussion on the
guile-users list.
The compile works with --without-guile, but I get two possibly harmless
warnings:
Hm. I'll check with the Guile folks.
I think these warnings are harmless but I'll look.
load.c, line 85: warning: assignment type mismatch:
pointer to function(pointer to const struct {pointer to const char
filenm, unsigned long lineno}) returning int = pointer to void
load.c, line 90: warning: assignment type mismatch:
pointer to function(pointer to const struct {pointer to const char
filenm, unsigned long lineno}) returning int = pointer to void
Additionally, there are a number of tests failing:
features/archives ... FAILED (4/10
passed)
features/escape . FAILED (6/8
passed)
features/output-sync FAILED (8/9
passed)
targets/ONESHELL Error running
/home/dam/mgar/pkg/gmake/trunk/work/solaris10-sparc/build-isa-sparcv8plus/make-3.99.91/tests/../make
(expected 0; got 512):
/home/dam/mgar/pkg/gmake/trunk/work/solaris10-sparc/build-isa-sparcv8plus/make-3.99.91/tests/../make
-f work/targets/ONESHELL.mk.1
FAILED (5/6 passed)
variables/SHELL . Error running
/home/dam/mgar/pkg/gmake/trunk/work/solaris10-sparc/build-isa-sparcv8plus/make-3.99.91/tests/../make
(expected 0; got 512):
/home/dam/mgar/pkg/gmake/trunk/work/solaris10-sparc/build-isa-sparcv8plus/make-3.99.91/tests/../make
-f work/variables/SHELL.mk.6
FAILED (7/8 passed)
This is much more concerning.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: Suffix rules with dependencies
On Mon, 2013-09-16 at 08:48 +0300, Eli Zaretskii wrote: In this thread: http://lists.gnutls.org/pipermail/gnutls-devel/2013-September/006453.html and specifically in this message and its followups: http://lists.gnutls.org/pipermail/gnutls-devel/2013-September/006460.html there's evidence that GNU Make no longer treats suffix rules with prerequisites as normal files with funny names, as described in the manual. Did the behavior indeed change, and if so, in what version of Make? I couldn't find anything in NEWS, FWIW. I went back to GNU make 3.74 and I can't find any version that behaves as the manual documents, including 3.8* or current HEAD. However they all behave the same way which means that I can't reproduce the problem described in the gnutls mailing list either. Maybe my test makefile has a flaw? #-- .SUFFIXES: .SUFFIXES: .x .y .q .r .x.y: foo.h ; @echo cp $ $@ .q.r: ; @echo cp $ $@ #-- Then I ran touch foo.h foo.x foo.q. Now when I run both make foo.r and make foo.y, it runs the appropriate suffix rule. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make 3.99.91 release candidate is available
On Mon, 2013-09-16 at 20:52 +0200, Dagobert Michelsen wrote: /opt/SUNWspro/bin/cc -D_REENTRANT -pthreads -I/opt/csw/include/guile/2.0 -I/opt/csw/include -xO3 -m32 -xarch=sparc -m32 -xarch=sparc -L/opt/csw/lib -o make ar.o arscan.o commands.o default.o dir.o expand.o file.o function.o getopt.o getopt1.o implicit.o job.o load.o loadapi.o main.o misc.o output.o read.o remake.o rule.o signame.o strcache.o variable.o version.o vpath.o hash.o remote-stub.o guile.o glob/libglob.a -lkstat /opt/csw/lib/libintl.so -R/opt/csw/lib -L/opt/csw/lib -lguile-2.0 -lgc-lrt ld: fatal: soname option (-h, --soname) is incompatible with building a dynamic executable ld: fatal: flags processing errors Hm. Seems like there's something wrong with the link line here. The error is odd because it's suggesting that we're including a soname option, which we clearly are not doing (at least I don't see one). It may be some flags or options that we need to add in order to facilitate the new dynamic loading feature, or to link with Guile. Unfortunately I wasn't able to find much info using google. Please let me know if you need more information. Unfortunately I don't have a copy of Sun Studio and I don't have any SPARC hardware. I'll need someone with access to these to assist. The first thing to try is re-running configure with the --without-guile option then rebuild, to see if that helps. If it does, then I think we'll need to reconvene this discussion on the guile-users list. If turning off Guile doesn't help, please re-run configure adding --disable-load and see if that works. If it doesn't help then that will be a serious poser. If it does help, we'll need to find someone who has deeper knowledge of the Solaris compiler suite and who can advise us on the magic needed to have it support dlopen() etc. properly. Thanks for testing! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Thu, 2013-09-12 at 04:19 -0400, Paul Smith wrote: I think there may still be some change needed for directory tracking for the -Orecurse mode. If we're collecting output for an entire recursive make invocation we don't need enter/leave notifications around each individual target or line, which is what we have now. We only need it around the entire makefile. I'll look into this. OK, this should be all fixed now. I believe it's operating the way I want. Please try it out and report anything unusual or that you have problems with. I have a few more bugs to fix then RC2. Hopefully RC2 this weekend. Please make an effort to test things if you have access to Git. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #39943] Add an alternative parsing mode that regards space and tab as identical tokens
On Wed, 2013-09-04 at 18:15 -0400, David Boyce wrote: On Wed, Sep 4, 2013 at 4:42 PM, Paul D. Smith invalid.nore...@gnu.org wrote: Follow-up Comment #1, bug #39943 (project make): IMO _any_ editor which automatically replaces TABs with spaces should never be considered to be a useful programming tool. But YMMV of course. You know what they say about standards and how great it is that there's so many? The Python coding standard calls for no use of tabs and thus many people who do Python coding have their editors configured to turn tabs into spaces. I do, for example. I meant, editors which always replace TABs with spaces and can't be configured to behave other ways. I have my editor configured to replace TABs with spaces as well, for many of the file types I use. But my editor (Emacs, obviously) makes it quite trivial to use different settings for different types of files. Of course it's usually possible to configure the editor to use different profiles with different languages but that gets harder since makefiles have no standard extension. Well, most files DO have standard extensions; my attitude is your editor should never reformat your file unless you specifically request it, and so I leave the default setting as no replacement, and only set please replace on those file types where I want/need it. Most of those, including Python, DO have standard extensions so it's no problem. Or, of course, you can also use an editor which is capable of basing its decisions on more complex features than just an extension :-): Emacs knows when I'm editing Makefile or makefile as well. However I do agree that Stu's decision is a good case study when deciding risk/reward of maintaining backward compatibility :-p. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] Use spawn() in GNU Make on Cygwin, updated
On Fri, 2013-08-16 at 20:59 +0400, Pavel Fedin wrote: Friday, August 16, 2013, 19:19:58 you wrote: Also, when I'm making changes to the exec() code I don't spend a lot of time worrying about spawn() so it is possible that it will be broken from time to time and, in fact, I think you actually noticed some breakage in the cygwin list. Which one ? It seems that the discussion is not directly addressing the issue here. Personally I don't worry that spawn() will break, and I wouldn't mind a more performant version of make for windows/cygwin, either optionally or by default. However, if you look at the code in job.c that make uses to actually fork/exec a process you will see that there are a number of operations we perform in the child process between the fork and the exec. In fact I recently had to abandon using vfork() and use fork(), because some of the operations performed in the child before the exec() are not valid in a vfork() implementation. So, the question is very simple: is it technically possible to ensure that the operations make takes today in the child between fork and exec can be handled properly in a spawn-based implementation? I know some operations can be handled, such as close-on-exec. Others I'm not sure about. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] Use spawn() in GNU Make on Cygwin, updated
On Fri, 2013-08-16 at 13:30 -0400, Christopher Faylor wrote: On Fri, Aug 16, 2013 at 01:12:28PM -0400, Paul Smith wrote: On Fri, 2013-08-16 at 20:59 +0400, Pavel Fedin wrote: Friday, August 16, 2013, 19:19:58 you wrote: Also, when I'm making changes to the exec() code I don't spend a lot of time worrying about spawn() so it is possible that it will be broken from time to time and, in fact, I think you actually noticed some breakage in the cygwin list. Which one ? It seems that the discussion is not directly addressing the issue here. Personally I don't worry that spawn() will break, and I wouldn't mind a more performant version of make for windows/cygwin, either optionally or by default. You may not be concerned on whether spawn() in Cygwin actually works right or not but I am. If this becomes the default option for Cygwin I will definitely figure out how to turn it off for the Cygwin release, just like I'm not allowing MS-DOS paths in the released version of Cygwin's make. Sorry, I was too flippant here. What I would like, in the abstract, is that if you download the GNU make tarball from the GNU site and run ./configure and build in a Cygwin environment, the resulting executable has the same features you'd get if you downloaded the binary version of GNU make with the official Cygwin distribution. As far as I'm concerned, you (Christopher) know best and have final say as to what that configuration is, and I have little interest in getting involved in that. I'm willing to entertain the idea of configure options that allow for alternate behaviors, that people could get by performing their own builds. I have no exact criteria for what I would agree to, but in general it would have to be of significant utility to a significant number of people, and it can't be too ugly or difficult to maintain. I'm much less enthused about runtime flags, but I don't like to take anything completely off the table... yet. So, the question is very simple: is it technically possible to ensure that the operations make takes today in the child between fork and exec can be handled properly in a spawn-based implementation? This is, IMO, just a variation of the same question that Eli raised. Perhaps: I haven't gone back and re-read the whole thread. In any event I don't recall anyone (Pavel) specifically answering it. That's what I'm waiting for: the results of an investigation as to what works and what doesn't, based on examination of the code rather than anecdotal evidence (works for me). Presumably make works at least 99% correctly on Windows using spawn*(). I don't doubt at all that the patch actually works great with most uses of make in Cygwin. However, I would rather be 100% correct and slower than 99% correct with head scratching corner case errors. Exactly, hence the reason for my question. I'm not interested in adding this if, when it's enabled, things don't work correctly. On the other hand I'm not sure it's not possible to get things working correctly. Or, perhaps it's possible to make them work correctly if we disable some (already optional) features: then the configuration of this alternate mode should disable those as well. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH] Use spawn() in GNU Make on Cygwin, updated
On Fri, 2013-08-16 at 22:52 +0400, Pavel Fedin wrote: Exactly, hence the reason for my question. I'm not interested in adding this if, when it's enabled, things don't work correctly. On the other hand I'm not sure it's not possible to get things working correctly. Or, perhaps it's possible to make them work correctly Let me come in again ? Let's take a look at what we have. We have a pending change, which has undergone some testing. It works. At the other hand, there can be some corner cases. May be let's be constructive and test them, instead of just speculating about what can happen and what cannot ? Testing is very useful to be sure. However, the complexity of make is such that I suspect it will be EASIER, faster, and probably more accurate, to look at the (limited amount of) code in question than to try to test all the possible corner cases. Even the current make regression suite, at over 500 individual tests, is a very far cry from complete. A small analogy: you can speculate that tomorrow, when you come out of your home, a brick can fell from the roof and kill you. Now what ? Don't ever come out of house ? Wear builder's helmet ? :) If someone is replacing my chimney I'd definitely look up whenever I came out of my house, and watch for falling debris. I'd also ask them to think about what precautions to take so as not to drop things on me. To me current situation looks non-constructive. You say: Current implementation works, new implementation theoretically may fail (because it's new), so we must not change the code. That, though, is not what I said. What I asked for, perhaps not clearly enough, was for someone to investigate the current operations that make performs in the child between the fork and exec calls (of which there not so many) and decide, for each one, whether (a) it has no purpose in a Cygwin environment, or (b) it's not needed when using spawn(), or (c) it is needed but there's a reasonable way to get the same behavior with spawn(), or (d) it can't be duplicated, but it's only needed for some optional feature of GNU make (e.g., output sync) which can be disabled when spawn() is set up, or (e) it's needed and there's no mitigation. And for anything in category (e), it would be good if we can understand what the lost/incorrect functionality looks like. Oh and this should be done on the current HEAD of the Git master branch, or at least the current RC1 release, not on the code in GNU make 3.82. That would be a productive, and probably quite interesting, conversation that would move us in a positive direction, rather than spinning our wheels as we seem to be currently. After all, there are reasons to consider using spawn-like calls on UNIX systems, where available, as well. Maybe this could enlighten us as to what might be possible. I'm quite happy to discuss or explain the code, if necessary. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH 2/2] Do not use DOS paths on Cygwin
On Mon, 2013-08-05 at 10:56 -0400, Christopher Faylor wrote: Then maybe really add something like --enable-dos-paths which defaults to no on Cygwin and Yes on MinGW ? ac_cv_dos_paths=no configure works just fine and does not require a command-line switch. If this is a common/necessary thing maybe we should consider adding it to the appropriate README file. However, we don't seem to have a README for cygwin in the standard GNU make install. Is that something anyone is interested in generating? In general for Cygwin issues with GNU make we've traditionally pointed people to the cygwin list. Is there any interest on the cygwin side for a closer support relationship? Or is what we have now working OK? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH 2/2] Do not use DOS paths on Cygwin
On Wed, 2013-07-31 at 10:37 +0400, Pavel Fedin wrote: Looks like, if you want DOS paths, and running under Cygwin, an explicit conversion has to be performed on getcwd() result using cygwin_conv_path(). However i did not test this further because i follow official Cygwin way of doing things, and Cygwin deprecates usage of DOS style paths, obviously because they create lots of problems in UNIX utilities which are not modified to handle them. Cygwin even warns you when some API functions get DOS paths. Perhaps we need to split the HAVE_DOS_PATHS flag into two flags: something like ACCEPT_DOS_PATHS which enables various pathname parsing functions to understand DOS-style paths, and something like RETURN_DOS_PATHS, which is used in functions that generate absolute pathnames like abspath etc. to decide whether the generated pathname should be DOS-like or UNIX-like. I'm not sure what cygwin_conv_path() does or whether it is needed or useful in this situation. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH1/2] Use spawn() on Cygwin
On Tue, 2013-07-30 at 18:39 +0300, Eli Zaretskii wrote: In general, I feel it's wrong to do this: Cygwin is a Posix platform, so it should be using the Posix code, to be as compatible with other Posix platforms as possible. EMX is not a Posix platform, so using its code will likely make the Cygwin Make deviate from Posix behavior at times. If we decide to take this change I think we should reduce it to a single #define, such as HAVE_W32_SPAWN or similar, as we did with the HAVE_DOS_PATHS, set it in one place based on __EMX__ etc., then use that single macro where we want to check for spawn() in the code. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #37703] job starts before dependency
On Tue, 2013-07-09 at 10:43 -0400, Jay Lawrence wrote: My bad, I see that patch now, please disregard... In your comment below you mentioned you'd tried the latest source but it still failed. I believe I fixed this bug with a commit in April. Can you reproduce the issue with the latest source code; for example, the RC1 source package? I am seeing a very similar scenario with 3.82. I have also downloaded the latest source and see the same issue. When a very complex parallel build runs, and intermediate archive is generated, and some executables that depend on that archive are built before that parallel job completes. If I immediately reinvoke make with the same command line, it does then relink the executables. There is no patch information in this bug. Can you let me know what the change was so that I can verify that it is in the source code release I am building? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Make run in parallel mode with output redirected to a regular file can randomly drop output lines
On Mon, 2013-05-27 at 20:13 +0300, Eli Zaretskii wrote: and if so, does it guarantee non-conflicting writes? Not sure I understand what you are asking here. Can you elaborate? The original issue reported is that if you do something like this: make -j make.out and your make environment is recursive so you invoke one or more sub-makes, your output may not just be interspersed (that is output between multiple jobs are mixed together) but you will actually lose some output: it will never appear at all. The reason is that when you have multiple processes trying to update the same file at the same time using standard output file mode, there is a race condition between when the output is written to the file and when the current offset value is updated, where multiple processes could be overwriting the same part of the file. The suggested solution (not modifying make) is to use this instead: : make.out # truncate the file make -j make.out POSIX guarantees that if you open a file in O_APPEND mode, the above race can never happen because the kernel updates the file offset as the file is being written. Frank's question is whether other, non-POSIX systems have the same behavior with O_APPEND. Of course if they don't I don't see how it would make things worse than they are now. What I was suggesting was having make itself reset the mode of stdout and stderr to add O_APPEND, so that the first (most common) syntax would work correctly. POSIX says that you can change the mode of an open file descriptor using fcntl(). This wouldn't hurt anything in the above case, because when the shell opens the output file (with O_TRUNC) it will be truncated, then it will give the FD to make and make will change the mode to O_APPEND, so the file will still be truncated as you expect. The only possible way this could burn someone is if they are invoking make from a program where they've specifically opened make's stdout/stderr without O_APPEND and without O_TRUNC, and they expect make to start overwriting the file from the beginning rather than appending to the end. I cannot conceive of any situation where something like that would be done intentionally. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Make run in parallel mode with output redirected to a regular file can randomly drop output lines
On Sun, 2013-05-26 at 22:05 +0200, Stefano Lattarini wrote: On 05/26/2013 09:57 PM, Paul Smith wrote: [SNIP] Might be worthwhile checking the FreeBSD code for their make, to see if they do something like this. Nope, Frank was right: when run in parallel mode, FreeBSD make unconditionally behaves like GNU make does with the '-O' option enabled (I behavior I actively dislike, since it cannot be worked around). And it also has several other terrible hacks and quirks. For more info, see: http://www.gnu.org/software/autoconf/manual/autoconf.html#Parallel-Make Nevertheless, I do wonder whether forcing stdout/stderr into O_APPEND mode would be worthwhile. It would fix this problem in any event. I'm having a hard time coming up with a reason NOT to do it. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make release candidate 3.99.90 available
On Mon, 2013-05-20 at 19:06 +0300, Eli Zaretskii wrote: I still don't want to add back the pointer to the struct. Memory usage by GNU make is becoming a sore spot, especially as larger and larger build systems start to move to non-recursive make. If necessary we'll need to make the list a bit more sophisticated so we can add new elements to it if/when we rename targets. How about if we bite the bullet now, and simply add a fully qualified absolute file name of the dynamic object? I'm not against it, but I'm not sure it's sufficient. What we need is to ensure that whatever name exists in the struct file can be looked up in that list. Make generally has a problem with multiple names for the same target. I had played with the idea of storing inode information to try to determine if two different paths are the same file, but (a) that doesn't work on non-POSIX systems so well, and (b) it doesn't help if some rule recreates a file, giving it a new inode. We have the name and hname values already, and the hname value is intended to be a hashed lookup name which different struct file's referring to the same file will share, but this are not guaranteed to be enough, I think. Unfortunately finding the fully-qualified pathname of a file is VERY expensive, so I don't think we can just do it as a matter of course. To get this working efficiently we'd need to keep some kind of cache of fully-qualified directory paths, than we can try to figure out what directory a given file belongs to. Even this will be problematic dealing with make rules which create/delete directories and/or symlinks, and might invalidate the cache. And handling symlinks that might change directory paths can lead to problems trying to resolve paths like foo/../bar. This entire area needs careful thought. I believe there is at least one Savannah bug related to this already. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Bug-make post from cev...@gmail.com requires approval
On Thu, 2013-05-23 at 04:10 -0400, bug-make-ow...@gnu.org wrote: define PROGRAM_template = This syntax is not available in GNU make 3.81. It was introduced in GNU make 3.82. Thus the PROGRAM_template variable is not defined, and expands to the empty string, and thus your eval does nothing. Try using define PROGRAM_template, without the = at the end. It's best to install the version of the GNU make manual that comes with your distribution and use that for instruction; it should be the right one for the version of GNU make that comes with your distribution. Trying to use the examples from the manual of a newer version of GNU make will only lead to more frustration. Or else get the current stable version of GNU make and use that: then the online manual will match. Failing that, be sure to read the NEWS file for the latest release of GNU make to understand what features were added since your version was released: http://git.savannah.gnu.org/cgit/make.git/tree/NEWS ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #26596] MAKEFLAGS documentation tweak
On Wed, 2013-05-22 at 22:09 +0200, Stefano Lattarini wrote: On 05/22/2013 06:56 PM, Paul Smith wrote: I've reworked the MFLAGS / MAKEFLAGS generation to be more regular and rigorous yesterday, for 4.0, and to preserve _some_ backward-compat; I had thought about this issue when I did so. Please comment on the rules: I'll rework your wording to reference mostly MFLAGS, since that is what both mainline Automake and Automake-NG currently use. Unfortunately I think that as currently implemented in Git, MFLAGS is not as good as MAKEFLAGS for testing flags inside a makefile. See below. For MFLAGS: 1. In all cases where an option has both single-letter and long formats, the single-letter format will be used regardless of what appeared on the command line. Single-letter/no argument options they will always be preceded by a -. This is good. 2. If there are no options or variable assignments for MAKEFLAGS, it will resolve to the empty string. And I assume that variable assignments will still *not* be placed in $(MFLAGS), right? Automake has so far been relying on that utterly. Correct. 2a. If there are no single-letter / no argument options, the whole section is not present (i.e., no leading single dash, no leading space, etc.) OK. This is where I think you'll run into problems with MFLAGS. See below. 3. Any single-letter / no argument options will always be in the first word; there will be no - prefix to this word Even for MFLAGS? That would be a bad, backward-incompatible change. But I see this is not the case luckily: $ make -f- -Ifoo -k -i 'all:; @echo $$MFLAGS' -ik -Ifoo so I must have misunderstand you. You did... for MFLAGS, I said: 1. Rules 1-3,5,6 above hold, except that if there are single-letter/no argument options they will always be preceded by a -. Note the exception. So, if there are single-letter/no argument options they will always start with - is now superseded by this one (IMHO much clearer and more manageable): $ make -f- 'all:; @echo $(MFLAGS)' -Id -k -i -ki -Ifoo Correct. That change don't have any effect on Automake AFAICS, so no objection from me. The problem is item #2a as you list it above. It means that if you run: echo 'all:;@echo $(MFLAGS)' | make -f- --no-print-directory then in previous versions of make you'd get: - --no-print-directory while in new versions you'll get: --no-print-directory That is, that initial single - is no longer present in MFLAGS. What that means is that this statement: As a result, it should be completely reliable to use something like this to test for single-character, no argument options: $(if $(findstring k,$(firstword -$(MAKEFLAGS))),found k,no k) is not true when you substitute '$(MFLAGS)' instead of '-$(MAKEFLAGS)', because the firstword of MFLAGS might be a long option like --no-print-directory. With MAKEFLAGS, since it starts with a space if there are no single-letter/no argument options, '-$(MAKEFLAGS)' will resolve to - --no-print-directory and firstword will be -. I made this change to MFLAGS on purpose because, in the future, I'd like to be able to remove this weird special handling of - targets (that would require breaking makefiles which currently use '-$(MAKEFLAGS)' on their recursive make invocations, but that usage is illegal already and will already break in some situations). But, if this change to MFLAGS is a big problem we should discuss it. Is this tested in the GNU make testsuite? I'd love such a simple, sane behavior not to regress involuntarily. It should be, yes, although today it's not. In fact, all the various command line options should have regression tests for their behavior WRT MAKEFLAGS and recursion but most don't. And long options like this: $(if $(filter --trace%,$(MAKEFLAGS)),found --trace,no --trace) Aren't you missing a '=' after '--trace' here. Other than that, this seems good as well. --trace takes an optional argument, so --trace is a valid flag by itself with no =. I understand that from a backward-compatibility standpoint relying on this behavior is problematic. The important thing for me is that the new behavior doesn't break the existing idioms employed by Automake. I've run the Automake testsuite with the latest GNU make from git (both for mainline Automake and for Automake-NG), and found no regression, so I can declare myself happy for the moment. Ah! That's good news indeed. I wonder if you can comment on the above change (to remove - from MFLAGS in those situations) and whether you think it will cause problems or not. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make release candidate 3.99.90 available
On Sat, 2013-05-18 at 14:20 +0300, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Date: Fri, 17 May 2013 04:12:15 -0400 Hi all. The first release candidate for the next release of GNU make, GNU make 4.0, is now available for download: Paul, can you please add 4.0 to the list of versions accepted by the Savannah bug tracking UI, so that bugs fixed before the release could be marked as fixed in that version? Typically what I do is have all issues resolved before the release marked with SCM, then at release time I change all the bugs marked as fixed in SCM to be marked fixed in 4.0. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make release candidate 3.99.90 available
On Fri, 2013-05-17 at 19:42 +0200, Denis Excoffier wrote:
Compared with make-3.82, the new make-3.99.90 breaks those Makefiles,
like in tiff-v3.6.1 (rather old i know, before 2003 at least), that
use the construction:
make -${MAKEFLAGS}
Hrm. This is actually specifically discouraged by the documentation.
However reading the POSIX standard shows that make is required to accept
this format, at least for standard arguments.
The problem is that the new flags we're adding are causing some pain; I
may need to tweak the algorithm that generates the MAKEFLAGS values.
I'll take another look at this.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make release candidate 3.99.90 available
On Fri, 2013-05-17 at 22:35 +0300, Eli Zaretskii wrote: When a dynamic extension is being remade, it is unloaded by calling unload_file. The latter looks up its argument in a linked list of loaded objects. Now, unload_file is called with file-name as its argument; is it 100% sure that this string will be identical to what was used in load_file? IOW, do we never change relative file names to absolute ones (or vice versa), mirror backslashes to forward slashes, or do any other transformations that produce a different string which points to the same file? If we can do such transformations, unloading will be unreliable. Yes, I thought about this as I was making the change. I believe it is true that the initial pathname will be identical. However there are ways in which the path might change: for example through use of VPATH. I'm not exactly sure how all this will play out so it needs some testing. I still don't want to add back the pointer to the struct. Memory usage by GNU make is becoming a sore spot, especially as larger and larger build systems start to move to non-recursive make. If necessary we'll need to make the list a bit more sophisticated so we can add new elements to it if/when we rename targets. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: GNU make release candidate 3.99.90 available
On Mon, 2013-05-20 at 11:09 -0400, Boris Kolpackov wrote: This is because in the current algorithm, every single time we do an implicit rule search and compute possible target and dependency names they are all added to the string cache, even if they are deemed to be useless and not needed because that implicit rule is not chosen. In cases where there are lots of futile implicit rule searches the string cache gets bloated with these useless strings. Seeing that you haven't fixed it for 4.0, I assume there is no obvious or easy solution ;-). I've poked at it a few times for just a couple of minutes but nothing trivial occurred to me. I was hoping you'd look at it :-p :-). I don't think it's SO difficult to fix but it can be fiddly--this entire area is a bit fiddly. I care a lot less about memory than about speed and I believe it is the same for most other users these days. So maybe we should try to tune the hash (i.e., the number of buckets) so that lookup doesn't suffer too much? The complaint I got was that GNU make 3.82 was using 100's of MB of RAM more than it used to and this was the problem, not the lookup time. I'm not really jazzed about adding a workaround like this; the strcache is intended only for truly static strings that should exist for the lifetime of the process. It shouldn't be abused that way IMO. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Sun, 2013-05-05 at 04:37 +0200, Frank Heckenbach wrote:
COMMANDS_RECURSE _does_ mean to recurse. The reason for the '+'
prerequisite is to tell make that this line, even though it may not look
like it, will run a recursive make.
OK, let me just say that the meaning of recursive may not be
perfectly clear. Though the manual says: The @samp{+} prefix marks
those recipe lines as ``recursive'' so that they will be executed
despite use of the @samp{-t} flag., the example immediately
preceding this sentence has:
+touch archive.a
+ranlib -t archive.a
which are clearly not recursive make invocations.
I gather that make uses recursive in a wider sense as anything to
be run regardless (because it probably arrages by itself not to do
anything serious in a dry run or so), while the current
implementation of output-sync uses it in the more specific meaning
of a recursive invocation of GNU make (which will do its own
syncing).
It's not just this new feature that relies on this meaning. The
jobserver feature, which also wants to know which commands are running
recursive make, also does.
If people misuse it then they'll get odd behavior. I don't see that
there's anything we can or should do about that.
You're right, though, that this example in the make manual might not be
the best.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse
On Sun, 2013-05-05 at 00:44 +0200, Stefano Lattarini wrote: The test 'features/output-sync' now fails for me: Test timed out after 6 seconds Error running /storage/home/stefano/src/gnu/make/tests/../make \ (expected 0; got 14): /storage/home/stefano/src/gnu/make/tests/../make \ -f work/features/output-sync.mk.1 -j --output-sync=target Caught signal 14! FAILED (4/6 passed) Can you reproduce the failure? If not, let me know which information you need to debug this, and I'll try to provide them. It didn't fail for me. However, it's possible that the 6 second timeout is just a little too short for reliability. Look at line 178 in the test/scripts/features/output-sync file. It will look like this: #MAKE#[1]: Leaving directory '#PWD#/bar'\n, 0, 6); The 6 at the end is the timeout. Try changing that to 10 just to see if it helps. If not then it's a real problem. I'll need details about your system. Also please send the contents of the work subdirectory after the failure. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse
On Sun, 2013-05-05 at 11:11 +0200, Stefano Lattarini wrote: Sorry to add this only now, but I realized the failure is only reproducible if I run the testsuite with make -j, as in make -j8 check; and even in that case, the failure is racy. With a bare make check, things work for me as well. On the other hand, increasing the parallelism even more, other tests start to fail as well: The test suite definitely cannot be run in parallel. However this should not happen (and does not happen in my environment when I run the commands above) because the test harness cleans out the environment, which will remove any of the MAKEFLAGS or MFLAGS variables that might tell the make to run in parallel when it's not expected. Can you examine your shell configuration files etc. to see if they're setting MAKEFLAGS or MFLAGS? Although if that's true then the tests should fail all the time. Can you verify that there don't seem to be any leftover test files in the tests directory? Sometimes if something doesn't get cleaned up correctly that can cause future builds to fail. However if that were the case then make check without -j would fail as well. I don't have an explanation for this. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote: However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. Well, because the gnumake.h file is intended to be installed into the user's /usr/include or similar, and included in a binary package build of make such as RPM or DEB or whatever, and be included by the user's code, and when it's included there it should NEVER (IIUC) be using the in-make variant. I wanted to separate that in-make variant out so that users would never see it or have the possibility of accidentally using it, so I moved it into our internal headers which are never installed anywhere outside our source tree and would not be included in a GNU make binary package. This is slightly more potential maintenance on our part internally but is much safer for the user which is a tradeoff I'll almost always choose. However, if you really want it back the way it was please do choose a more unique name than MAIN. Something prefixed with GMK_ at least. One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? Make never changes the I/O mode of its standard streams. And it shouldn't, since everything Make itself writes or reads is always plain text. Since the standard streams always start in text mode, your suggestion boils down to make input from the temporary file be always in text mode. Well, I assumed that something that invoked make could set the mode and then make could inherit that mode. I don't know if that's how it works or not in Windows. And of course I doubt anyone does that. I understand your point. I just wonder if this difference might end up being visible to the user in some way. But, we'll leave it as-is. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Output sync completed (?)
Hi all. I've recently pushed changes to solve the last open issues that I'm aware of with the --output-sync feature: * If command line printing is not suppressed (@ is not used) the command line is attached to the output. * Extraneous enter/leave lines are not printed any longer. * I renamed the options to -Oline, -Otarget (not changed) and -Orecurse. * I took another stab at documentation; please check the Parallel Output node and see if it is more understandable now. Some picky details of -Otarget I didn't spell out but hopefully it's more clear. Please test the heck out of this with the different modes in your most difficult build environments, and let me know if you see any anomalies. I'm also interested to know if the enter/leave stuff is really correct now; it is in my testing but it sort of happened more by accident than real planning. We are right on the cusp of a release candidate for the next GNU make version. If you have the ability to build and test from Git feel free to start your testing early. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
On Sat, 2013-05-04 at 09:57 +0300, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Cc: reinp...@win.tue.nl, bug-make@gnu.org Date: Fri, 03 May 2013 16:51:47 -0400 I think enabling [-O] by default will be a no-brainer if/when we come up with a way to get it to produce the same output as without -j. IOW, run a parallel build, but output its results as if it were a non-parallel build. If you mean literally exactly the same as a non-parallel build, that is enormously difficult. Yes, literally, but with one exception: the order of producing on the screen output from targets that are remade in parallel (i.e. are independent of each other), and are on the same level of recursion, does not have to be preserved. Well that's not literally exactly the same then :-). But I won't agree to the caveats here, in particular the phrase on the same level of recursion. Today with parallel builds and recursion, we don't guarantee anything about the order of execution between different sub-makes and as I mentioned, I see no justification for adding that new requirement to synchronized output. Example: all: a b a: xa @echo $@ xa xb xc: @echo $@ b: xb xc xd @echo $@ xd: $(MAKE) foo @echo $@ By contrast, I _would_ mind to see this, for example: xa xb a xc xd $(MAKE) b I agree that is less than ideal and is one of the two issues I mention below. This will be fixed. However, you may see this: xa xb a $(MAKE) foo xc xd b There's nothing that can be done about that (and that's true of today's parallel build as well). My position is if we can get output sync to a level where it is no worse than today's parallel build behavior If we want it to be no worse, then why do we need it at all, let alone have it turned on by default? I thought -O should actually improve something, so no worse is too weak to describe that, IMO. Obviously we gain synchronized output. I mean, no worse in terms of other behavior. And what are our criteria for deciding whether it's no worse? The current default behavior might be confusing and hard to interpret in some cases, but at least it's familiar. -O changes that to a different behavior which is confusing (at least to me) in different situations, and is unfamiliar. I believe we can get to the point where anyone who can read and understand parallel output can even more easily read and understand the output from -O. Conceptually all we're doing is ensuring the output comes out at the same time, uninterrupted, instead of interleaved. It should not be hard to understand that. There are two known issues right now that are causing confusing output. Hopefully once I fix those the output will make more sense than normal parallel output. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #33138] .PARLLELSYNC enhancement with patch
On Sat, 2013-05-04 at 08:57 +0200, Frank Heckenbach wrote:
I shouldn't have written that. :-( Shortly afterwards, I found a bug
or perhaps two:
foo:
@echo foo
+@echo bar
(a)
% make -Ojob
foo
bar
foo
(b)
% make -Otarget
bar
foo
As you see, (a) -Ojob writes foo twice and (b) -Otarget writes
the messages in the wrong order.
The second one is known and I mentioned it the other day (hard to keep
up with all the messages, I know). I'm working on a fix.
The first one I've seen but hadn't had time to debug. I'll look at your
patch. I left the truncate where it was rather than doing it after the
sync_output() because I was hoping to avoid truncating a file that we'll
never use again anyway, but I guess it isn't a big deal.
COMMANDS_RECURSE _does_ mean to recurse. The reason for the '+'
prerequisite is to tell make that this line, even though it may not look
like it, will run a recursive make.
Since make doesn't parse the command line it can't know for sure which
ones actually recurse. It uses a heuristic, by looking for $(MAKE) or
${MAKE} in the unexpanded line. But this is easily defeated if your
sub-make invocation doesn't use that variable for some reason. Hence,
using + to force it.
___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make
Re: possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
On Fri, 2013-05-03 at 12:55 -0400, Paul Smith wrote: Suppose we do this: if we're about to invoke a line marked recursive and we're in -Otarget mode, then before we run it we'll show the current contents of the temp file (using the normal synchronized output function). I've implemented this feature and it seems to work as expected. I also implemented the fix to the duplicate output being shown in some cases. I have two open issues I want to address before calling this feature done: first, fix make's writing of the command it's going to run (for rules that don't start with @) as that's not working right. Second, fix the enter/leave issue. It turns out that these are currently somewhat bound together so I may have to solve the second one first. Oh, and a renaming as well :-) Eli, I did some cleanup in job.c to try to make things simpler and reduce duplication. I tried to be careful but it's quite possible I did something to disrupt the Windows version again. It's up to you if you want to fix any problems now or wait until I solve the above two issues and look at it all at the same time. There will be more disruption I think. One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On Fri, 2013-05-03 at 09:50 +0300, Eli Zaretskii wrote: From: Paul Smith psm...@gnu.org Cc: stefano.lattar...@gmail.com, bug-make@gnu.org Date: Thu, 02 May 2013 16:21:36 -0400 The one and only difference between them is that when running a recursive make, -Otarget WILL NOT capture the output of the sub-make, leaving whatever it prints going to the same stdout as the parent make, and -Omake WILL capture the output of the sub-make in the temporary file, to be dumped after the recipe is complete. Thanks for explaining that. I will have to try a few things to make sure I really get it this time, but one thing I can already say is that 'target' and 'make' are not very good names for these modes, since their semantics is quite different from the literal meaning of these two words. That difference creates a semantic dissonance that we should try to avoid, I think. It's good that we're having this discussion: I want to use it to try to inform my editing of the GNU make manual to be sure it's as clear as possible. I don't think the names are so inaccurate. I don't want to name things based solely on the details of how they differ from one another, which is all I was describing above. I prefer to name them based on how their most salient behavior manifests to the user. The way the user experiences the -Ojob option's results is that the output of every line of each recipe is dumped as soon as that line is complete. The way the user experiences the -Otarget option's results is that the output of all the lines of a given target are dumped at the same time once the target is completely built. The way the user experiences the -Omake option's results is that the output of an entire recursive make invocation is dumped, together, once the recursive make has completed. The issue of how -Otarget handles recursive make is, IMO, a detail necessitated by the architecture of recursive make invocations. I don't know that it's feasible to reflect that detail in the name. On the other hand I'm certainly not married to the current terms and I'm quite happy to change them if better ones can be found. It has already been suggested that -Oline would be better than -Ojob, and -Orecipe would be better than -Otarget, and -Omakefile would be better than -Omake. The current names are based more around _actions_ while the new suggestions are based more around semantic elements of make. To me -Omake is the most problematic. -Omakefile is not much better; in fact it might be worse (after all you can and often do invoke a recursive make on the same makefile). It would be nice to be more clear about the fact it applies only to recursive make invocations. Something like -Osubmake might be more accurate, except that I don't think we use the term sub-make in the documentation: we use recursive make. Is -Orecursive better? ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
possible solution for -Otarget recurse (was: Re: Some serious issues with the new -O option)
I have a solution for this problem that I think will work well, and will be simple to implement. Suppose we do this: if we're about to invoke a line marked recursive and we're in -Otarget mode, then before we run it we'll show the current contents of the temp file (using the normal synchronized output function). This will ensure that output from lines before the recursive make will be shown before the targets in the recursive make. It's not 100% identical but I can't see any way to do better. Thoughts? On Fri, 2013-05-03 at 16:39 +0300, Eli Zaretskii wrote: then how about if this exemption will only be applied if the recipe has a single command? In this case, if a recipe consisted of only one line then every target in the submake will be output immediately when it's finished, but as soon as you add another line to the recipe, like @echo Build is done!, now all of a sudden you get no output from the entire sub-make until the end. That would be too confusing. If the single-command-in-recursive-invocation is _not_ the use case which -Otarget is optimized for, then what use case is? -Otarget is not really about recursive invocations at all. It's there for the non-recursive targets. It would be nice if it worked well for the recursive targets, too, obviously. Consider every target in the entirety of build, including all submakes and all their targets as well, as one long list of targets (for example the results of a serial build). The fraction of those targets that are invoking sub-makes will be, at most, very small. -Otarget wants to collect the output of each individual target, then show the output for each target as soon as that target finishes. That's what users (should) expect when using this option. In the case of recursive make targets, this presents an unsolveable contradiction: how can you both show the output of EVERY target (including the targets run by the submake) as soon as it completes, and still not show the output of any target (including the targets that invoke a submake) before it's complete? You can't do both! The -Omake option chooses the latter as more important: it will delay the output until the submake is complete. The -Otarget option chooses the former as more important: it wants to behave properly for the large majority of targets which are not invoking a recursive make. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request: parallel builds feature
On Wed, 2013-05-01 at 20:38 -0700, Jim Michaels wrote: again, problem solved with what I proposed. think. separate shell window for each job. You can do that today by just writing your recipes such that they start a screen session or xterm or whatever. Those tools allocate and manage their own PTY's and so each has its own stdin. You haven't provided any use case description, at a level _above_ the implementation. Sure, the manual documents a restriction but not every restriction needs to be lifted. We only do that work if there's a real need for it that can't be met more easily a different way. So we need to understand at a higher level what problem you're trying to solve. Then maybe there's a good way to do it with existing make capabilities, maybe the best way is using capabilities available outside of make in conjunction with make, and maybe the best way is to enhance make. We'll have to see. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
Eli Zaretskii e...@gnu.org writes: If you want different behavior you can change your rule to use + on the two echo lines, so that they're also considered recursive and not saved up. If I do that, the echo from rec1 and rec2 mix up: D:\gnu\make-3.82.90_GIT_2013-05-01gnumake -f mkfsync3 -j -O start rec2start rec1 gnumake -f mkfsync simple gnumake -f mkfsync simple Is this also expected? You're right, I'm wrong. Using + is not really like -Ojob, because with -Ojob we still sync the output of each line. Using + the job just sends the output directly to the terminal with no sync. Thus this mixing up is what I'd expect (same as not using -O at all). I begin to wonder if this really requires a new per-line special prefix character like @, +, -, that controls the syncing of that particular line. I'm very reluctant to go there as it is a BIG change and a backward-compat issue. Also it seems that unlike the existing prefix characters, for this one we'd have as much need for a way to turn OFF sync as to turn it ON... bleah. On Thu, 2013-05-02 at 05:53 +0300, Eli Zaretskii wrote: Now, if you do nothing special for recursive make, you'll get no output from the entire build until it is completely done, because all the output from the recursive make command is going to the temporary file for that target, then it all gets dumped at the same time. Not every Makefile looks like that on its top level. I agree that we should cater to the above, but perhaps we could do that without punishing the other use cases. For example, perhaps we should have a -Osub-make option that will _not_ activate sync-output on the top level, only in the sub-make's. This should produce the desired results, I think. Can you clarify what the desired results are? I seem to have lost the thread. What is the behavior you see now that you are trying to avoid (or you don't see that you're trying to achieve)? Capture of the sub-make will mean that the entire output of that sub-make, and all of its recipes including ITS sub-sub-makes, will be sent to a temporary file and not displayed on the screen until the entire sub-make is completed. In what situation would we want to choose this, regardless of level of sub-make? In general I see no benefit in trying to special-case any particular level of make. For some builds the top level does all the work. For some the second level. For some the third. For many, different levels for different parts of the same build. I don't understand the change that you're suggesting. That's exactly what -Omake does today: whenever any recipe finishes it is flushed to the screen as a single unit, and no special handling is given to recursive makes. In my case, I see all the output at once. Maybe I misunderstand what -Omake is supposed to do, too. I think you and I said the same thing: the output from recursive makes is saved up and flushed all at once...? Tim Murphy tnmur...@gmail.com writes: One optimisation I have thought of in the past for this situation would be to allow a single job to hold onto the lock when it obtained it. This way it could output directly to the console while all other jobs would have to buffer. When it released, the next job lucky enough to grab the lock might have a full buffer already. It might appear to be less choppy. Not sure how it would perform. It might be less choppy (or it might not: it depends on your targets: are they all more-or-less equally chatty?) but we discussed this possibility and decided that it would be too costly in terms of performance. All you need is for one long-running job to get the terminal and pretty soon it's the only job running as all others have finished but can't continue with the next one until they can grab the terminal and dump their output. Personally I think this is a serious enough problem that it's probably not even worth doing the work in GNU make to provide this as an option. I suspect people would probably much rather just not use -O and live with interleaved output, or use -O and live with some choppy output, than suffer essentially random increases in build times. I could be wrong though. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Another issue with -O?
On Thu, 2013-05-02 at 20:30 +0300, Eli Zaretskii wrote: With this simple Makefile: all: @echo foobar! true Yes this is a bug. I thought of this while we were having our discussion yesterday. Unfortunately in all our tests we were using @ to silence make's output of the command line, so we didn't notice that the command line make prints before it runs the command is not printed using the child's output context, it's just printed directly to stdout. This causes the mis-order you see. I have a solution partly-implemented. I think the solution will actually make things a little more clean. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On Thu, 2013-05-02 at 20:24 +0300, Eli Zaretskii wrote: The desired results in my original use case are that the output of remaking each target is shown as one chunk in the order in which it is expected, i.e. from the first command to the last. Remaking a target, a.k.a. recipe in this context are all the commands that rebuild a single target. E.g., in this snippet: foo: bar cmd1 cmd2 $(MAKE) -C foo something-else cmd3 all the 4 commands and whatever they emit should be output in one go, and in the order they are in the Makefile. This is the way -Omake works. If you want that, you should set -Omake. The desired results in the example you showed, i.e. all: config.h $(MAKE) $(AM_MAKEFLAGS) all-recursive are that -O is in effect only for the sub-make which runs all-recursive. I'm not sure what you mean by ONLY for the sub-make but if I understand correctly this is the way -Otarget works. So, you can have either method. Maybe what you're suggesting is that you want BOTH methods, but for different targets, in the same build? Today we don't support that. This is what I was talking about when I referred to a new prefix character on a per line basis to turn on/off ordered mode. I think trying to control this through the command line, by choosing different MAKELEVEL values to behave differently, is not a good solution. However I don't want to go there yet. It's a big, disruptive change and I'm not convinced that (a) we can't do something to obviate it and (b) even if we can't that the use-case is important enough to justify it. The build still behaves just as it did before, it's just that in your original case the output is shown in a strange order. In general I see no benefit in trying to special-case any particular level of make. For some builds the top level does all the work. For some the second level. For some the third. For many, different levels for different parts of the same build. The user always knows what she is going to run, or at least ought to know. I think we already established that blindly appending -O to the Make command might cause surprising and even disappointing or annoying results, if one does it for a use case that does not play well with this feature. So some degree of adaptation between -O and its sub-modes to the depth and breadth (in parallelism sense) of the build is necessary anyway. I don't think I agree with much of the above. But it's a matter of opinion so we'll just have to wait and see. I don't understand the change that you're suggesting. That's exactly what -Omake does today: whenever any recipe finishes it is flushed to the screen as a single unit, and no special handling is given to recursive makes. In my case, I see all the output at once. Maybe I misunderstand what -Omake is supposed to do, too. I think you and I said the same thing: the output from recursive makes is saved up and flushed all at once...? No, that's not what I said. I said whenever a _recipe_ finishes, not whenever the entire Make run finishes. -Omake only has any relevance when doing recursive makes. If you run make -Otarget and make -Omake in a non-recursive make environment, you will get identical behavior. The one and only difference between them is that when running a recursive make, -Otarget WILL NOT capture the output of the sub-make, leaving whatever it prints going to the same stdout as the parent make, and -Omake WILL capture the output of the sub-make in the temporary file, to be dumped after the recipe is complete. There's no -O mode which will save up the entire output of a non-recursive make and dump it all at the end. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request: parallel builds feature
On Tue, 2013-04-30 at 17:20 -0700, Jim Michaels wrote: I wasn't digressing. I was explaining the point. the concept I am trying to present as a solution to the problem of making parallel stdin for --jobs in gnu make (which currenty doesn't work and is I guess single-threaded) is to make a separate terminal or command shell for each job, such as via a generated batch file or shell script. this is as simple as I can make it. You need to give a concrete example of the problem you're trying to solve. When the manual discusses stdin it means that only one job at a time can read from the make program's stdin. Multithreading won't help because there is only one input to read from, regardless of how many threads do the reading. What this limitation is discussing is if there were a makefile like this: read: read1 read2 read1 read2: ; @echo $@: enter a word: ; read word ; echo $@: $$word Both of these two targets read from stdin. If you run them serially, it works: $ make read1: enter a word: fooblatz read1: fooblatz read2: enter a word: barflatz read2: barflatz If you run in parallel then both the read1 and read2 targets run at the same time and both want input from stdin, at the same time. There's no way this can work: when you typed a word how could you know or specify which read operation got it? So make arbitrarily chooses one of the jobs to get the input and the others have their stdin closed. But if course, this doesn't impact in any way rules like this: read: read1 read2 read1 read2: ; @word=`cat $@.input`; echo $@: $$word Now if you have files like read1-input and read2-input, those will be read inside these rules and behave properly. I have learned that on a machine with 12 threads and 64GB of memory, you can have 50+ jobs running. This depends very much on what those jobs are doing. Obviously you CAN run as many jobs as you want. However I've never heard of being able to get more than #CPUs plus a few jobs running at the same time without making the build _slower_. At some point the kernel will simply thrash trying to keep all those jobs running at the same time, if they seriously outnumber the cores available to run them on. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
