Re: dynamic object searching (was: Re: Dynamic objects)
Since you can't (in my recent experience) load a 64-bit DLL into a 32-bit program, the real issue is what architecture was make itself built with. It's sort of a matter of make identifying itself rather than telling you that you're running on ubuntu 13.04 or solaris 10. Regards, Tim On 30 April 2013 03:38, Eli Zaretskii e...@gnu.org wrote: From: Paul Smith psm...@gnu.org Cc: Tim Murphy tnmur...@gmail.com, bug-make@gnu.org Date: Mon, 29 Apr 2013 16:34:01 -0400 On Mon, 2013-04-29 at 22:34 +0300, Eli Zaretskii wrote: Yes, that should be possible. My concern is that, at least on UNIX, the rules for this are complex and I don't want to reimplement the runtime linker :-). Maybe something like, first try the path as given and if that fails, try adding arch-specific extensions? No, much simpler: _always_ append _a_single_ arch-specific extension, and try loading that. We should document that extension; using the one that is used by default by the compiler for producing shared libraries should be good enough, I think. It's not so simple, though, as just .so vs. .dll. MacOS for example uses .dylib. And I think AIX does something else weird that I've forgotten about. Others probably do as well. Plus on UNIX any extension is acceptable since we're using dlopen() (even with the normal linker you can give any library name you want, it's only the -l flag that makes assumptions). Maybe someone wants to write pattern rules to build their GNU make loadable objects with a suffix .gmkso to distinguish it (and use a different rule) from building normal .so shared objects. I want to be sure the benefits outweight the loss of flexibility before we go down that path. OK, how about a lesser solution: have a builtin variable, say $SOEXT, which will have a platform-specific default extension of shared objects? Then whoever wants to use it for a portable Makefile, could do that, and people who want to use .gmkso can do that, too. WDYT? It's probably a good idea to have make predefine a variable containing the host architecture, to avoid the need for uname. That's a good feature regardless, but I think we should provide some solution for the extension as well. -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Duplicated Entering/Leaving directory when new option -O is used
Here is a minimal reproducer: $ cd /tmp $ mkdir x $ cd x $ mkdir src $ echo 'all:; $(MAKE) -C src $@' Makefile $ echo 'all:; @echo src' src/Makefile $ make # Work es expected make -C src all make[1]: Entering directory '/tmp/x/src' src make[1]: Leaving directory '/tmp/x/src' $ make -O # Duplicated lines in output make -C src all make[1]: Entering directory '/tmp/x/src' make[1]: Entering directory '/tmp/x/src' src make[1]: Leaving directory '/tmp/x/src' make[1]: Leaving directory '/tmp/x/src' The above has been obtained with GNU make built from latest git version (commit 'moved-to-git-46-g19a69ba'). Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Some serious issues with the new -O option
I'm sorry to say that the new -O option can interact *badly* with Automake-generated parallel testsuites (at least when they are being run interactively, that is, with the output of make connected to a tty). Let me elaborate a little (if you still have doubts or objections after reading the considerations below, please do not hesitate to ask for clarifications). For long-running testsuite with lots of tests, the use of the '-O' option prevents the results from the tests already run to be displayed on the screen until the *whole* testsuite has run. IMO, this destroys feedback and readability of the output. Try building GNU coreutils and running its testsuite with make -j2 -O on a slowish system to see how this can be annoying in practice. Moreover, since the output of the recipes is no longer connected to a terminal, automatic colorization of output no longer work: no more good results in green and failures in red (unless the user forces them by exporting AM_COLOR_TESTS to always). In defense of the '-O' option, I must say that I see how such an option can be actually useful to produce more readable logs when the output is being redirected to a file (so that one doesn't care about real-time feedback on the console); but for most interactive usages, I believe its downsides definitely overweight its advantages. So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
I'm guessing here but I imagine the main problem comes with delaying the results of submakes? I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. Regards, Tim On 30 April 2013 10:48, Stefano Lattarini stefano.lattar...@gmail.comwrote: I'm sorry to say that the new -O option can interact *badly* with Automake-generated parallel testsuites (at least when they are being run interactively, that is, with the output of make connected to a tty). Let me elaborate a little (if you still have doubts or objections after reading the considerations below, please do not hesitate to ask for clarifications). For long-running testsuite with lots of tests, the use of the '-O' option prevents the results from the tests already run to be displayed on the screen until the *whole* testsuite has run. IMO, this destroys feedback and readability of the output. Try building GNU coreutils and running its testsuite with make -j2 -O on a slowish system to see how this can be annoying in practice. Moreover, since the output of the recipes is no longer connected to a terminal, automatic colorization of output no longer work: no more good results in green and failures in red (unless the user forces them by exporting AM_COLOR_TESTS to always). In defense of the '-O' option, I must say that I see how such an option can be actually useful to produce more readable logs when the output is being redirected to a file (so that one doesn't care about real-time feedback on the console); but for most interactive usages, I believe its downsides definitely overweight its advantages. So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. Colouring is another issue which I would imagine could be done another way to let us have the best of both worlds. Regards, Tim On 30 April 2013 10:55, Tim Murphy tnmur...@gmail.com wrote: I'm guessing here but I imagine the main problem comes with delaying the results of submakes? I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. Regards, Tim On 30 April 2013 10:48, Stefano Lattarini stefano.lattar...@gmail.comwrote: I'm sorry to say that the new -O option can interact *badly* with Automake-generated parallel testsuites (at least when they are being run interactively, that is, with the output of make connected to a tty). Let me elaborate a little (if you still have doubts or objections after reading the considerations below, please do not hesitate to ask for clarifications). For long-running testsuite with lots of tests, the use of the '-O' option prevents the results from the tests already run to be displayed on the screen until the *whole* testsuite has run. IMO, this destroys feedback and readability of the output. Try building GNU coreutils and running its testsuite with make -j2 -O on a slowish system to see how this can be annoying in practice. Moreover, since the output of the recipes is no longer connected to a terminal, automatic colorization of output no longer work: no more good results in green and failures in red (unless the user forces them by exporting AM_COLOR_TESTS to always). In defense of the '-O' option, I must say that I see how such an option can be actually useful to produce more readable logs when the output is being redirected to a file (so that one doesn't care about real-time feedback on the console); but for most interactive usages, I believe its downsides definitely overweight its advantages. So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On 04/30/2013 11:55 AM, Tim Murphy wrote: I'm guessing here but I imagine the main problem comes with delaying the results of submakes? I think so, yes. The Distributed Make from Sun implements a similar output synchronization feature, and doesn't have the output delay problems I have experienced with GNU make (while it shares the issues with automatic output colorization). I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. Indeed. Regards, Tim I leave my original message quoted here in tis entirety, for reference: On 30 April 2013 10:48, Stefano Lattarini stefano.lattar...@gmail.comwrote: I'm sorry to say that the new -O option can interact *badly* with Automake-generated parallel testsuites (at least when they are being run interactively, that is, with the output of make connected to a tty). Let me elaborate a little (if you still have doubts or objections after reading the considerations below, please do not hesitate to ask for clarifications). For long-running testsuite with lots of tests, the use of the '-O' option prevents the results from the tests already run to be displayed on the screen until the *whole* testsuite has run. IMO, this destroys feedback and readability of the output. Try building GNU coreutils and running its testsuite with make -j2 -O on a slowish system to see how this can be annoying in practice. Moreover, since the output of the recipes is no longer connected to a terminal, automatic colorization of output no longer work: no more good results in green and failures in red (unless the user forces them by exporting AM_COLOR_TESTS to always). In defense of the '-O' option, I must say that I see how such an option can be actually useful to produce more readable logs when the output is being redirected to a file (so that one doesn't care about real-time feedback on the console); but for most interactive usages, I believe its downsides definitely overweight its advantages. So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). Thanks, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On 04/30/2013 12:01 PM, Tim Murphy wrote: What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. I wasn't even aware of those differences; as of latest Git commit 'moved-to-git-46-g19a69ba', I don't see them documented in either the help screen, the manpage, the texinfo manual, nor the NEWS file. Colouring is another issue which I would imagine could be done another way to let us have the best of both worlds. That is not trivial to do I think. For example, Automake-generated testsuites check whether the stdout is a tty to decide whether or not to automatically enable output colorization. And testsuites produced by Autotest do the same, AFAIK. If the make connects the stdout of those processes to non-tty files behind the scene, those checks are doomed to fail. Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
I know this isn't going to go down all that well, but I really think the output should be annotated in such a way that colourisation could be applied to the log file after a build has already finished. e..g you load a makefile into VIM - it can colourise it. Or a bit of C source code. Why not the log of a build you did yesteday? It's still very nice to be able to distinguish things by colour later on. Regards, Tim On 30 April 2013 11:16, Stefano Lattarini stefano.lattar...@gmail.comwrote: On 04/30/2013 12:01 PM, Tim Murphy wrote: What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. I wasn't even aware of those differences; as of latest Git commit 'moved-to-git-46-g19a69ba', I don't see them documented in either the help screen, the manpage, the texinfo manual, nor the NEWS file. Colouring is another issue which I would imagine could be done another way to let us have the best of both worlds. That is not trivial to do I think. For example, Automake-generated testsuites check whether the stdout is a tty to decide whether or not to automatically enable output colorization. And testsuites produced by Autotest do the same, AFAIK. If the make connects the stdout of those processes to non-tty files behind the scene, those checks are doomed to fail. Regards, Stefano -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
On 04/30/2013 01:22 PM, Tim Murphy wrote: I know this isn't going to go down all that well, but I really think the output should be annotated in such a way that colourisation could be applied to the log file after a build has already finished. While this might (underline might ;-) be an interesting feature, it is orthogonal to the issue under question -- that is, the need to retain the ability to automatically determine whether the output is going to a tty or a log file (that is, any file that is not a tty). e..g you load a makefile into VIM - it can colourise it. Or a bit of C source code. Why not the log of a build you did yesteday? It's still very nice to be able to distinguish things by colour later on. And notice that, currently, such a post-run colorization is easy to obtain, if you are only interested in having it working 99% of the time: just color a leading PASS: string in green, a leading FAIL: in red, e leading SKIP: in blue, etc. Regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: dynamic object searching (was: Re: Dynamic objects)
On Tue, Apr 30, 2013 at 3:57 AM, Tim Murphy tnmur...@gmail.com wrote: Since you can't (in my recent experience) load a 64-bit DLL into a 32-bit program, the real issue is what architecture was make itself built with. It's sort of a matter of make identifying itself rather than telling you that you're running on ubuntu 13.04 or solaris 10. Which is exactly what make_host tells you. It provides the architecture make was built for in the form of e.g. x86_64-unknown-linux-gnu or whatever configure comes up with. On Mon, Apr 29, 2013 at 8:03 PM, David Boyce david.s.bo...@gmail.com wrote: On Mon, Apr 29, 2013 at 7:53 PM, Paul Smith psm...@gnu.org wrote: Well, David, when you suggested it I wasn't so sure. But now that I've thought of it myself... brilliant!! :-p :-) But now I'm having second thoughts ... That was, I hope it's clear, a feeble joke. I find Paul's insight to be a stroke of pure genius. -David ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Duplicated Entering/Leaving directory when new option -O is used
On Tue, 2013-04-30 at 11:19 +0200, Stefano Lattarini wrote: The above has been obtained with GNU make built from latest git version (commit 'moved-to-git-46-g19a69ba'). Yes. I know the email lately has been daunting but if you wade through it you'll see a number of emails discussing this issue; it definitely needs to be addressed before release. Thanks for testing Stefano! ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Duplicated Entering/Leaving directory when new option -O is used
On 04/30/2013 03:06 PM, Paul Smith wrote: On Tue, 2013-04-30 at 11:19 +0200, Stefano Lattarini wrote: The above has been obtained with GNU make built from latest git version (commit 'moved-to-git-46-g19a69ba'). Yes. I know the email lately has been daunting but if you wade through it you'll see a number of emails discussing this issue; it definitely needs to be addressed before release. Sorry, I didn't follow most of the e-mail discussions (and I failed to properly search the archives, oops). Anyway, good to know this is on your radar. Thanks for testing Stefano! Thanks, and best regards, Stefano ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[PATCH] cosmetics: fix few innocuous typos
Most of these were found using Lucas De Marchi's 'codespell' tool. * ChangeLog: Fix minor typos. * ChangeLog.2: Likewise. * README.Amiga: Likewise. * TODO.private: Likewise. * function.c: Likewise. * glob/glob.h: Likewise. * job.c: Likewise. * main.c: Likewise. * readme.vms: Likewise. * remake.c: Likewise. * tests/ChangeLog: Likewise. * tests/NEWS: Likewise. * tests/README: Likewise. * tests/scripts/variables/private: Likewise. * vmsdir.h: Likewise. * signame.c: Likewise. While at it, improve line wrapping in the touched comment. Copyright-paperwork-exempt: yes Signed-off-by: Stefano Lattarini stefano.lattar...@gmail.com --- ChangeLog | 24 ChangeLog.2 | 2 +- README.Amiga| 12 ++-- TODO.private| 2 +- function.c | 2 +- glob/glob.h | 2 +- job.c | 2 +- main.c | 4 ++-- readme.vms | 2 +- remake.c| 2 +- signame.c | 8 tests/ChangeLog | 10 +- tests/NEWS | 2 +- tests/README| 4 ++-- tests/scripts/variables/private | 2 +- vmsdir.h| 2 +- 16 files changed, 53 insertions(+), 29 deletions(-) diff --git a/ChangeLog b/ChangeLog index 6badda8..3ad0050 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,27 @@ +2013-04-30 Stefano Lattarini stefano.lattar...@gmail.com (tiny change) + + cosmetics: fix few innocuous typos + + Most of these were found using Lucas De Marchi's 'codespell' tool. + + * ChangeLog: Fix minor typos. + * ChangeLog.2: Likewise. + * README.Amiga: Likewise. + * TODO.private: Likewise. + * function.c: Likewise. + * glob/glob.h: Likewise. + * job.c: Likewise. + * main.c: Likewise. + * readme.vms: Likewise. + * remake.c: Likewise. + * tests/ChangeLog: Likewise. + * tests/NEWS: Likewise. + * tests/README: Likewise. + * tests/scripts/variables/private: Likewise. + * vmsdir.h: Likewise. + * signame.c: Likewise. While at it, improve line wrapping in the + touched comment. + 2013-04-29 Eli Zaretskii e...@gnu.org * w32/include/dlfcn.h: New file. diff --git a/ChangeLog.2 b/ChangeLog.2 index 1705b76..0816454 100644 --- a/ChangeLog.2 +++ b/ChangeLog.2 @@ -3080,7 +3080,7 @@ Sun May 12 19:19:43 1996 Aaron Digulla digu...@fh-konstanz.de * main.c: (main), variable.c Changed handling of ENV-vars. Make stores now the names of the variables only and reads their contents when they are accessed to reflect that these variables are really - global (ie. they CAN change WHILE make runs !) This handling is + global (i.e., they CAN change WHILE make runs !) This handling is made in lookup_variable() * Makefile.ami: renamed file.h to filedep.h diff --git a/README.Amiga b/README.Amiga index aa82251..fa18814 100644 --- a/README.Amiga +++ b/README.Amiga @@ -25,8 +25,8 @@ GNU make): endif @@\ $(CC) Link Make.o To make -works. Note that the @@ must stand alone (ie. make@@\ is illegal). -Also be carefull that there is a space after the \ (ie, at the +works. Note that the @@ must stand alone (i.e., make@@\ is illegal). +Also be careful that there is a space after the \ (i.e., at the beginning of the next line). - Can be made resident to save space and time - Amiga specific wildcards can be used in $(wildcard ...) @@ -36,17 +36,17 @@ BUGS: dummy.h : src/*.c -tries to make dummy.h from src/*.c (ie. no wildcard-expansion takes +tries to make dummy.h from src/*.c (i.e., no wildcard-expansion takes place). You have to use $(wildcard src/*.c) instead. COMPILING FROM SCRATCH -- -To recompile, you need SAS/C 6.51. make itself is not neccessary, there +To recompile, you need SAS/C 6.51. make itself is not necessary, there is an smakefile. 1. Copy config.ami to config.h -2. If you use make to compie, copy Makefile.ami to Makefile and +2. If you use make to compile, copy Makefile.ami to Makefile and glob/Makefile.ami to glob/Makefile. Copy make into the current directory. @@ -54,7 +54,7 @@ is an smakefile. INSTALLATION -Copy make somewhere in your search path (eg. sc:c or sc:bin). +Copy make somewhere in your search path (e.g., sc:c or sc:bin). If you plan to use recursive makes, install make resident: Resident make Add diff --git a/TODO.private b/TODO.private index c3dda97..abb7410 100644 --- a/TODO.private +++ b/TODO.private @@ -6,7 +6,7 @@ This list comes both from the authors and from users of GNU make. They are listed in no particular order! -Also, I don't gaurantee that all of them will be ultimately deemed good +Also, I don't guarantee that all of them
Re: Some serious issues with the new -O option
Just to be clear, you're saying that the testsuite runs as one long operation, updating one target, and the recipe invokes one test script, right? I can see how that environment might be problematic for this new feature. It works much better for lots of smaller targets. However, you could avoid this issue if you specify that the target to be built is a sub-make, by prefixing it with a + operator for example. If this is true and you use -Otarget or -Ojob then make will not capture the output from that job. Of course this also has the unfortunate side-effect that running make -n will still try to run the test scripts. Hm. On Tue, 2013-04-30 at 11:48 +0200, Stefano Lattarini wrote: So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). This is a possibility for sure. I have to say that my experience with parallel builds hasn't been as wonderful as others here. I often get output which is corrupted, and not just by intermixing whole lines but also by having individual lines intermixed (that is the beginning of the line is one job's output, then the end of the line is another job's output, etc.) This kind of corruption is often completely unrecoverable and I simply re-run the build without parallelism enabled. I think it depends on how much output jobs emit and how many you are running at the same time. It could also be that Windows is better about avoiding interrupted writes to the same device. Tim Murphy tnmur...@gmail.com writes: What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. I always intended that and never suggested -Omake would be the default. I think -Omake is only really useful for completely automated, background builds. It wouldn't ever be something someone would use if they wanted to watch the build interactively. I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. This is the default. If you use -Otarget or (-Ojob) then what is supposed to happen is that make uses the same sub-make detection algorithms used for jobserver, etc. and if it determines that a job it's going to run is a submake it does NOT collect its output. However, I have suspicions that this is not working properly. I have a make-based cross-compilation environment (building gcc + tools) I've been using for years, and I tried it with the latest make and -O and saw similar problems (sub-make output collected into one large log instead of per-target). Thinking about it right now I think I might know what the problem is, actually. I'll look at this later. Stefano Lattarini stefano.lattar...@gmail.com writes: I wasn't even aware of those differences; as of latest Git commit 'moved-to-git-46-g19a69ba', I don't see them documented in either the help screen, the manpage, the texinfo manual, nor the NEWS file. I don't see where that code comes from? There is no Git commit in the standard GNU make archive with a SHA g19a69ba. The current HEAD on master is: 19a69ba Support dynamic object loading on MS-Windows. At any rate, the new option and option arguments are documented in the manual and there is an overview including the option arguments in the man page. The NEWS file doesn't discuss the individual option arguments, only the option. Ditto the --help output. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Some serious issues with the new -O option
Hi Paul. On 04/30/2013 03:37 PM, Paul Smith wrote: Just to be clear, you're saying that the testsuite runs as one long operation, updating one target, and the recipe invokes one test script, right? No; the testsuite runs as a recursive make invocation (yes, this is sadly truly needed in order to support all the features offered by the Automake parallel testsuite harness --- sorry), but each test script (and there can be hundreds of them, as is the case for GNU coreutils or GNU automake itself) is run as a separate target, explicit for tests which have no extension and pattern-based for tests that have an extension. I can see how that environment might be problematic for this new feature. It works much better for lots of smaller targets. See above: we are speaking of hundreds of little targets here ... However, you could avoid this issue if you specify that the target to be built is a sub-make, The problem here is exactly the fact that the output of the sub-make is not displayed until its execution has ended. by prefixing it with a + operator for example. BTW, the + operator is not portable soem other make implementations still targeted by Automake (e.g., Solaris make). If this is true and you use -Otarget or -Ojob then make will not capture the output from that job. Of course this also has the unfortunate side-effect that running make -n will still try to run the test scripts. This would be horrific of course ;-) Hm. On Tue, 2013-04-30 at 11:48 +0200, Stefano Lattarini wrote: So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). This is a possibility for sure. I have to say that my experience with parallel builds hasn't been as wonderful as others here. OTOH, mine has been very good, and I almost alway use make -j8 or higher when building GNU software (or other good software the relies on GNU make, like Git). The only real issues I've experienced so far have been when I used the '-O' option ... I often get output which is corrupted, and not just by intermixing whole lines but also by having individual lines intermixed (that is the beginning of the line is one job's output, then the end of the line is another job's output, etc.) Most of these issues are solved using silent-rules, as offered by Automake or implemented ad-hoc by build systems of other software (Git, Linux). I think the best ways to use parallel GNU make are: - with silent rules when the output goes to a tty (it makes the output visually scannable, and makes warnings and error messages form compilers and most other tools stand out much more clearly). - with verbose rules and possibly the -O option enabled when the output goes to a log, to simplify post-build debugging and analysis (especially useful with CI systems and the like). This kind of corruption is often completely unrecoverable and I simply re-run the build without parallelism enabled. I think it depends on how much output jobs emit and how many you are running at the same time. Indeed; this corresponds to my experiences so far. It could also be that Windows is better about avoiding interrupted writes to the same device. Not using windows, I cannot say anything here, sorry. Tim Murphy tnmur...@gmail.com writes: What I mean is that: ./make -Otarget might be a good interactive default rather than -Omake. I always intended that and never suggested -Omake would be the default. I think -Omake is only really useful for completely automated, background builds. It wouldn't ever be something someone would use if they wanted to watch the build interactively. I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise. This is the default. If you use -Otarget or (-Ojob) then what is supposed to happen is that make uses the same sub-make detection algorithms used for jobserver, etc. and if it determines that a job it's going to run is a submake it does NOT collect its output. This appears not to be the case, actually ... However, I have suspicions that this is not working properly. I have a make-based cross-compilation environment (building gcc + tools) I've been using for years, and I tried it with the latest make and -O and saw similar problems (sub-make output collected into one large log instead of per-target). Glad to know you can reproduce this. Thinking about it right now I think I might know what the problem is, actually. I'll look at this later. Thanks! Stefano Lattarini stefano.lattar...@gmail.com writes: I wasn't even aware of those
Re: Some serious issues with the new -O option
On Tue, 2013-04-30 at 16:04 +0200, Stefano Lattarini wrote: On 04/30/2013 03:37 PM, Paul Smith wrote: Just to be clear, you're saying that the testsuite runs as one long operation, updating one target, and the recipe invokes one test script, right? No; the testsuite runs as a recursive make invocation (yes, this is sadly truly needed in order to support all the features offered by the Automake parallel testsuite harness --- sorry), but each test script (and there can be hundreds of them, as is the case for GNU coreutils or GNU automake itself) is run as a separate target, explicit for tests which have no extension and pattern-based for tests that have an extension. This should work very well with -Otarget then, except for the colorization/highlighting issue... once it works as expected. I'll look into this issue later and I would be interested to see your experience with it once it's resolved. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[PATCH 1/3] build: require Autoconf = 2.62 and Automake = 1.11.1
Older versions of those tools should be considered fully obsolete. Also, GNU make already requires Gettext = 0.18.1, which has been released six months after Automake 1.11.1 and two years after Autoconf 2.62; so the new requirement shouldn't be problematic for people already bootstrapping GNU make from the Git repository. * configure.ac (AC_PREREQ): Require Autoconf 2.62 or later. (AM_INIT_AUTOMAKE): Require Automake 1.11.1 or later (1.11 had some serious bugs, and should not be used). Copyright-paperwork-exempt: yes Signed-off-by: Stefano Lattarini stefano.lattar...@gmail.com --- ChangeLog| 14 ++ configure.ac | 6 +++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/ChangeLog b/ChangeLog index 6badda8..5278d49 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,17 @@ +2013-04-30 Stefano Lattarini stefano.lattar...@gmail.com (tiny change) + + build: require Autoconf = 2.62 and Automake = 1.11.1 + + Older versions of those tools should be considered fully obsolete. + Also, GNU make already requires Gettext = 0.18.1, which has been + released six months after Automake 1.11.1 and two years after + Autoconf 2.62; so the new requirement shouldn't be problematic + for people already bootstrapping GNU make from the Git repository. + + * configure.ac (AC_PREREQ): Require Autoconf 2.62 or later. + (AM_INIT_AUTOMAKE): Require Automake 1.11.1 or later (1.11 had + some serious bugs, and should not be used). + 2013-04-29 Eli Zaretskii e...@gnu.org * w32/include/dlfcn.h: New file. diff --git a/configure.ac b/configure.ac index d610343..548bc42 100644 --- a/configure.ac +++ b/configure.ac @@ -1,6 +1,6 @@ # Process this file with autoconf to produce a configure script. # -# Copyright (C) 1993-2012 Free Software Foundation, Inc. +# Copyright (C) 1993-2013 Free Software Foundation, Inc. # This file is part of GNU Make. # # GNU Make is free software; you can redistribute it and/or modify it under @@ -18,7 +18,7 @@ AC_INIT([GNU make],[3.82.90],[bug-make@gnu.org]) -AC_PREREQ([2.59]) +AC_PREREQ([2.62]) # Autoconf setup AC_CONFIG_AUX_DIR([config]) @@ -26,7 +26,7 @@ AC_CONFIG_SRCDIR([vpath.c]) AC_CONFIG_HEADERS([config.h]) # Automake setup -AM_INIT_AUTOMAKE([1.9]) +AM_INIT_AUTOMAKE([1.11.1]) # Checks for programs. AC_PROG_CC -- 1.8.3.rc0.19.g7e6a0cc ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[PATCH 2/3] build: enable 'subdir-objects' and 'silent-rules' automake options
* configure.ac (AM_INIT_AUTOMAKE): Here. The future major Automake version 2.0 (ETA about one, one and half year from now) will likely enable them by default, so better prepare ourselves. Signed-off-by: Stefano Lattarini stefano.lattar...@gmail.com --- ChangeLog| 8 configure.ac | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index 5278d49..6abc7bc 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,13 @@ 2013-04-30 Stefano Lattarini stefano.lattar...@gmail.com (tiny change) + build: enable 'subdir-objects' and 'silent-rules' automake options + + * configure.ac (AM_INIT_AUTOMAKE): Here. The future major Automake + version 2.0 (ETA about one, one and half year from now) will likely + enable them by default, so better prepare ourselves. + +2013-04-30 Stefano Lattarini stefano.lattar...@gmail.com (tiny change) + build: require Autoconf = 2.62 and Automake = 1.11.1 Older versions of those tools should be considered fully obsolete. diff --git a/configure.ac b/configure.ac index 548bc42..c6a5482 100644 --- a/configure.ac +++ b/configure.ac @@ -26,7 +26,7 @@ AC_CONFIG_SRCDIR([vpath.c]) AC_CONFIG_HEADERS([config.h]) # Automake setup -AM_INIT_AUTOMAKE([1.11.1]) +AM_INIT_AUTOMAKE([1.11.1 subdir-objects silent-rules]) # Checks for programs. AC_PROG_CC -- 1.8.3.rc0.19.g7e6a0cc ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[PATCH 0/3] Few patches for the build system of GNU make
The second patch requires the first (as silent rules were only introduced in automake 1.11). The third patch should be fully independent. Stefano Lattarini (3): build: require Autoconf = 2.62 and Automake = 1.11.1 build: enable 'subdir-objects' and 'silent-rules' automake options build: get rid of 'HAVE_ANSI_COMPILER' C preprocessor conditional ChangeLog | 35 +++ config.ami.template | 3 --- config.h-vms.template | 3 --- config.h.W32.template | 3 --- configh.dos.template | 3 --- configure.ac | 12 +++- 6 files changed, 38 insertions(+), 21 deletions(-) -- 1.8.3.rc0.19.g7e6a0cc ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[PATCH 3/3] build: get rid of 'HAVE_ANSI_COMPILER' C preprocessor conditional
GNU make already assume C89 or later throughout the codebase, and that preprocessor conditional was no longer used anyway. * configure.ac: Remove AC_DEFINE of HAVE_ANSI_COMPILER. * config.ami.template: Remove #define of HAVE_ANSI_COMPILER. * config.h-vms.template: Likewise. * config.h.W32.template: Likewise. * configh.dos.template: Likewise. Signed-off-by: Stefano Lattarini stefano.lattar...@gmail.com --- ChangeLog | 13 + config.ami.template | 3 --- config.h-vms.template | 3 --- config.h.W32.template | 3 --- configh.dos.template | 3 --- configure.ac | 6 -- 6 files changed, 13 insertions(+), 18 deletions(-) diff --git a/ChangeLog b/ChangeLog index 6abc7bc..433ac5f 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,18 @@ 2013-04-30 Stefano Lattarini stefano.lattar...@gmail.com (tiny change) + build: get rid of 'HAVE_ANSI_COMPILER' C preprocessor conditional + + GNU make already assume C89 or later throughout the codebase, and + that preprocessor conditional was no longer used anyway. + + * configure.ac: Remove AC_DEFINE of HAVE_ANSI_COMPILER. + * config.ami.template: Remove #define of HAVE_ANSI_COMPILER. + * config.h-vms.template: Likewise. + * config.h.W32.template: Likewise. + * configh.dos.template: Likewise. + +2013-04-30 Stefano Lattarini stefano.lattar...@gmail.com (tiny change) + build: enable 'subdir-objects' and 'silent-rules' automake options * configure.ac (AM_INIT_AUTOMAKE): Here. The future major Automake diff --git a/config.ami.template b/config.ami.template index 8fe26bd..78e2e75 100644 --- a/config.ami.template +++ b/config.ami.template @@ -157,9 +157,6 @@ this program. If not, see http://www.gnu.org/licenses/. */ /* Define if the 'S_IS*' macros in sys/stat.h do not work properly. */ /* #undef STAT_MACROS_BROKEN */ -/* Define if your compiler conforms to the ANSI C standard. */ -#define HAVE_ANSI_COMPILER 1 - /* Define if you have the ANSI C header files. */ #define STDC_HEADERS diff --git a/config.h-vms.template b/config.h-vms.template index 1b99dd4..dd1aa1e 100644 --- a/config.h-vms.template +++ b/config.h-vms.template @@ -330,9 +330,6 @@ this program. If not, see http://www.gnu.org/licenses/. */ /* Define to 1 if you have the ndir.h header file. */ /* #undef HAVE_NDIR_H */ -/* Define to 1 if your compiler conforms to the ANSI C standard. */ -#define HAVE_ANSI_COMPILER 1 - /* Define to 1 if you have the stdlib.h header file. */ #define HAVE_STDLIB_H 1 diff --git a/config.h.W32.template b/config.h.W32.template index 8d59a13..82d83ad 100644 --- a/config.h.W32.template +++ b/config.h.W32.template @@ -62,9 +62,6 @@ this program. If not, see http://www.gnu.org/licenses/. */ */ /* #undef HAVE_ALLOCA_H */ -/* Define to 1 if your compiler conforms to the ANSI C standard. */ -#define HAVE_ANSI_COMPILER 1 - /* Define to 1 if you have the 'atexit' function. */ #define HAVE_ATEXIT 1 diff --git a/configh.dos.template b/configh.dos.template index 61857ab..098fef5 100644 --- a/configh.dos.template +++ b/configh.dos.template @@ -76,9 +76,6 @@ this program. If not, see http://www.gnu.org/licenses/. */ /* Define to 1 if you have the select function. */ #define HAVE_SELECT 1 -/* Define to 1 if your compiler conforms to the ANSI C standard. */ -#define HAVE_ANSI_COMPILER 1 - /* Define to 1 if you have the stricmp function. */ #define HAVE_STRICMP 1 diff --git a/configure.ac b/configure.ac index c6a5482..8431143 100644 --- a/configure.ac +++ b/configure.ac @@ -64,12 +64,6 @@ AC_HEADER_TIME AC_CHECK_HEADERS([stdlib.h locale.h unistd.h limits.h fcntl.h string.h \ memory.h sys/param.h sys/resource.h sys/time.h sys/timeb.h]) -# Set a flag if we have an ANSI C compiler -AS_IF([test $ac_cv_prog_cc_stdc != no], -[ AC_DEFINE([HAVE_ANSI_COMPILER], [1], -[Define to 1 if your compiler conforms to the ANSI C standard.]) -]) - AM_PROG_CC_C_O AC_C_CONST AC_TYPE_SIGNAL -- 1.8.3.rc0.19.g7e6a0cc ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [PATCH 2/3] build: enable 'subdir-objects' and 'silent-rules' automake options
On 04/30/2013 05:11 PM, Stefano Lattarini wrote: * configure.ac (AM_INIT_AUTOMAKE): Here. The future major Automake version 2.0 (ETA about one, one and half year from now) will likely enable them by default, so better prepare ourselves. Please drop this patch. Enabling 'subdir-objects' would require sublter changes to several other parts of the build system that I don't know how to test properly. We should only enable the 'silent-rules' option for the moment. The updated patch below does this. Sorry for the noise, Stefano 8 8 8 8 8 8 8 8 From 46623411f017a447caa8fe75b3b42ec2fbeac458 Mon Sep 17 00:00:00 2001 Message-Id: 46623411f017a447caa8fe75b3b42ec2fbeac458.1367335124.git.stefano.lattar...@gmail.com From: Stefano Lattarini stefano.lattar...@gmail.com Date: Tue, 30 Apr 2013 16:30:04 +0200 Subject: [PATCH] build: enable the 'silent-rules' automake options * configure.ac (AM_INIT_AUTOMAKE): Here. The future major Automake version 2.0 (ETA about one, one and half year from now) will enable it by default, so better prepare ourselves. Signed-off-by: Stefano Lattarini stefano.lattar...@gmail.com --- ChangeLog| 8 configure.ac | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index 5278d49..ad5356c 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,5 +1,13 @@ 2013-04-30 Stefano Lattarini stefano.lattar...@gmail.com (tiny change) + build: enable the 'silent-rules' automake options + + * configure.ac (AM_INIT_AUTOMAKE): Here. The future major Automake + version 2.0 (ETA about one, one and half year from now) will enable + it by default, so better prepare ourselves. + +2013-04-30 Stefano Lattarini stefano.lattar...@gmail.com (tiny change) + build: require Autoconf = 2.62 and Automake = 1.11.1 Older versions of those tools should be considered fully obsolete. diff --git a/configure.ac b/configure.ac index 548bc42..f2482c0 100644 --- a/configure.ac +++ b/configure.ac @@ -26,7 +26,7 @@ AC_CONFIG_SRCDIR([vpath.c]) AC_CONFIG_HEADERS([config.h]) # Automake setup -AM_INIT_AUTOMAKE([1.11.1]) +AM_INIT_AUTOMAKE([1.11.1 silent-rules]) # Checks for programs. AC_PROG_CC -- 1.8.3.rc0.19.g7e6a0cc ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: dynamic object searching (was: Re: Dynamic objects)
Date: Tue, 30 Apr 2013 08:57:23 +0100 From: Tim Murphy tnmur...@gmail.com Cc: Paul D. Smith psm...@gnu.org, bug-make@gnu.org bug-make@gnu.org Since you can't (in my recent experience) load a 64-bit DLL into a 32-bit program, the real issue is what architecture was make itself built with. That's unrelated. I was talking about the fact that load foo.so is inherently non-portable, whereas load foo or load foo$(SOEXT) (with $(SOEXT) determined automatically by Make) is much more portable. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: dynamic object searching (was: Re: Dynamic objects)
On 30 April 2013 17:28, Eli Zaretskii e...@gnu.org wrote: Since you can't (in my recent experience) load a 64-bit DLL into a 32-bit program, the real issue is what architecture was make itself built with. That's unrelated. I was talking about the fact that load foo.so is inherently non-portable, whereas load foo or load foo$(SOEXT) (with $(SOEXT) determined automatically by Make) is much more portable. Everything's non-portable - trying to solve this doesn't help at all - just complicates the rest of the process. I have to generate those plugins anyhow so there's all the non-portable code to call whatever compiler and linker are needed for that particular platform. All of that code has to specify the extension anyhow so what do I gain? i.e. I don't just have load X.dll I have to supply the recipe to build it on windows: X.dll: cl.exe /Fdo$@ # use microsoft's compiler and on Linux X.so: gcc -o $@ ... # using gcc On top of this, the extension doesn't help when you're on 32-bit platforms because it doesn't indicate whether the plugin is 32 or 64 bit. So to me it just seems like a complication because it can't solve the whole problem that make generally needs explicit effort to support builds on multiple platforms. Regards, Tim -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: dynamic object searching (was: Re: Dynamic objects)
Date: Tue, 30 Apr 2013 17:48:42 +0100 From: Tim Murphy tnmur...@gmail.com Cc: Paul D. Smith psm...@gnu.org, bug-make@gnu.org bug-make@gnu.org That's unrelated. I was talking about the fact that load foo.so is inherently non-portable, whereas load foo or load foo$(SOEXT) (with $(SOEXT) determined automatically by Make) is much more portable. Everything's non-portable - trying to solve this doesn't help at all - just complicates the rest of the process. I disagree. I have to generate those plugins anyhow so there's all the non-portable code to call whatever compiler and linker are needed for that particular platform. All of that code has to specify the extension anyhow so what do I gain? i.e. I don't just have load X.dll I have to supply the recipe to build it on windows: X.dll: cl.exe /Fdo$@ # use microsoft's compiler Not relevant: people who use GNU Make are unlikely to use cl, which comes with Nmake. I'm talking about building GNU projects using GNU Make and MinGW GCC. Every problem can be turned upside down or made overly complicated by throwing enough corner cases on it. Let's solve the 80% part and leave the other 20% to Someone Else. and on Linux X.so: gcc -o $@ ... # using gcc That's exactly what I do on Windows as well. On top of this, the extension doesn't help when you're on 32-bit platforms because it doesn't indicate whether the plugin is 32 or 64 bit. Neither does GCC. That's unrelated. So to me it just seems like a complication because it can't solve the whole problem that make generally needs explicit effort to support builds on multiple platforms. You can always refrain from using it. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: dynamic object searching (was: Re: Dynamic objects)
On Tue, 2013-04-30 at 17:48 +0100, Tim Murphy wrote: i.e. I don't just have load X.dll I have to supply the recipe to build it on windows: X.dll: cl.exe /Fdo$@ # use microsoft's compiler and on Linux X.so: gcc -o $@ ... # using gcc Actually this supports Eli's point perfectly. This is no problem. You can just put both of those rules into your makefile, and if make defines an extension EXT for the current platform you can use load X.$(EXT) and when you're on Windows it will build one way and when you're on Linux it will build the other way. However, I'm still undecided on how to handle this. I'll look at it again shortly. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: dynamic object searching (was: Re: Dynamic objects)
That stuff about only using gnu tools is far from the case. Re this usecase supporting multiple platforms. The reason why it doesn't help is that all the rest of your makefile has similar multiplatform problems which you have to address with ifeq and its friends and it is nastier if you have make second guessing your decisions. Regards, Tim On 30/04/2013, Paul Smith psm...@gnu.org wrote: On Tue, 2013-04-30 at 17:48 +0100, Tim Murphy wrote: i.e. I don't just have load X.dll I have to supply the recipe to build it on windows: X.dll: cl.exe /Fdo$@ # use microsoft's compiler and on Linux X.so: gcc -o $@ ... # using gcc Actually this supports Eli's point perfectly. This is no problem. You can just put both of those rules into your makefile, and if make defines an extension EXT for the current platform you can use load X.$(EXT) and when you're on Windows it will build one way and when you're on Linux it will build the other way. However, I'm still undecided on how to handle this. I'll look at it again shortly. -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Loading dynamic objects on Windows
I ran all of the load and loadapi tests in the test suite and found a couple of problems in the current implementation that were based on unportable assumptions. The solutions touch to some extent the platform independent code and build requirements, so I'd like to discuss them here before I actually push them. (For the impatient: see the patches near the end of this mail.) The two problems I discovered and fixed are: 1. How to build the extension shared libraries. The current code and instructions (and also the test suite) simply compile and link the extensions with -shared compiler switch. This assumes the linker lets you get away with unresolved externals (the gmk_* functions provided by Make), and let these references be resolved at run time, when the extension is loaded by dlopen/dlsym. This doesn't work on Windows. There, the linker _must_ see some symbols that allow it to resolve the references _at_link_time_. So if I try the original command used by the load test, i.e.: gcc -I../../.. -g -shared -fPIC -o testload.dll testload.c it complains about unresolved externals and bails out. There are 2 ways of solving this on Windows: . Call the gmk_* functions via pointers (so that the linker doesn't see the function calls at all). These pointers need to be assigned the addresses of the functions at the dlopen time, and they need to have the exact same type as the functions they will call. I did manage to get this to work, but found that this method has a lot of limitations. For example, since the pointer variables need to have the same names as the functions (to make it possible to compile the same extension on Unix and on Windows), these variables need to be static (to put them in gnumake.h header), and the function to be called by dlopen needs also to be static. But using static functions and variables means that an extension that has more than 1 C file will have several unrelated copied of these... So I prefer the second alternative, which is: . Use an import library. An import library is a library of stubs that provide enough info for the linker to be happy at link time, and leave the actual resolution of the references at dlopen time. GCC can create such a library for select functions that are declared with a special type. Then the import library is submitted to the linker when the extension is built, and that's it -- the extension can freely call functions exported by Make, as if they were defined in the extension. (Normally, a shared library exports its functions and an import library is used to link the main program. But our case is the opposite: we need the main program to export some functions. But the mechanism works either way.) This requires minor changes to the build_w32.bat script which builds Make, and also to the test suite (if we want it to be runnable on Windows; I ran the test by hand). Additional changes are needed to put the necessary decorations on the functions exported by Make. It also requires that the import library be distributed with the Make binaries for Windows. 2. When Make decides that it needs to remake a dynamic object, it does so with the object still loaded. Windows does not allow to overwrite a shared library that is being used by some program, so the remake fails. To fix this, I record the pointer returned by dlopen in the file structure of the dynamic object, and call dlclose before running the job to remake it. This means that any features defined by the object cannot be used for remaking the object. I hope this does not impose any real restrictions, since if the object is needed to remake itself, it means that object cannot be created from scratch. That's it. Now I let the patch talk: --- commands.c~22013-04-28 06:41:56.0 +0300 +++ commands.c 2013-04-30 11:42:46.345909500 +0300 @@ -14,6 +14,8 @@ A PARTICULAR PURPOSE. See the GNU Gener You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/. */ +#include dlfcn.h + #include makeint.h #include dep.h #include filedef.h @@ -468,6 +470,12 @@ execute_file_commands (struct file *file set_file_variables (file); + /* If this is a loaded dynamic object, unload it before remaking. + Some systems don't allow to overwrite a loaded shared + library. */ + if (file-dlopen_ptr) +dlclose (file-dlopen_ptr); + /* Start the commands running. */ new_job (file); } --- filedef.h~2 2013-04-29 06:47:10.0 +0300 +++ filedef.h 2013-04-30 11:56:47.364106900 +0300 @@ -61,6 +61,8 @@ struct file int command_flags; /* Flags OR'd in for cmds; see commands.h. */ char update_status; /* Status of the last attempt to update, or -1 if none has been made. */ +void *dlopen_ptr; /* For dynamic loaded objects: pointer to +
Re: feature request: parallel builds feature
I wasn't digressing. I was explaining the point. the concept I am trying to present as a solution to the problem of making parallel stdin for --jobs in gnu make (which currenty doesn't work and is I guess single-threaded) is to make a separate terminal or command shell for each job, such as via a generated batch file or shell script. this is as simple as I can make it. at the end of the shell script, you can put in whatever you like, such as synchronization stuff saying I am done by creating a semaphore file, a flag file. the problem would then be porting BASH to windows and other platforms in order to handle --jobs. I have learned that on a machine with 12 threads and 64GB of memory, you can have 50+ jobs running. From: Paul Smith psm...@gnu.org To: Jim Michaels jmich...@yahoo.com Cc: bug-make@gnu.org Sent: Monday, April 22, 2013 10:56 AM Subject: Re: feature request: parallel builds feature On Mon, 2013-04-22 at 00:42 -0700, Jim Michaels wrote: it currently has a problem with stdin, because at this point there is only one of those, only 1 of them gets it, and the others starve. so if your build needs stdin or creates files from the commandline using heredocs, you can't use it (check first!). you will get an error. gnu has not yet figured out a solution yet (I have, multiple shells IF you can control them... probably can't without some work creating batch files for the jobs). so there is a solution. even with a batch file, make would need some sort of way of reporting back error conditions. I think there are ways of managing that with files via presence-detect, sort of like semaphores. they should get cleared when the job ends, or when a switch is given to clear the state for that session if the session was broken with ctrl-c. well, I suppose a ctrl-c handler should still kill those terminals or cmd shells and clear up those files. what do you think? if a terminal is opened, it should be created without a window. some OS's have that option. some don't, like freeDOS, which would not have the ability to do terminals, parallel shell windows, or even the --jobs feature (but that's implementation-dependent). Please keep the mailing list CC'd. Thanks. I'm afraid I still don't understand what you're asking for here. You'll need to back up and provide a description of your needs in a clear, orderly way without digressions. Yes, it's true that GNU make only provides its stdin to one job at a time and which job gets it is essentially random. In order to address this we'd need to see a specific use-case or requirement, but my suspicion is that all such possible use-cases are better solved by a change of process at a level above what make can provide. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request: parallel builds feature
Jim Michaels wrote: I wasn't digressing. I was explaining the point. the concept I am trying to present as a solution to the problem of making parallel stdin for --jobs in gnu make (which currenty doesn't work and is I guess single-threaded) is to make a separate terminal or command shell for each job, such as via a generated batch file or shell script. this is as simple as I can make it. Who said stdin was a problem? Fundamentally the jobs spawned by make are batch jobs - they should not be requesting input from stdin in the first place. Semantically the effect of running parallel make must be identical to running serial make. You cannot guarantee this to be true if jobs are reading from stdin because stdin's supply of data is inherently serial but the order it gets read is non-deterministic in a parallel build. If the jobs you're spawning from make require input from stdin you need to rewrite those jobs. at the end of the shell script, you can put in whatever you like, such as synchronization stuff saying I am done by creating a semaphore file, a flag file. the problem would then be porting BASH to windows and other platforms in order to handle --jobs. bash has already been ported to Windows. I have learned that on a machine with 12 threads and 64GB of memory, you can have 50+ jobs running. -- *From:* Paul Smith psm...@gnu.org *To:* Jim Michaels jmich...@yahoo.com *Cc:* bug-make@gnu.org *Sent:* Monday, April 22, 2013 10:56 AM *Subject:* Re: feature request: parallel builds feature On Mon, 2013-04-22 at 00:42 -0700, Jim Michaels wrote: it currently has a problem with stdin, because at this point there is only one of those, only 1 of them gets it, and the others starve. so if your build needs stdin or creates files from the commandline using heredocs, you can't use it (check first!). you will get an error. gnu has not yet figured out a solution yet (I have, multiple shells IF you can control them... probably can't without some work creating batch files for the jobs). so there is a solution. even with a batch file, make would need some sort of way of reporting back error conditions. I think there are ways of managing that with files via presence-detect, sort of like semaphores. they should get cleared when the job ends, or when a switch is given to clear the state for that session if the session was broken with ctrl-c. well, I suppose a ctrl-c handler should still kill those terminals or cmd shells and clear up those files. what do you think? if a terminal is opened, it should be created without a window. some OS's have that option. some don't, like freeDOS, which would not have the ability to do terminals, parallel shell windows, or even the --jobs feature (but that's implementation-dependent). Please keep the mailing list CC'd. Thanks. I'm afraid I still don't understand what you're asking for here. You'll need to back up and provide a description of your needs in a clear, orderly way without digressions. Yes, it's true that GNU make only provides its stdin to one job at a time and which job gets it is essentially random. In order to address this we'd need to see a specific use-case or requirement, but my suspicion is that all such possible use-cases are better solved by a change of process at a level above what make can provide. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request: parallel builds feature
what if you in your makefile are creating files from scratch using echo, based on system configuration information? I know I have to do that in order to create XML manifest files for resources to compile and link in via resource compiler for windows builds. I can give you a sample of some of the lines I have to convert from .cmd batch files as they are now, to makefiles as the system will be eventually when I get the migration completed. echo -creating 32-bit 9x+ manifest... don't change this particular block or nothing will work. echo ^?xml version=1.0 encoding=UTF-8 standalone=yes?^32\%outfile%.%extension%.manifest echo ^assembly xmlns=urn:schemas-microsoft-com:asm.v1 manifestVersion=1.0^32\%outfile%.%extension%.manifest rem it works with description on XP, but I don't know about other platforms, so I am leaving it out. rem echo ^description^%manifest_apptitle%^/description^32\%outfile%.%extension%.manifest echo ^ms_asmv2:trustInfo xmlns:ms_asmv2=urn:schemas-microsoft-com:asm.v2^32\%outfile%.%extension%.manifest echo ^ms_asmv2:security^32\%outfile%.%extension%.manifest echo ^ms_asmv2:requestedPrivileges^32\%outfile%.%extension%.manifest echo ^ms_asmv2:requestedExecutionLevel level=%manifest_executionlevel%^32\%outfile%.%extension%.manifest rem it works on XP, but I don't know about other platforms, so I am leaving it out. rem echo ^ms_asmv2:requestedExecutionLevel level=%manifest_executionlevel% uiAccess=%manifest_uiaccess%^32\%outfile%.%extension%.manifest echo ^/ms_asmv2:requestedExecutionLevel^32\%outfile%.%extension%.manifest echo ^/ms_asmv2:requestedPrivileges^32\%outfile%.%extension%.manifest echo ^/ms_asmv2:security^32\%outfile%.%extension%.manifest echo ^/ms_asmv2:trustInfo^32\%outfile%.%extension%.manifest echo ^/assembly^32\%outfile%.%extension%.manifest echo -creating 64-bit 9x+ manifest... don't change this particular block or nothing will work. echo ^?xml version=1.0 encoding=UTF-8 standalone=yes?^64\%outfile%.%extension%.manifest echo ^assembly xmlns=urn:schemas-microsoft-com:asm.v1 manifestVersion=1.0^64\%outfile%.%extension%.manifest rem it works with description on XP, but I don't know about other platforms, so I am leaving it out. rem echo ^description^%manifest_apptitle%^/description^64\%outfile%.%extension%.manifest echo ^ms_asmv2:trustInfo xmlns:ms_asmv2=urn:schemas-microsoft-com:asm.v2^64\%outfile%.%extension%.manifest echo ^ms_asmv2:security^64\%outfile%.%extension%.manifest echo ^ms_asmv2:requestedPrivileges^64\%outfile%.%extension%.manifest echo ^ms_asmv2:requestedExecutionLevel level=%manifest_executionlevel%^64\%outfile%.%extension%.manifest rem it works on XP, but I don't know about other platforms, so I am leaving it out. rem echo ^ms_asmv2:requestedExecutionLevel level=%manifest_executionlevel% uiAccess=%manifest_uiaccess%^64\%outfile%.%extension%.manifest echo ^/ms_asmv2:requestedExecutionLevel^64\%outfile%.%extension%.manifest echo ^/ms_asmv2:requestedPrivileges^64\%outfile%.%extension%.manifest echo ^/ms_asmv2:security^64\%outfile%.%extension%.manifest echo ^/ms_asmv2:trustInfo^64\%outfile%.%extension%.manifest echo ^/assembly^64\%outfile%.%extension%.manifest there is more. there are if statements involved, etc. currently, there is no manifest tool in the compiler set for mingw-w64 or mingw. nothing is planned. From: Howard Chu h...@highlandsun.com To: Jim Michaels jmich...@yahoo.com; psm...@gnu.org psm...@gnu.org Cc: bug-make@gnu.org bug-make@gnu.org Sent: Tuesday, April 30, 2013 6:55 PM Subject: Re: feature request: parallel builds feature Jim Michaels wrote: I wasn't digressing. I was explaining the point. the concept I am trying to present as a solution to the problem of making parallel stdin for --jobs in gnu make (which currenty doesn't work and is I guess single-threaded) is to make a separate terminal or command shell for each job, such as via a generated batch file or shell script. this is as simple as I can make it. Who said stdin was a problem? Fundamentally the jobs spawned by make are batch jobs - they should not be requesting input from stdin in the first place. Semantically the effect of running parallel make must be identical to running serial make. You cannot guarantee this to be true if jobs are reading from stdin because stdin's supply of data is inherently serial but the order it gets read is non-deterministic in a parallel build. If the jobs you're spawning from make require input from stdin you need to rewrite those jobs. at the end of the shell script, you can put in whatever you like, such as synchronization stuff saying I am
Re: feature request: parallel builds feature
Jim Michaels wrote: what if you in your makefile are creating files from scratch using echo, based on system configuration information? I know I have to do that in order to create XML manifest files for resources to compile and link in via resource compiler for windows builds. echo writes to stdout. That has nothing to do with stdin. Looks to me like you have no F'ing idea what you're talking about. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: feature request: parallel builds feature
you only have to read the documentation to know that it doesn't support parallel stdin. you would know that if you had followed the thread. here's the documention, read around the middle. http://www.gnu.org/software/make/manual/html_node/Parallel.html From: Howard Chu h...@highlandsun.com To: Jim Michaels jmich...@yahoo.com; psm...@gnu.org psm...@gnu.org Cc: bug-make@gnu.org bug-make@gnu.org Sent: Tuesday, April 30, 2013 6:55 PM Subject: Re: feature request: parallel builds feature Jim Michaels wrote: I wasn't digressing. I was explaining the point. the concept I am trying to present as a solution to the problem of making parallel stdin for --jobs in gnu make (which currenty doesn't work and is I guess single-threaded) is to make a separate terminal or command shell for each job, such as via a generated batch file or shell script. this is as simple as I can make it. Who said stdin was a problem? Fundamentally the jobs spawned by make are batch jobs - they should not be requesting input from stdin in the first place. Semantically the effect of running parallel make must be identical to running serial make. You cannot guarantee this to be true if jobs are reading from stdin because stdin's supply of data is inherently serial but the order it gets read is non-deterministic in a parallel build. If the jobs you're spawning from make require input from stdin you need to rewrite those jobs. at the end of the shell script, you can put in whatever you like, such as synchronization stuff saying I am done by creating a semaphore file, a flag file. the problem would then be porting BASH to windows and other platforms in order to handle --jobs. bash has already been ported to Windows. I have learned that on a machine with 12 threads and 64GB of memory, you can have 50+ jobs running. -- *From:* Paul Smith psm...@gnu.org *To:* Jim Michaels jmich...@yahoo.com *Cc:* bug-make@gnu.org *Sent:* Monday, April 22, 2013 10:56 AM *Subject:* Re: feature request: parallel builds feature On Mon, 2013-04-22 at 00:42 -0700, Jim Michaels wrote: it currently has a problem with stdin, because at this point there is only one of those, only 1 of them gets it, and the others starve. so if your build needs stdin or creates files from the commandline using heredocs, you can't use it (check first!). you will get an error. gnu has not yet figured out a solution yet (I have, multiple shells IF you can control them... probably can't without some work creating batch files for the jobs). so there is a solution. even with a batch file, make would need some sort of way of reporting back error conditions. I think there are ways of managing that with files via presence-detect, sort of like semaphores. they should get cleared when the job ends, or when a switch is given to clear the state for that session if the session was broken with ctrl-c. well, I suppose a ctrl-c handler should still kill those terminals or cmd shells and clear up those files. what do you think? if a terminal is opened, it should be created without a window. some OS's have that option. some don't, like freeDOS, which would not have the ability to do terminals, parallel shell windows, or even the --jobs feature (but that's implementation-dependent). Please keep the mailing list CC'd. Thanks. I'm afraid I still don't understand what you're asking for here. You'll need to back up and provide a description of your needs in a clear, orderly way without digressions. Yes, it's true that GNU make only provides its stdin to one job at a time and which job gets it is essentially random. In order to address this we'd need to see a specific use-case or requirement, but my suspicion is that all such possible use-cases are better solved by a change of process at a level above what make can provide. ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
