Re: Dump the database to a makefile and invoke make on the dumped makefile.
On Sat, 2012-02-18 at 18:46 +, Tim Murphy wrote: The option you need is: -p, --print-data-base Print make's internal database. This is the (only) place to start but just to warn you: the output of this option was not designed to be used this way and we don't guarantee that the format etc. will not change in future versions of make. This output is intended for human consumption and debugging, not specifically for further automated processing. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Dump the database to a makefile and invoke make on the dumped makefile.
Hi, In one of my projects I've written a small make script which includes given makefiles and then prints out valid make code with definitions of all new variables found in the included makefiles (based on the value of $(.VARIABLES)). You can see the source code of the script here: http://code.google.com/p/embox/source/browse/trunk/embox/mk/script/mk-cache.mk?r=4825 And a usage example: http://code.google.com/p/embox/source/browse/trunk/embox/mk/load.mk?r=4827 Hope this will help. 2012/2/18 Torbjörn Svensson az...@svenskalinuxforeningen.se -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello list, The short version Is there any way to make make dump it's database to a makefile and then reuse that dumped makefile to build the source? The long version In one of our projects at work, we use a rather complex set of macros[1] to make it easier for the developers to define how a component is to be built. The developer creates a file called 'makerules.mk' and enters data into a few predefined variable names, like $(component1_SRCS), $(component1_INCL), $(component1_LIBS) where component1 is the name of the resulting executable of library. When invoking make, the macros are expanded based on the data in those variables to a set of rules and then make just does it's magic on those. As the source code usually is put in a tree, the number of makerules.mk files that the developers are entering data into can be rather large and as a result, the part of the build process that expands the macros takes a lot of time. For example, for one of our components, the expansion takes roughly 40 seconds even as there is nothing to left to do. To save some time, I'm now thinking that our build process could be divided into a few stages. Stage 1 Verify the timestamps on each and every makerules.mk file and if any of the timestamps is newer than the complete makefile, it needs to be regenerated. Possible way to implement: Could be resolved with another temporary makefile that just sets the complete makefile as target and have all the makerules.mk files as dependencies. Stage 2 Generate the complete makefile and expand all the macros. Possible way to implement: The option -p sounds interesting. The problem with -p is that the dumped database also includes -p in MFLAGS and MAKEFLAGS so it's not possible to just invoke make on the complete makefile without first stripping of this option. At first glans, I would like an option like -p but it should not be added to the dumped database, but I'm a bit worried that not adding it there would screw up recursive make. I also would like the option to print to a file rather than stdout as stdout could be used in the makefile that you build and thus make will be confused if invoked on the dumped file. Stage 3 Execute make on the complete makefile. To sum up this question, is it possible to do this without patching make? If not, is it something that you would be interested in including in an official version of make? Thank you in advance. // Torbjörn [1] http://gitorious.org/bobuild/bob/trees/master -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8/3CUACgkQeY7jmtvbDP2S6gCgyOrwVsZVUy2x0gQ4gDb6G4qd 2coAoI2QgDNAHX84hNy2RArrPd7liSE4 =rnlv -END PGP SIGNATURE- ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make -- Best regards, Eldar Sh. Abusalimov ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35525] buffer overflow in make
Update of bug #35525 (project make): Status:None = Fixed Assigned to:None = psmith Open/Closed:Open = Closed Fixed Release:None = CVS ___ Follow-up Comment #1: Fix promoted to CVS. Thanks! ___ Reply to this item at: http://savannah.gnu.org/bugs/?35525 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: Dump the database to a makefile and invoke make on the dumped makefile.
Hi, That's well understood and the problem at the moment is that $(info) statements etc are executed and pollute the output. It does point out an area of scalability though because in our testing it made an immense difference. Some of our macros are perhaps 50k (effectively although that's really a bunch of macros) and might get invoked perhaps in the low 10s of thousands of times. Parsing the makefile becomes a bottleneck because make doesn't do this in parallel and parsing get a worse and worse problem the larger your build if you want make to have a total dependency view. In an ideal world, make would have a total view of dependencies but read and parse makefiles in parallel. It would also have ways of generalising dependencies (trading precise-incrementalism for lower memory usage and complexity as packaging systems do when any change however insubstantial in package A triggers a rebuild of package B - not precise but easy to compute). You'd choose what you wanted when you started your build - it might even adapt depending on how big the build turned out to be. Another capability would be to learn how to deal cleverly with common dependencies to save processing and memory. e.g. when we use cpp to generate deps on our code, make chokes with the huge number of them but since many header files are common to all our code, one can chop memory usage by about 5 if one finds the common dependencies and puts them behind a phony target and then only supplies target: prereq rules for the dependencies that are unique to a particular file. e.g. COMMON_HEADERS: frodo.h stdio.h myfile1.cpp: COMMON_HEADERS fred.h bob.h myfile2.cpp: COMMON_HEADERS alice.h jane.h tiffany.h myfile3.cpp: COMMON_HEADERS alice.h jane.h tiffany.h This is also an example of an instance where one could trade precision for lower complexity (big O) - one could make all headers common - would cause your build system to rebuild some things totally unnecessarily but it would require very little dependency checking and in a large enough build with the right structure that might be best. It would be more sophisticated to have different gradations e.g. COMMON_H BOY_H GIRL_H and so on. It would be much more useful if the build tool could work out how to make efficient sets like this automatically. e.g. COMMON_H: frodo.h stdio.h BOY_H: fred.h bob.h GIRL_H alice.h jane.h tiffany.h myfile1.cpp: COMMON_H BOY_H myfile2.cpp: COMMON_H GIRL_H myfile3.cpp: COMMON_H GIRL_H If you know that program X uses library Y and you also know that library Y owns a set of header files then you could have a great win by introducing a dependency that says object files of library X depend on the headers of library Y. This is also a generalisation which will often be untrue (not all headers in library Y affect all object files in X) in a conservative way but it cuts down on the amount of decisions f make has to manage by a lot and at a certain scale it looks like a great idea even if people who build little programs with 100 compiles would not be impressed with what it did to their incremental builds. One can do a lot of this my designing makefiles in a certain way but then you are committed. Working well on the micro scale is sometimes at odds with working well at the macro scale and an ideal make tool would somehow help one to deal with that without one having to maintain different and very complicated makefiles for each situation. Regards, Tim On 26 February 2012 19:51, Paul Smith psm...@gnu.org wrote: On Sat, 2012-02-18 at 18:46 +, Tim Murphy wrote: The option you need is: -p, --print-data-base Print make's internal database. This is the (only) place to start but just to warn you: the output of this option was not designed to be used this way and we don't guarantee that the format etc. will not change in future versions of make. This output is intended for human consumption and debugging, not specifically for further automated processing. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
[bug #35485] New $(.MFDIR) built-in variable
Follow-up Comment #2, bug #35485 (project make): I don't think this implementation is appropriate. The value of MFDIR will change wildly over time if makefiles are included which can't possibly be what you want. For example: .MFDIR = $(abspath $(dir $(lastword $(MAKEFILE_LIST $(info MFDIR = $(.MFDIR)) include foo/bar.mk $(info MFDIR = $(.MFDIR)) all:;: will show: MFDIR = /tmp MFDIR = /tmp/foo : which doesn't seem like what you'd want. In order to properly implement this, it would need to be handled as a special variable that would be reset properly as make started and stopped processing included makefiles. Or maybe I'm misunderstanding what you're looking for... perhaps a more specific definition would help. ___ Reply to this item at: http://savannah.gnu.org/bugs/?35485 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #35485] New $(.MFDIR) built-in variable
On Thu, 2012-02-09 at 17:25 +, Tim Murphy wrote: I also think that it can be expensive to append things onto very long lists in make but that's just suspicion and I really need to take a proper look at it instead of making accusations. Appending is not very expensive. Make doesn't know anything about lists internally. Everything in make is a string. If you append to a variable then make basically does a strlen() of the string, makes sure there's enough memory for the new content in the buffer (or if not, reallocs enough space), then strcat()'s the value. Or the equivalent. So I suppose yes, in some ways the longer the string the longer it takes to do the strlen() but this should be very fast I'd expect. Much of the string manipulation in make (and make does a LOT of string manipulation) is bog-stupid strlen, strcat, strcpy, strdup which gets expensive in some situations. I've considered implementing some kind of fancy string encoding . The thing that DOES get noticeably more complex is manipulating the variable as if it were a list, with functions like firstword, lastword, etc. because those functions must first chop the string up into words, then operate on those, and that happens every time the function is invoked. -- --- Paul D. Smith psm...@gnu.org Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net Please remain calm...I may be mad, but I am a professional. --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
Re: [bug #35485] New $(.MFDIR) built-in variable
Paul Smith wrote: On Thu, 2012-02-09 at 17:25 +, Tim Murphy wrote: I also think that it can be expensive to append things onto very long lists in make but that's just suspicion and I really need to take a proper look at it instead of making accusations. Appending is not very expensive. Make doesn't know anything about lists internally. Everything in make is a string. If you append to a variable then make basically does a strlen() of the string, makes sure there's enough memory for the new content in the buffer (or if not, reallocs enough space), then strcat()'s the value. Or the equivalent. So I suppose yes, in some ways the longer the string the longer it takes to do the strlen() but this should be very fast I'd expect. Much of the string manipulation in make (and make does a LOT of string manipulation) is bog-stupid strlen, strcat, strcpy, strdup which gets expensive in some situations. I've considered implementing some kind of fancy string encoding . You've just described an O(n^2) behavior. This is definitely a drag; but it really depends on how heavily you make use of macros. In OpenLDAP 2.0 70% of our execution time was in libc; 35% in malloc/free and 35% in strlen/strcat and friends. I instituted a top-down replacement of all bare char *s in the code to struct bervals (char *ptr; int len) and eliminated all use of strlen in the code and our code base was immediately 60% faster. A number of other profiling and optimization exercises made OpenLDAP 2.1 *20* times faster than OpenLDAP 2.0, but getting rid of moronic uses of libc was the number 1 priority. The thing that DOES get noticeably more complex is manipulating the variable as if it were a list, with functions like firstword, lastword, etc. because those functions must first chop the string up into words, then operate on those, and that happens every time the function is invoked. If that happens a lot, then you should simply construct the lists once and keep them... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org https://lists.gnu.org/mailman/listinfo/bug-make
