automake -vs- huge projects
Tom Fitzsimmons (CCd) has been working on upgrading libgcj to use newer auto* tools. This has gone swimmingly, except one problem with automake. A little background. libgcj is pretty big. It has 2,243 .java files at the moment. Previously it has been using its own slightly hacked automake 1.4. It used to use its own % rules to handle compiling .java (since 1.4 couldn't do this). It is part of GCC, which recently decided as a project that requiring GNU make is ok. We have to use subdir-objects, both because nobody wants 2000 .o files in . and because we have unavoidable basename clashes between .java files. Also, we use a single top-level Makefile.am, as it is way more convenient. The problem is, automake generates an explicit rule for each compilation. Our resulting Makefile.in is nearly 9 megabytes. This is really much too large -- compare to 200K with automake 1.4. One idea we had for a fix is to introduce a new gnu-make option that would allow automake to generate code relying on GNU make. Then we could replace all those rules with a single %.o: %.java. I thought this probably wouldn't be a popular idea, but figured I'd send it past you just in case. We definitely don't want to have a local fork of automake any more -- it is too much of a pain, and GCC as a whole is trying to update and standardize on one set of tools for the whole tree. Any other ideas for how to fix this problem? In absence of a real fix, another option for us is to just hand-write our Makefile. I'd really rather not do that, either. Though I suppose if we start from Automake's output it won't be too awful -- we can still preserve dependency tracking and the like. The long term maintenance on this is likely to be hard though. Tom
Re: automake -vs- huge projects
On 16 Dec 2003, Tom Tromey wrote: The problem is, automake generates an explicit rule for each compilation. Our resulting Makefile.in is nearly 9 megabytes. This is really much too large -- compare to 200K with automake 1.4. I have observed the same problem myself. It is not language dependent. At some point (perhaps when subdirectory builds were introduced) Automake diverged from using common rules, and started to insert per-target rules. It seems that a bit more smarts can be used to get close to automake 1.4 efficiency. Per-subdirectory rules and definitions can be added in order to significantly reduce the amount of redundant code, and to re-enable the capability to usefully override parts of the default Makefile.in. One idea we had for a fix is to introduce a new gnu-make option that would allow automake to generate code relying on GNU make. Then we could replace all those rules with a single %.o: %.java. This doesn't seem to be necessary. Bob == Bob Friesenhahn [EMAIL PROTECTED] http://www.simplesystems.org/users/bfriesen
Re: automake -vs- huge projects
%% Bob Friesenhahn [EMAIL PROTECTED] writes: bf Per-subdirectory rules and definitions can be added in order to bf significantly reduce the amount of redundant code, and to bf re-enable the capability to usefully override parts of the default bf Makefile.in. Not if you want to continue to generate portable makefiles. There is no way in POSIX make (for example) to generate a target in a subdirectory using a suffix rule. -- --- Paul D. Smith [EMAIL PROTECTED] Find some GNU make tips at: http://www.gnu.org http://make.paulandlesley.org Please remain calm...I may be mad, but I am a professional. --Mad Scientist
Re: automake -vs- huge projects
On 16 Dec 2003, Paul D. Smith wrote: %% Bob Friesenhahn [EMAIL PROTECTED] writes: bf Per-subdirectory rules and definitions can be added in order to bf significantly reduce the amount of redundant code, and to bf re-enable the capability to usefully override parts of the default bf Makefile.in. Not if you want to continue to generate portable makefiles. There is no way in POSIX make (for example) to generate a target in a subdirectory using a suffix rule. Bummer. I wasn't sure about that. At the very least, an effort could be made to decrease the Makefile size by making maximum use of macro substitutions. Bob == Bob Friesenhahn [EMAIL PROTECTED] http://www.simplesystems.org/users/bfriesen
Re: automake -vs- huge projects
On Tue, 16 Dec 2003 09:49:44 -0700, Tom Tromey wrote: Tom Fitzsimmons (CCd) has been working on upgrading libgcj to use newer auto* tools. This has gone swimmingly, except one problem with automake. A little background. libgcj is pretty big. It has 2,243 .java files at the moment. Previously it has been using its own slightly hacked automake 1.4. It used to use its own % rules to handle compiling .java (since 1.4 couldn't do this). It is part of GCC, which recently decided as a project that requiring GNU make is ok. We have to use subdir-objects, both because nobody wants 2000 .o files in . and because we have unavoidable basename clashes between .java files. Also, we use a single top-level Makefile.am, as it is way more convenient. The problem is, automake generates an explicit rule for each compilation. Our resulting Makefile.in is nearly 9 megabytes. This is really much too large -- compare to 200K with automake 1.4. One specifc problem I noticed is that automake is generating three explicit rules for each target. For example, for one .java file, these rules are generated: java/awt/print/Pageable.o: java/awt/print/Pageable.java @am__fastdepGCJ_TRUE@ if $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -MT java/awt/print/Pageable.o -MD -MP -MF java/awt/print/$(DEPDIR)/Pageable.Tpo \ @am__fastdepGCJ_TRUE@ -c -o java/awt/print/Pageable.o `test -f 'java/awt/print/Pageable.java' || echo '$(srcdir)/'`java/awt/print/Pageable.java; \ @am__fastdepGCJ_TRUE@ then mv -f java/awt/print/$(DEPDIR)/Pageable.Tpo java/awt/print/$(DEPDIR)/Pageable.Po; \ @am__fastdepGCJ_TRUE@ else rm -f java/awt/print/$(DEPDIR)/Pageable.Tpo; exit 1; \ @am__fastdepGCJ_TRUE@ fi @AMDEP_TRUE@@am__fastdepGCJ_FALSE@ source='java/awt/print/Pageable.java' object='java/awt/print/Pageable.o' libtool=no @AMDEPBACKSLASH@ @AMDEP_TRUE@@am__fastdepGCJ_FALSE@ depfile='java/awt/print/$(DEPDIR)/Pageable.Po' tmpdepfile='java/awt/print/$(DEPDIR)/Pageable.TPo' @AMDEPBACKSLASH@ @AMDEP_TRUE@@am__fastdepGCJ_FALSE@ $(GCJDEPMODE) $(depcomp) @AMDEPBACKSLASH@ @am__fastdepGCJ_FALSE@ $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -c -o java/awt/print/Pageable.o `test -f 'java/awt/print/Pageable.java' || echo '$(srcdir)/'`java/awt/print/Pageable.java java/awt/print/Pageable.obj: java/awt/print/Pageable.java @am__fastdepGCJ_TRUE@ if $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -MT java/awt/print/Pageable.obj -MD -MP -MF java/awt/print/$(DEPDIR)/Pageable.Tpo \ @am__fastdepGCJ_TRUE@ -c -o java/awt/print/Pageable.obj `if test -f 'java/awt/print/Pageable.java'; then $(CYGPATH_W) 'java/awt/print/Pageable.java'; else $(CYGPATH_W) '$(srcdir)/java/awt/print/Pageable.java'; fi`; \ @am__fastdepGCJ_TRUE@ then mv -f java/awt/print/$(DEPDIR)/Pageable.Tpo java/awt/print/$(DEPDIR)/Pageable.Po; \ @am__fastdepGCJ_TRUE@ else rm -f java/awt/print/$(DEPDIR)/Pageable.Tpo; exit 1; \ @am__fastdepGCJ_TRUE@ fi @AMDEP_TRUE@@am__fastdepGCJ_FALSE@ source='java/awt/print/Pageable.java' object='java/awt/print/Pageable.obj' libtool=no @AMDEPBACKSLASH@ @AMDEP_TRUE@@am__fastdepGCJ_FALSE@ depfile='java/awt/print/$(DEPDIR)/Pageable.Po' tmpdepfile='java/awt/print/$(DEPDIR)/Pageable.TPo' @AMDEPBACKSLASH@ @AMDEP_TRUE@@am__fastdepGCJ_FALSE@ $(GCJDEPMODE) $(depcomp) @AMDEPBACKSLASH@ @am__fastdepGCJ_FALSE@ $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -c -o java/awt/print/Pageable.obj `if test -f 'java/awt/print/Pageable.java'; then $(CYGPATH_W) 'java/awt/print/Pageable.java'; else $(CYGPATH_W) '$(srcdir)/java/awt/print/Pageable.java'; fi` java/awt/print/Pageable.lo: java/awt/print/Pageable.java @am__fastdepGCJ_TRUE@ if $(LIBTOOL) --mode=compile $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -MT java/awt/print/Pageable.lo -MD -MP -MF java/awt/print/$(DEPDIR)/Pageable.Tpo \ @am__fastdepGCJ_TRUE@ -c -o java/awt/print/Pageable.lo `test -f 'java/awt/print/Pageable.java' || echo '$(srcdir)/'`java/awt/print/Pageable.java; \ @am__fastdepGCJ_TRUE@ then mv -f java/awt/print/$(DEPDIR)/Pageable.Tpo java/awt/print/$(DEPDIR)/Pageable.Plo; \ @am__fastdepGCJ_TRUE@ else rm -f java/awt/print/$(DEPDIR)/Pageable.Tpo; exit 1; \ @am__fastdepGCJ_TRUE@ fi @AMDEP_TRUE@@am__fastdepGCJ_FALSE@ source='java/awt/print/Pageable.java' object='java/awt/print/Pageable.lo' libtool=yes @AMDEPBACKSLASH@ @AMDEP_TRUE@@am__fastdepGCJ_FALSE@ depfile='java/awt/print/$(DEPDIR)/Pageable.Plo' tmpdepfile='java/awt/print/$(DEPDIR)/Pageable.TPlo' @AMDEPBACKSLASH@ @AMDEP_TRUE@@am__fastdepGCJ_FALSE@ $(GCJDEPMODE) $(depcomp) @AMDEPBACKSLASH@ @am__fastdepGCJ_FALSE@ $(LIBTOOL) --mode=compile $(GCJ) $(AM_GCJFLAGS) $(GCJFLAGS) -c -o java/awt/print/Pageable.lo `test -f 'java/awt/print/Pageable.java' || echo '$(srcdir)/'`java/awt/print/Pageable.java I would only expect rules for .lo files, since we are generating libtool libraries. Getting rid of the other targets would cut out *a lot* of the bulk. Tom
Re: automake -vs- huge projects
Another thing that would help reduce Makefile size is to introduce synonyms for subdirectory paths. For example, java/awt/print/$(DEPDIR)/ could be reduced to $(am_f123). This would of course obfusticate the Makefile, so it should be made optional, or allow the user to specify part of the name so it is still comprehensible. Bob == Bob Friesenhahn [EMAIL PROTECTED] http://www.simplesystems.org/users/bfriesen
Re: automake -vs- huge projects
I would like to point out that my previous gripe about the unnecessary translation of Automake target names (particularly when the target includes a path), and the subsequent proposal to add user-level target synonyms, dovetails nicely with the need to reduce the Makefile size. The synonyms can help eliminate the size increase caused by the directory path itself. For typical subdirectory path lengths, this could result in a 40% Makefile size reduction. Bob == Bob Friesenhahn [EMAIL PROTECTED] http://www.simplesystems.org/users/bfriesen
Re: automake -vs- huge projects
Tom == Tom Tromey [EMAIL PROTECTED] writes: [...] Tom We have to use subdir-objects [...] Tom Also, we use a single top-level Makefile.am [...] Tom The problem is, automake generates an explicit rule for each Tom compilation. Our resulting Makefile.in is nearly 9 megabytes. This Tom is really much too large -- compare to 200K with automake 1.4. Tom One idea we had for a fix is to introduce a new gnu-make option that Tom would allow automake to generate code relying on GNU make. Then we Tom could replace all those rules with a single %.o: %.java. Couldn't we use the (existing) .java.o: inference rule in this case? Actually, is there a difference between `%.o: %.java' and `.java.o:' beside portability? -- I'm not asking about the general % construction, just about this precise case where both sides are expected to be in the same directory. I've done some limited testing (with GNU make, BSD make, OSF1/Tru64 Make, and HP-UX make) and .src.dest: rules appear to work for files in subdirectory even in VPATH configurations. I wonder if there are Make implementations where this does not work. So the simplest part of the fix would be to disable the output of explicit rule for subdirectory sources without per-target flags when subdir-objects is used. The other side of the coin is that dependency tracking will not work anymore, because the dependency stuff for subdir/X.o should go into subdir/.deps/X.Po but the default suffix rule will put it in ./.deps/subdir/X.Po. I don't see an easy way to fix this, unless we add some clumsy shell computation in the suffix rules (while this can probably be folded into depcomp when it is used, it does not seem to fit well in fastdep rules). Note that this issue is unrelated to the %.o:%.java vs. .java.o: choice. [...] -- Alexandre Duret-Lutz
Re: automake -vs- huge projects
Bob == Bob Friesenhahn [EMAIL PROTECTED] writes: Bob Another thing that would help reduce Makefile size is to Bob introduce synonyms for subdirectory paths. Better: LZW compression using Makefile variables. Any taker? wink -- Alexandre Duret-Lutz
Re: automake -vs- huge projects
On Tue, 16 Dec 2003, Alexandre Duret-Lutz wrote: Bob == Bob Friesenhahn [EMAIL PROTECTED] writes: Bob Another thing that would help reduce Makefile size is to Bob introduce synonyms for subdirectory paths. Better: LZW compression using Makefile variables. Any taker? wink Is this your way of poking fun at my idea? :-( Bob == Bob Friesenhahn [EMAIL PROTECTED] http://www.simplesystems.org/users/bfriesen
Re: automake -vs- huge projects
Thomas == Thomas Fitzsimmons [EMAIL PROTECTED] writes: [...] Thomas One specifc problem I noticed is that automake is Thomas generating three explicit rules for each target. For Thomas example, for one .java file, these rules are generated: [...] Thomas I would only expect rules for .lo files, since we are Thomas generating libtool libraries. Getting rid of the other Thomas targets would cut out *a lot* of the bulk. Indeed! I think we can construct cases that will break if any of the .o or .lo rules are left out, for instance bin_PROGRAMS = foo foo_SOURCES = a.c b.c foo_LDADD = @MOREFOO@ lib_LTLIBRARIES = libbar.la libbar_la_SOURCES = 1.c 2.c libbar_la_LIBADD = @MOREBAR@ now think about @MOREFOO@ being substituted into 1.o or 2.o, and @MOREBAR@ being replaced by a.la or b.la... However it seems fair to require that the above snippet be rewritten as bin_PROGRAMS = foo foo_SOURCES = a.c b.c foo_LDADD = @MOREFOO@ EXTRA_foo_SOURCES = 1.c 2.c lib_LTLIBRARIES = libbar.la libbar_la_SOURCES = 1.c 2.c libbar_la_LDADD = @MOREBAR@ EXTRA_libbar_la_SOURCES = a.c b.c like the manual says (oh well, I hope it says so). And in this second example it is clear which rules Automake needs to output. Furthermore, generally it does not work to compile both the .o and .lo objects of a source file (in the last example Automake is expected to warn that these files are being built both with and without Libtool), so it sounds safe to remove the rules which are not used. -- Alexandre Duret-Lutz
cialis is the new drug
Save over 80% on New official drugs approved by FDA Come In and Visit us
Autoconf 2.59 available from ftp.gnu.org
Autoconf 2.59 is now available from the following URLs: ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.59.tar.gz ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.59.tar.bz2 Here are the MD5 checksums: d4d45eaa1769d45e59dcb131a4af17a0 autoconf-2.59.tar.gz 1ee40f7a676b3cfdc0e3f7cd81551b5f autoconf-2.59.tar.bz2 There is only one major change in Autoconf 2.59: ** ac_abs_builddir etc. Absolute paths were actually relative in 2.58. Autoconf 2.59 was originally released on 2003-11-12; see http://mail.gnu.org/archive/html/autotools-announce/2003-11/msg3.html. Unfortunately it wasn't copied to ftp.gnu.org until today due to problems that we had with the new ftp.gnu.org upload procedures. Sorry about the delay.
Re: automake -vs- huge projects
adl == Alexandre Duret-Lutz [EMAIL PROTECTED] writes: adl Couldn't we use the (existing) .java.o: inference rule in this adl case? Actually, is there a difference between `%.o: %.java' and adl `.java.o:' beside portability? -- I'm not asking about the adl general % construction, just about this precise case where both adl sides are expected to be in the same directory. My recollection is that we tried this and ran into real problems with it. I don't remember what they are any more. There's an old thread on this: http://sources.redhat.com/ml/automake/1999-04/threads.html#00033 At this point I still believed that these suffix rules would work. I couldn't find the point where we changed things. adl So the simplest part of the fix would be to disable the output adl of explicit rule for subdirectory sources without per-target adl flags when subdir-objects is used. Yeah. That would help us a lot, but... adl The other side of the coin is that dependency tracking will not adl work anymore, because the dependency stuff for subdir/X.o should adl go into subdir/.deps/X.Po but the default suffix rule will put adl it in ./.deps/subdir/X.Po. I don't see an easy way to fix this, adl unless we add some clumsy shell computation in the suffix rules adl (while this can probably be folded into depcomp when it is used, adl it does not seem to fit well in fastdep rules). Hmm, maybe this is the issue from way back. We definitely want to keep dependency tracking. This is pretty important. fastdep would be great, since we know we'll always be using gcc. If somehow dropping fastdep would get the Makefile.in to a reasonable size, though, I'd be in favor of that. Alternatively running sed or whatever once or twice before the compilation isn't going to hurt as much as having a 9M Makefile.in. So we could just do the rewrite in the suffix rule and pay the cost. adl Note that this issue is unrelated to the %.o:%.java vs. .java.o: adl choice. Not completely, since GNU make might give us handy macros that would let us do this transformation in-line. Tom
Re: automake -vs- huge projects
adl == Alexandre Duret-Lutz [EMAIL PROTECTED] writes: adl Furthermore, generally it does not work to compile both the .o adl and .lo objects of a source file (in the last example Automake adl is expected to warn that these files are being built both with adl and without Libtool), so it sounds safe to remove the rules adl which are not used. This would be a great change, but unfortunately it only gets us down to 3M, which is still about 6x too large. Tom
