How to speed up 'automake'
Hi all:
I have this cross-compiling Autoconf/Automake project:
https://github.com/rdiez/JtagDue/tree/master/Project
This is just an example project, I have others that are bigger, and I rebuild
many of them automatically overnight. That is why I am looking at speeding up
the Autotools steps.
When I run this command, I get the following trace:
time automake --verbose --warnings=all --add-missing --copy
automake: thread 0: running WARNINGS=none autoconf
--trace=AC_CANONICAL_BUILD:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_CANONICAL_HOST:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_CANONICAL_TARGET:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_CONFIG_AUX_DIR:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_CONFIG_FILES:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_CONFIG_HEADERS:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_CONFIG_LIBOBJ_DIR:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_CONFIG_LINKS:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_FC_SRCEXT:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_INIT:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_LIBSOURCE:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_REQUIRE_AUX_FILE:\$f:\$l::\$d::\$n::\${::}%
--trace=AC_SUBST_TRACE:\$f:\$l::\$d::\$n::\${::}%
--trace=AM_AUTOMAKE_VERSION:\$f:\$l::\$d::\$n::\${::}%
--trace=AM_CONDITIONAL:\$f:\$l::\$d::\$n::\${::}%
--trace=AM_EXTRA_RECURSIVE_TARGETS:\$f:\$l::\$d::\$n::\${::}%
--trace=AM_GNU_GETTEXT:\$f:\$l::\$d::\$n::\${::}%
--trace=AM_GNU_GETTEXT_INTL_SUBDIR:\$f:\$l::\$d::\$n::\${::}%
--trace=AM_INIT_AUTOMAKE:\$f:\$l::\$d::\$n::\${::}%
--trace=AM_MAINTAINER_MODE:\$f:\$l::\$d::\$n::\${::}%
--trace=AM_PROG_AR:\$f:\$l::\$d::\$n::\${::}%
--trace=AM_PROG_MKDIR_P:\$f:\$l::\$d::\$n::\${::}%
--trace=LT_SUPPORTED_TAG:\$f:\$l::\$d::\$n::\${::}%
--trace=_AM_COND_ELSE:\$f:\$l::\$d::\$n::\${::}%
--trace=_AM_COND_ENDIF:\$f:\$l::\$d::\$n::\${::}%
--trace=_AM_COND_IF:\$f:\$l::\$d::\$n::\${::}%
--trace=_AM_SUBST_NOTMAKE:\$f:\$l::\$d::\$n::\${::}%
--trace=_LT_AC_TAGCONFIG:\$f:\$l::\$d::\$n::\${::}%
--trace=m4_include:\$f:\$l::\$d::\$n::\${::}%
--trace=m4_sinclude:\$f:\$l::\$d::\$n::\${::}%
--trace=sinclude:\$f:\$l::\$d::\$n::\${::}%
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/header-vars.am
automake: thread 0: reading Makefile.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/configure.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/progs.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/libs.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/library.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/program.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/compile.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/depend.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/depend2.am
automake: thread 0: Sources ending in .S become .o
automake: thread 0: Sources ending in .S become .obj
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/lang-compile.am
automake: thread 0: Sources ending in .c become .o
automake: thread 0: Sources ending in .c become .obj
automake: thread 0: Sources ending in .cpp become .o
automake: thread 0: Sources ending in .cpp become .obj
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/texinfos.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/data.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/inst-vars.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/tags.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/distdir.am
automake: thread 0: reading
/home/rdiez/rdiez/LocalSoftware/Autotools/autoconf-2.71-automake-1.16.5-libtool-2.4.6-bin/share/automake-1.16/am/footer.am
automake: thread 0: reading
bug#45756: Prepending '+' to the recipe line when linking with GCC's -flto=jobserver
[Resending in hopes it will attach to the new bug. --karl] [...] At any rate, it would be extremely helpful to have a minimal-as-possible runnable (automake-able) example showing the case where the + needs to be prepended. rdiez, can you create such a mini-project? > [...] I normally use Autoconf, and I do not understand very much the separation between Autoconf and Automake. I do not know who is responsible for the generation of the makefile rules to link the executable. Either Autoconf or Automake must decide that GCC is not just used for compiling each object file, but also for linking, and that rule is not visible in the makefile.am file. Of course, such a linking rule does not user $(MAKE), and there is no '+' prefix, so the GNU Make jobserver file descriptors will not be passed to child processes. This is documented in the GNU Make manual. You do not need any special demo project for this. Just take any existing Automake project written in C or C++, and use these compilation flags in configure.ac : AM_CFLAGS="-flto=jobserver" AM_CXXFLAGS="-flto=jobserver" If you run the makefile with "make -j 2", GCC will receive environment variable MAKEFLAGS with a setting like "--jobserver-fds=xxx", but GNU Make will close the file descriptors mentioned there before executing the rule and running GCC. This issue is not visible in GCC yet due to this bug I reported: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94330 I am not sure how I can demonstrate this in a project, there is not actually much to demonstrate. This is not just an issue while linking. Like I said, any stage, including compilation of a single object file, could use the GNU Make jobs server. So there should be a global option in Autoconf or Automake to prepend a '+' to all generated rules. In fact, you would have thought that this should be the default, because any tool, at any stage, could decide in the future to go multithread, and there is no reason to ignore an existing GNU Make job server to limit the overall system load. Regards, rdiez
Re: Prepending '+' to the recipe line when linking with GCC's -flto=jobserver
[...] At any rate, it would be extremely helpful to have a minimal-as-possible runnable (automake-able) example showing the case where the + needs to be prepended. rdiez, can you create such a mini-project? > [...] I normally use Autoconf, and I do not understand very much the separation between Autoconf and Automake. I do not know who is responsible for the generation of the makefile rules to link the executable. Either Autoconf or Automake must decide that GCC is not just used for compiling each object file, but also for linking, and that rule is not visible in the makefile.am file. Of course, such a linking rule does not user $(MAKE), and there is no '+' prefix, so the GNU Make jobserver file descriptors will not be passed to child processes. This is documented in the GNU Make manual. You do not need any special demo project for this. Just take any existing Automake project written in C or C++, and use these compilation flags in configure.ac : AM_CFLAGS="-flto=jobserver" AM_CXXFLAGS="-flto=jobserver" If you run the makefile with "make -j 2", GCC will receive environment variable MAKEFLAGS with a setting like "--jobserver-fds=xxx", but GNU Make will close the file descriptors mentioned there before executing the rule and running GCC. This issue is not visible in GCC yet due to this bug I reported: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94330 I am not sure how I can demonstrate this in a project, there is not actually much to demonstrate. This is not just an issue while linking. Like I said, any stage, including compilation of a single object file, could use the GNU Make jobs server. So there should be a global option in Autoconf or Automake to prepend a '+' to all generated rules. In fact, you would have thought that this should be the default, because any tool, at any stage, could decide in the future to go multithread, and there is no reason to ignore an existing GNU Make job server to limit the overall system load. Regards, rdiez
Prepending '+' to the recipe line when linking with GCC's -flto=jobserver
Hi all: I am developing firmware similar to this one: https://github.com/rdiez/JtagDue First, I am building a GCC 8.3.0 cross-compiler toolchain for ARM Cortex-M4F on Ubuntu 18.04.4 LTS, and I am then using it to build some embedded software with a cross-compiling Autoconf project. GCC introduced some time ago option -flto=jobserver in order to use the GNU Make jobserver when parallelising LTO builds. I am actually building many such Autoconf projects from a top-level makefile, so using the top-level jobserver prevents overloading the system with too many parallel jobs. That is what the jobserver was designed for. When doing a recursive make, you need to place a '+' character at the beginning of the recipe line in order to let GNU Make pass the jobserver file descriptors to the child processes. Alternatively, if you reference variable $(MAKE) inside the recipe like, you get the same effect. Otherwise, GNU Make does not let child processes inherit those file descriptors (or so I heard, probably with "close fd on exec"). In my top-level makefile, I added those '+' prefixes manually, so all Autoconf-generated makefiles are building in parallel but using the top-level jobserver. The trouble is, invoking GCC in the linking phase with -flto=jobserver is actually a similar "recursive make" situation: GCC needs to inherit those jobserver file descriptors. However, Autoconf/Automake does not generate that '+' prefix for the linking recipe. Is there a way to overcome this limitation? This will probably become a general issue in the future, as each stage could theoretically benefit from multi-core CPUs. See for example: https://gcc.gnu.org/wiki/ParallelGcc Thanks in advance, rdiez
Please, please fix the subdir-objects out-of-tree bug
Hi Automake devs: I need to build some files outside of the build repository, and I cannot use option 'subdir-objects' because of an Automake bug, so I constantly get many warnings like this with newer Automake versions: warning: source file '$(BLAH_BLAH_DIR)/src/blah_blah.cpp' is in a subdirectory, The (apparent) problem with 'subdir-objects' is that ./configure creates a directory literally named $(whater_variable_you_used), instead of substituting the variable with its value. None of the work-arounds is satisfactory. For instance, I am developing, so I cannot disable dependency tracking. Here is an example project of mine with this problem: https://github.com/rdiez/JtagDue I am compiling vendor-specific external sources like ARM's CMSIS and Atmel Software Framework into an .a library and then linking it to many projects, and I have different sandboxes that build out-of-tree with different configurations. Other work-arounds like copying the sources to each project's directory tree would drastically slow down compilation times. This Automake issue has been known for years. Could someone please, please fix this bug? It annoys me everday. The Internet is full of the warning message above. I wonder how many people have wasted time on this. I wish I had the necessary shell/Automake/M4 skills to do that myself. You will find detailed information on this problem if you follow the links here: "Autotools build fails due to subdir-objects option in AM_INIT_AUTOMAKE" http://stackoverflow.com/questions/21609580/autotools-build-fails-due-to-subdir-objects-option-in-am-init-automake If fixing it is not feasible, could you disable that warning? Or at least provide a flag to disable that one warning? Or is there such a flag already? I couldn't find a way to disable just one warning, and I do not want to disable all warnings. Regards, rdiez
Re: Recompiling all sources when the makefile changes
While it's your project and you can do this if you want, making all objects depend on the makefile sounds like a really silly idea. Nobody wants to spend 30 minutes recompiling because they added one source file to a library, or because they added an additional test case. I suggest choosing some specific features to test, and make things depend on THAT. For example, you could rebuild if CFLAGS are changed by storing the previous setting of CFLAGS in a file, update that file if the current setting differs, then add prerequisites on THAT file. That is not so easy. First of all, each build type will have its own flags. For example, C will use CPPFLAGS and CFLAGS, C++ will use CPPFLAGS and CXXFLAGS, and the assembler, linker and librarian have different flags too. And that is just for a C/C++/asm project. Then it may happen that each target uses its own flags. I think Automake allows this. I am not sure if you can specify different flags for a single .cpp file, but you can do that per program or library. Furthermore, CFLAGS and co are probably not the only things that matter. The autotools most like choose other compiler arguments based on other macros or options. For example, AM_SILENT_RULES only changes the arguments passed to GNU Make, and that sort of change may be enough to justify a recompile. Your example is valid: if you add a C++ source file to a program, there is no need to recompile any other files. The problem is how you figure out which changes to the makefile should trigger a recompile, which ones only need a relink, and what modifications do not actually affect the build. You could argue that this is just a rebuild optimisation problem, and rebuilding everything after any little makefile change is always the safest way. The main reason why I am using the autotools is not portability, but automatic dependency handling. However, it turns out that I need to manually care about makefile and compiler flag dependencies. So I feel the autotools (in this case Automake) are letting me down. If I change some property in a Microsoft Visual Studio C++ project, the development environment automatically knows what needs to be rebuilt and what does not. I miss that kind of intelligence in the autotools. At the very least, I would expect a safe flag/option that makes sure the outcome is always correct, even it it means rebuilding too often. Regards, rdiez
Re: Recompiling all sources when the makefile changes
First of all, thanks for your e-mail. If configure is changing something, an easy and reliable option is to ensure that it changes config.h (or some other configuration header), which will naturally cause a rebuild of files that include the header. This is not as straightforward as it sounds. My project currently does not even have a config.h file, and even if it had one, there is no reliable way to make sure that all sources end up including that file. It may also be a different compiler flag that has no effect on the config.h file whatsoever. It is safer if all the object files depend on the makefile. It's not documented in the manual, but there are automake variables that can help you add prerequisites to your object files. They have the form mumble_OBJECTS and contain the list of object files for a particular program or library, corresponding to mumble_SOURCES. Thanks, that looks promising, I'll give it a go. The fact that is not documented makes me worry that it may change in the future without further notice. Rebuilding all sources after a makefile changes is a basic requirement. Is there any chance for somebody from the Automake team to either document this mumble_OBJECTS variable, or add some way to trigger this kind of rebuild automatically? Or at least add a PROGRAM_OBJECTS_EXTRA_DEPENDENCIES option. The object filenames are reliable as long as you don't change any settings (like subdir-objects, but also other things). So it is safer to use mumble_OBJECTS in my opinion. I am not sure what you mean here. Do you mean that, if I turn on subdir-objects or some other option, then mumble_OBJECTS may not work any more? Please copy me on any answers, as I am not subscribed to this list. Thanks in advance, rdiez
subdir-objects generates subdirs with unsubstituted variable names
Hi there: I have the following open-source project that cross-compiles to an embedded target: https://github.com/rdiez/JtagDue I am trying to keep automake happy in order to prevent the following kind of warnings: EmptyFirmware/Makefile.am:41: warning: source file '$(CMSIS_SRC_TEMPLATE_DIR)/system_sam3x.c' is in a subdirectory, EmptyFirmware/Makefile.am:41: but option 'subdir-objects' is disabled After all, as far as I am concerned, automake can create as many subdirectories as it wants in the build directory. However, when I add option subdir-objects, automate generates subdirectories like this when compiling: drwxrwxr-x 3 rdiez rdiez 4,096 Oct 6 20:53 $(BARE_METAL_SUPPORT_DIR) -rw-rw-r-- 1 rdiez rdiez 30,584 Oct 6 20:53 Makefile Note that the directory name contains a GNU Make variable that has not been substituted. Later on, linking fails because it does substitute properly when looking for the file. For example: Makefile:424: /home/rdiez/rdiez/arduino/asf-standalone-archive-3.19.0.95/sam/utils/cmsis/sam3x/source/templates/.deps/libcmsis_a-system_sam3x.Po: No such file or directory I looked further around the in build directory, and I found more directories like that: $ find . -name \$* ./JtagDue/JtagDue-obj/CmsisMakefile/$(CMSIS_BASE_DIR) ./JtagDue/JtagDue-obj/AsfForEmptyFirmware/$(ASF_BASEDIR) ./JtagDue/JtagDue-obj/JtagFirmware/$(BARE_METAL_SUPPORT_DIR) ./JtagDue/JtagDue-obj/AsfForJtagFirmware/$(LIBSAM_PMC_DIR) ./JtagDue/JtagDue-obj/AsfForJtagFirmware/$(ASF_BASEDIR) ./JtagDue/JtagDue-obj/AsfForJtagFirmware/$(LIBSAM_USB_UDC_DIR) ./JtagDue/JtagDue-obj/AsfForJtagFirmware/$(LIBSAM_USB_UDI_CDC_DEVICE_DIR) ./JtagDue/JtagDue-obj/EmptyFirmware/$(BARE_METAL_SUPPORT_DIR) Note that the $(ASF_BASEDIR) etc. variables have not been expanded either when creating those directories. Those variables do work well all over the place, as I am using AC_SUBST(ASF_BASEDIR) and so on in the top-level configure.ac file. I guess autoconf or automake are not dealing correctly with those absolute paths outside the project directory when they contain variables. I am using Ubuntu 14.04, which comes with autoconf 2.69 and automake 1.14.1. I am building out of the source tree. I could try with VPATH, but then there will be ambiguities if 2 files are called the same but live in different subdirectories in the source tree. The Atmel Software Framework (ASF) is a C source code library that, as far as I know, does not provide an autotool building environment. My project does use the autotools. Some C++ source files are in my project's directories, but some files live in the ASF directory, which can be anywhere on the system (user configurable, normally an absolute path). I thought I could just reference those ASF .c files from my autotools project, but with the latest versions I am getting warnings like the ones above. This is how I am referencing the files: libsam_a_SOURCES = \ $(LIBSAM_PMC_DIR)/pmc.c \ $(LIBSAM_PMC_DIR)/sleep.c \ $(LIBSAM_PIO_DIR)/pio.c \ $(ASF_BASEDIR)/sam/drivers/wdt/wdt.c \ $(ASF_BASEDIR)/sam/drivers/rstc/rstc.c \ $(ASF_BASEDIR)/sam/drivers/uart/uart.c \ $(ASF_BASEDIR)/common/utils/interrupt/interrupt_sam_nvic.c \ $(ASF_BASEDIR)/common/services/clock/sam3x/sysclk.c I wonder if that is the right way to reference those files. Maybe I should create a library for each foreign directory referenced that way. This kind of question has been asked before, but I have not found a satisfactory answer yet. For example: Source files in different directory? http://gnu-automake.7480.n7.nabble.com/Source-files-in-different-directory-td3483.html Here too: https://www.mail-archive.com/automake@gnu.org/msg18893.html Think, for example, of sources being in a read-only mount (like a CD-ROM). Other files do live in my project directory, but are special and get referenced directly from the main makefile: # See the comments in the Bare Metal Support library's Makefile.am # for information about why these files are compiled here. emptydue_elf_SOURCES += $(BARE_METAL_SUPPORT_DIR)/NewlibSyscalls.cpp emptydue_elf_SOURCES += $(BARE_METAL_SUPPORT_DIR)/crt0.cpp Is this a bug in automake? Or is there a way to keep Automake happy? Please copy me on any answers, as I am not subscribed to this list. Thanks in advance, rdiez
Setting ACLOCAL_AMFLAGS with ':=' vs '='
Hallo there: If I add this line to my Makefile.am (and I make sure that the 'm4' subdir is created beforehand), then it works as intended: ACLOCAL_AMFLAGS = -I m4 However, if I use this syntax: ACLOCAL_AMFLAGS := -I m4 Then I get the following warning: libtoolize: Consider adding `-I m4' to ACLOCAL_AMFLAGS in Makefile.am. I don't think using ':=' instead of '=' should matter here. Or am I missing something? I took a quick look in the Automake manual and did not find anything in this respect. I am using Automake vesion 1.14.1 on Ubuntu 14.04. Some background information to this matter: There are 2 variable flavors in GNU Make: recursively expanded variables (A=B) and Simply expanded variables (A:=B). I prefer the latter. From the GNU Make manual: Simply expanded variables generally make complicated makefile programming more predictable because they work like variables in most programming languages. They allow you to redefine a variable using its own value (or its value processed in some way by one of the expansion functions) and to use the expansion functions much more efficiently (see Functions for Transforming Text). That flavor is now a POSIX standard (with syntax ::=), so it should be portable too (at least in the future). Thanks in advance, rdiez
