Re: Avoid -Wl,--whole-archive
On Tue, 21 May 2019, Thomas Jahns wrote: On 5/16/19 5:00 PM, Bob Friesenhahn wrote: Convenience libraries are evil. Convenience libraries slow down your build process dramatically and they cause 'make' not to be aware of the actual underlying dependencies, so that it does much more work than is required each type you invoke 'make'. If you can enumerate the actual dependencies in the Makefile and avoid needless intermediate product generation, then you will achive a maximally parallel build which builds much faster on modern systems. Except when they don't actually slow you down: ar and ranlib provide for a lookup table that in my experience is beneficial to build times, especially when you have a network filesystem, where access to individual file system objects can be more expensive. These features are not used by convenience libraries. The fact that convenience libraries store object files in an archive file is just an artifact of its implementation. An alternative would have been to use pax or tar, but these require creating the whole file from scratch and are not as portably available (for compilation) as 'ar'. When a convenience library is linked with a shared library, all of its contained object files may be first extracted, and then the list of object files is supplied to the linker. This happens when -Wl,--whole-archive is not available. Given that convenience libraries do use the 'ar' format and are compiled as PIC when building shared libraries (if needed), an alternative way of referencing the archive file from the Makefile would likely avoid the -Wl,--whole-archive problem. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt___ https://lists.gnu.org/mailman/listinfo/libtool
Re: Avoid -Wl,--whole-archive
On 5/16/19 5:00 PM, Bob Friesenhahn wrote: Convenience libraries are evil. Convenience libraries slow down your build process dramatically and they cause 'make' not to be aware of the actual underlying dependencies, so that it does much more work than is required each type you invoke 'make'. If you can enumerate the actual dependencies in the Makefile and avoid needless intermediate product generation, then you will achive a maximally parallel build which builds much faster on modern systems. Except when they don't actually slow you down: ar and ranlib provide for a lookup table that in my experience is beneficial to build times, especially when you have a network filesystem, where access to individual file system objects can be more expensive. Regards, Thomas ___ https://lists.gnu.org/mailman/listinfo/libtool
Re: Avoid -Wl,--whole-archive
On 5/16/19 5:00 PM, Bob Friesenhahn wrote: > On Thu, 16 May 2019, Tim Rühsen wrote: > >> Hi, >> >> at GNU Wget2 we are just splitting a (shared) library into several >> smaller ones, grouped by functionality. >> >> We depend on gnulib and have a single libgnu.a. Each of the shared >> libraries just need a certain set of functions from libgnu.a. >> >> To avoid adding everything from libgnu.a to each of the libraries, we >> would like to avoid "-Wl,--whole-archive ../lib/.libs/libgnu.a >> -Wl,--no-whole-archive". > > There is no requirement to use "convenience" libraries. People who do > things due to "convenience" are often classified as "lazy". :-) Of course I am lazy - that's why I am developer ;-) Building things to make my life easier... > If you have time to re-do your build structure, then I recommend using a > non-recursive build and explicitly listing the objects which are needed > by each library in the single Makefile. Objects common to multiple > libraries will then be built just once and supplied to the linker as a > list of object files rather than fed to libtool like an "archive" file > which is then split into object files actually supplied to the linker. > > Convenience libraries are evil. Convenience libraries slow down your > build process dramatically and they cause 'make' not to be aware of the > actual underlying dependencies, so that it does much more work than is > required each type you invoke 'make'. If you can enumerate the actual > dependencies in the Makefile and avoid needless intermediate product > generation, then you will achive a maximally parallel build which builds > much faster on modern systems. I couldn't see any speed penalty so far. Indeed, enumerating the object files (.lo in this case) was my first approach. But we are talking about gnulib - the number of object files is different in each environment and also change over time. I immediately ran into issues when I pushed that code to our CI. The exact enumeration of objects is simply not foreseeable and thus not portable. Your are right for projects with a deterministic number of objects... But for projects that use gnulib *and* build more than one shared library a '-no-whole-archive' flag would great. > If your project is already fully built, then typing 'make' again should > return almost immediately without doing any work at all. If your > project does any work at all due to typing 'make' a second time, then it > is defective. A second 'make' always immediately returns - if not I would change my build recipe. Regards, Tim signature.asc Description: OpenPGP digital signature ___ https://lists.gnu.org/mailman/listinfo/libtool
Re: Avoid -Wl,--whole-archive
On Thu, 16 May 2019, Tim Rühsen wrote: Hi, at GNU Wget2 we are just splitting a (shared) library into several smaller ones, grouped by functionality. We depend on gnulib and have a single libgnu.a. Each of the shared libraries just need a certain set of functions from libgnu.a. To avoid adding everything from libgnu.a to each of the libraries, we would like to avoid "-Wl,--whole-archive ../lib/.libs/libgnu.a -Wl,--no-whole-archive". There is no requirement to use "convenience" libraries. People who do things due to "convenience" are often classified as "lazy". :-) If you have time to re-do your build structure, then I recommend using a non-recursive build and explicitly listing the objects which are needed by each library in the single Makefile. Objects common to multiple libraries will then be built just once and supplied to the linker as a list of object files rather than fed to libtool like an "archive" file which is then split into object files actually supplied to the linker. Convenience libraries are evil. Convenience libraries slow down your build process dramatically and they cause 'make' not to be aware of the actual underlying dependencies, so that it does much more work than is required each type you invoke 'make'. If you can enumerate the actual dependencies in the Makefile and avoid needless intermediate product generation, then you will achive a maximally parallel build which builds much faster on modern systems. If your project is already fully built, then typing 'make' again should return almost immediately without doing any work at all. If your project does any work at all due to typing 'make' a second time, then it is defective. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt___ https://lists.gnu.org/mailman/listinfo/libtool
Avoid -Wl,--whole-archive
Hi, at GNU Wget2 we are just splitting a (shared) library into several smaller ones, grouped by functionality. We depend on gnulib and have a single libgnu.a. Each of the shared libraries just need a certain set of functions from libgnu.a. To avoid adding everything from libgnu.a to each of the libraries, we would like to avoid "-Wl,--whole-archive ../lib/.libs/libgnu.a -Wl,--no-whole-archive". There several ways to work around this situation (e.g. creating one libgnu.a per library). The preferred one would be to have a new option to libtool, '-no-whole-archive'. A simple proof of concept is this patch: --- libtool 2019-05-16 12:46:46.551527339 +0200 +++ libtool.no-whole-archive2019-05-16 12:41:25.612879712 +0200 @@ -7700,6 +7700,11 @@ continue ;; + -no-whole-archive) +whole_archive_flag_spec="\$convenience " +continue +;; + -objectlist) prev=objectlist continue Maybe you have a better idea !? With this patch a current snippet from Makefile.am looks like lib_LTLIBRARIES += libwget_progress.la libwget_progress_la_SOURCES = bar.c libwget_progress_la_CPPFLAGS = $(libwget_la_CPPFLAGS) libwget_progress_la_LIBADD = ../lib/libgnu.la libwget_progress_la_LDFLAGS = $(libwget_la_LDFLAGS) -no-whole-archive If wanted, I am willing to write a real patch (some hints what to consider would be nice - the codebase is currently unknown to me). Regards, Tim signature.asc Description: OpenPGP digital signature ___ https://lists.gnu.org/mailman/listinfo/libtool
