Re: [patch #9687] bugfix: make -export-dynamic imply --whole-archive
On Sun, 21 Nov 2021, Alex Ameen wrote: Overall I think we're on the same page. I understand that `libtool' is ultimately intended to provide cross-platform consistency as a portability wrapper around each platform's various compilers, linkers, and loaders. It is certainly not my intention to promote a specific tool or platform over another. I am glad that our new maintainer is philosophically on the same page and also has excellent skills. I think that (similar to the influence of GNU/FSF philosophies on software development) libtool should help guide application developers to to use the most portable approaches while achieving their own objectives. From this standpoint, libtool (and Autotools in general) are not just 'tools' (like 'ld') but help guide users (developers) so that if they follow guidelines, and what the tools intend to promote, their applications are most likely to be portable and still work well. If the development/porting problem is looked at using Venn diagrams, then there would be a proportion of features/solutions in common amongst modern targets and those should be the features/solutions which are promoted by Autotools. In some cases the description of what is wanted can be at a high enough level that the desired low-level behavior can be accomplished entirely differently on different targets (because of how they work). In Autotools, the above has worked well, although there have been many complaints over the years about libtool behavior regarding explicit dependencies (via ".la" files) whereas leveraging implicit dependencies are usually prefered by distribution maintainers. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
Re: [patch #9687] bugfix: make -export-dynamic imply --whole-archive
Howdy Bob, I totally agree with you. `libtool' definitely should not be creating solutions that only work with a particular linker in mind, and at a high level I think it should ultimately work to provide a sense of consistency across a variety of linkers. In this case my concern is about ease of use for developers who might be confused about `libtool', `ld', and other link-editors having options with identical names which behave in very different ways. Here we have the flags `--whole-archive', `--no-whole-archive', and `--export-dynamic' which exist across almost all popular link-editors, GNU `ld' ( `ld.bfd' and `ld.gold' ), LLVM's `lld', Solaris' `ld', etc which have a consistent behavior. With that in mind I think having the flag `-export-dynamic' in `libtool' behave differently from "the norm" in discreet ways would likely cause confusion. To my knowledge HP UX and AIX are the only two systems which lack these flags ( I believe this is actually the type of thing that `libtool' should aim to port to those platforms ). And yeah I agree that convenience libraries are wonky, and that linking static archives into dynamic libraries and executables is a tricky task. The good news on this front is that this is actually the area of linking and loading that I have the most experience in, and I'm familiar with many ( but not all ) of the pitfalls that pop up on Linux especially, but also with AIX and Windows. The reason I recommended the use of a convenience library in this case is because `libtool' already knows to use `--whole-archive' and `--no-whole-archive' for these libraries, so it is a convenient way to accomplish those goals in existing releases. I took a look at the option parsing for `libtool --mode=link' in depth yesterday and I know why it has trouble handling `--whole-archive' and `--no-whole-archive' flags, fixing the issue isn't trivial just because of how the parser is written but I plan to get these flags to behave "as expected" for folks who are used to this feature in common link-editors. Overall I think we're on the same page. I understand that `libtool' is ultimately intended to provide cross-platform consistency as a portability wrapper around each platform's various compilers, linkers, and loaders. It is certainly not my intention to promote a specific tool or platform over another. -Alex On 11/21/21 8:32 AM, Bob Friesenhahn wrote: On Sat, 20 Nov 2021, Alex Ameen wrote: Thanks for the follow up, this gave me a much clearer idea of the underlying issue that you're trying to solve. I really do appreciate you taking the time to help improve `libtool'. You're absolutely right that `libtool' completely bungles the use of `--whole-archive' and `--no-whole-archive' which I see as a high priority issue. In several build systems I've built in the past I have needed to apply manual patches to `libtool' to work around this, and I plan on merging changes to fix this soon ( I'm taking my time to create test cases ). I don't pretend to understand the associated issues, but please take care that libtool does not promote solutions which only work with GNU ld (or linkers designed to perfectly emulate GNU ld) on a limited set of targets. Libtool is a portability solution to encourage development of software which is able to compile and run on many targets, even if the authors of the software have never experienced those targets. Linking static libraries into shared libraries is a complex topic and is a reason why libtool offers the clumsy/inefficient "convenience library" mechanism since it assures that the objects were compiled properly to be used in the library they are linked into. Bob
Re: [patch #9687] bugfix: make -export-dynamic imply --whole-archive
On Sat, 20 Nov 2021, Alex Ameen wrote: Thanks for the follow up, this gave me a much clearer idea of the underlying issue that you're trying to solve. I really do appreciate you taking the time to help improve `libtool'. You're absolutely right that `libtool' completely bungles the use of `--whole-archive' and `--no-whole-archive' which I see as a high priority issue. In several build systems I've built in the past I have needed to apply manual patches to `libtool' to work around this, and I plan on merging changes to fix this soon ( I'm taking my time to create test cases ). I don't pretend to understand the associated issues, but please take care that libtool does not promote solutions which only work with GNU ld (or linkers designed to perfectly emulate GNU ld) on a limited set of targets. Libtool is a portability solution to encourage development of software which is able to compile and run on many targets, even if the authors of the software have never experienced those targets. Linking static libraries into shared libraries is a complex topic and is a reason why libtool offers the clumsy/inefficient "convenience library" mechanism since it assures that the objects were compiled properly to be used in the library they are linked into. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
Re: [patch #9687] bugfix: make -export-dynamic imply --whole-archive
Thanks for the follow up, this gave me a much clearer idea of the underlying issue that you're trying to solve. I really do appreciate you taking the time to help improve `libtool'. You're absolutely right that `libtool' completely bungles the use of `--whole-archive' and `--no-whole-archive' which I see as a high priority issue. In several build systems I've built in the past I have needed to apply manual patches to `libtool' to work around this, and I plan on merging changes to fix this soon ( I'm taking my time to create test cases ). To answer your question about why `--whole-archive' and `--no-whole-archive' are useful flags in general : some code-bases create libraries with multiple builds of objects for differing configurations, which rely the "cherry picking" behavior of the linker in combination with dummy symbols to automatically select the appropriate form on an object for a given link. This is sometimes combined with `--as-needed' and `--no-as-needed' for more fine grained control. These use cases are niche, but they can be incredibly useful - and if I remember correctly I believe that `libpthread.so', GNU `libc.so.6', `libgcc', and several other "core" libraries use this behavior in combination with linker-scripts to "choose the right objects" ( the nitty gritty of how this is accomplished is an interesting rabbit hole to explore ). This reliance on "cherry picking" hopefully sheds light on why `--whole-archive' is not the default behavior, and why those flags exist. Now, having said that I agree that 99.9% of the time when I link a `libfoo.a' into a binary ( especially a shared object ), I usually want the entire archive to link - so `--whole-archive' is a flag I use frequently and it's a major pain that `libtool' doesn't handle it correctly at the moment. We could got back and forth on whether `-export-dynamic' should imply `--whole-archive' or not, but where I think we absolutely have agreement is that `libtool' needs to start handling `--whole-archive' and `--no-whole-archive' properly. Do you think if those flags were fixed, would invoking `libtool --mode=link -export-dynamic OBJECTS --whole-archive libfoo.a --no-whole-archive libbar.a ...;' be a suitable solution to your issue? Having thought about it more I think my gut concern was basically backwards-compatibility with existing `-export-dynamic' usage. On your last comment about all of the linking junk being super complex : I totally agree with you, and despite exploring niche `ld' junk for years now I am always learning. I think at bottom this is why I'm cautious about having `-export-dynamic' imply `--whole-archive' automatically; I don't personally know every possible use case for these flags and I would hate to unknowingly break someone's code-base by silently changing the behavior of `-export-dynamic'. On 11/20/21 10:28 AM, David Lamparter wrote: Follow-up Comment #2, patch #9687 (project libtool): Well, it's now 3 years later, and my memory of details on this is pretty much gone, but let's try. No warranty on anything here, I'm trying to reconstruct. [comment #1 comment #1:] Could you maybe elaborate on your use case? Perhaps there's something that I'm missing. Any executable linked with `-export-dynamic` that links static libraries may end up missing globally visible symbols if they are not used by the executable itself. If the program then tries to load a module that needs these symbols, it fails. In my opinion : the existing behavior both by `ld' and `libtool' is appropriate. Implying `--whole-archive' for dependency libraries in with `-export-dynamic' will prevent users from intentionally localizing symbols. `--whole-archive` has no impact on symbol visibility; I don't follow at all what you're trying to say here. A binary linked with `-export-dynamic` is essentially a shared library, and the `ld` docs state: "This is normally used to turn an archive file into a shared library, forcing every object to be included in the resulting shared library." I agree `ld` is doing the appropriate thing here, but `libtool` isn't. A binary with `-export-dynamic` needs to export ALL of its global symbols from ALL of its files for dynamically loaded modules to use. It is essentially a shared library at the same time and needs to be treated as such. NB: `--whole-archive` does NOT make all symbols globally visible! This is about *objects*, i.e. files. If you have a file getting linked in that has some global symbols, but none of them are used in the binary itself - `ld` will drop the entire file. That's the wrong thing to do for both creating a shared library as well as for creating an executable with `-export-dynamic`. I see these flags as having distinct use cases. I'll note that, the need for using `--whole-archive' with `ld' isn't necessarily intuitive to users, so I see the appeal of using it in many situations - but adding additional variance between `libtool' and
