Re: [patch #9687] bugfix: make -export-dynamic imply --whole-archive

[Bob Friesenhahn]

On Sun, 21 Nov 2021, Alex Ameen wrote:


Overall I think we're on the same page. I understand that `libtool' is 
ultimately intended to provide cross-platform consistency as a portability 
wrapper around each platform's various compilers, linkers, and loaders. It is 
certainly not my intention to promote a specific tool or platform over 
another.


I am glad that our new maintainer is philosophically on the same page 
and also has excellent skills.


I think that (similar to the influence of GNU/FSF philosophies on 
software development) libtool should help guide application developers 
to to use the most portable approaches while achieving their own 
objectives.  From this standpoint, libtool (and Autotools in general) 
are not just 'tools' (like 'ld') but help guide users (developers) so 
that if they follow guidelines, and what the tools intend to promote, 
their applications are most likely to be portable and still work well.


If the development/porting problem is looked at using Venn diagrams, 
then there would be a proportion of features/solutions in common 
amongst modern targets and those should be the features/solutions 
which are promoted by Autotools.  In some cases the description of 
what is wanted can be at a high enough level that the desired 
low-level behavior can be accomplished entirely differently on 
different targets (because of how they work).


In Autotools, the above has worked well, although there have been many 
complaints over the years about libtool behavior regarding explicit 
dependencies (via ".la" files) whereas leveraging implicit 
dependencies are usually prefered by distribution maintainers.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: [patch #9687] bugfix: make -export-dynamic imply --whole-archive

[Alex Ameen]

Howdy Bob,

I totally agree with you. `libtool' definitely should not be creating 
solutions that only work with a particular linker in mind, and at a high 
level I think it should ultimately work to provide a sense of 
consistency across a variety of linkers.


In this case my concern is about ease of use for developers who might be 
confused about `libtool', `ld', and other link-editors having options 
with identical names which behave in very different ways. Here we have 
the flags `--whole-archive', `--no-whole-archive', and 
`--export-dynamic' which exist across almost all popular link-editors, 
GNU `ld' ( `ld.bfd' and `ld.gold' ), LLVM's `lld', Solaris' `ld', etc 
which have a consistent behavior. With that in mind I think having the 
flag `-export-dynamic' in `libtool' behave differently from "the norm" 
in discreet ways would likely cause confusion. To my knowledge HP UX and 
AIX are the only two systems which lack these flags ( I believe this is 
actually the type of thing that `libtool' should aim to port to those 
platforms ).


And yeah I agree that convenience libraries are wonky, and that linking 
static archives into dynamic libraries and executables is a tricky task. 
The good news on this front is that this is actually the area of linking 
and loading that I have the most experience in, and I'm familiar with 
many ( but not all ) of the pitfalls that pop up on Linux especially, 
but also with AIX and Windows. The reason I recommended the use of a 
convenience library in this case is because `libtool' already knows to 
use `--whole-archive' and `--no-whole-archive' for these libraries, so 
it is a convenient way to accomplish those goals in existing releases.


I took a look at the option parsing for `libtool --mode=link' in depth 
yesterday and I know why it has trouble handling `--whole-archive' and 
`--no-whole-archive' flags, fixing the issue isn't trivial just because 
of how the parser is written but I plan to get these flags to behave "as 
expected" for folks who are used to this feature in common link-editors.


Overall I think we're on the same page. I understand that `libtool' is 
ultimately intended to provide cross-platform consistency as a 
portability wrapper around each platform's various compilers, linkers, 
and loaders. It is certainly not my intention to promote a specific tool 
or platform over another.


-Alex


On 11/21/21 8:32 AM, Bob Friesenhahn wrote:

On Sat, 20 Nov 2021, Alex Ameen wrote:

Thanks for the follow up, this gave me a much clearer idea of the 
underlying issue that you're trying to solve. I really do appreciate 
you taking the time to help improve `libtool'.


You're absolutely right that `libtool' completely bungles the use of 
`--whole-archive' and `--no-whole-archive' which I see as a high 
priority issue. In several build systems I've built in the past I 
have needed to apply manual patches to `libtool' to work around this, 
and I plan on merging changes to fix this soon ( I'm taking my time 
to create test cases ).


I don't pretend to understand the associated issues, but please take 
care that libtool does not promote solutions which only work with GNU 
ld (or linkers designed to perfectly emulate GNU ld) on a limited set 
of targets.


Libtool is a portability solution to encourage development of software 
which is able to compile and run on many targets, even if the authors 
of the software have never experienced those targets.


Linking static libraries into shared libraries is a complex topic and 
is a reason why libtool offers the clumsy/inefficient "convenience 
library" mechanism since it assures that the objects were compiled 
properly to be used in the library they are linked into.


Bob

Re: [patch #9687] bugfix: make -export-dynamic imply --whole-archive

[Bob Friesenhahn]

On Sat, 20 Nov 2021, Alex Ameen wrote:

Thanks for the follow up, this gave me a much clearer idea of the underlying 
issue that you're trying to solve. I really do appreciate you taking the time 
to help improve `libtool'.


You're absolutely right that `libtool' completely bungles the use of 
`--whole-archive' and `--no-whole-archive' which I see as a high priority 
issue. In several build systems I've built in the past I have needed to apply 
manual patches to `libtool' to work around this, and I plan on merging 
changes to fix this soon ( I'm taking my time to create test cases ).


I don't pretend to understand the associated issues, but please take 
care that libtool does not promote solutions which only work with GNU 
ld (or linkers designed to perfectly emulate GNU ld) on a limited set 
of targets.


Libtool is a portability solution to encourage development of software 
which is able to compile and run on many targets, even if the authors 
of the software have never experienced those targets.


Linking static libraries into shared libraries is a complex topic and 
is a reason why libtool offers the clumsy/inefficient "convenience 
library" mechanism since it assures that the objects were compiled 
properly to be used in the library they are linked into.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

Re: [patch #9687] bugfix: make -export-dynamic imply --whole-archive

[Alex Ameen]

Thanks for the follow up, this gave me a much clearer idea of the 
underlying issue that you're trying to solve. I really do appreciate you 
taking the time to help improve `libtool'.


You're absolutely right that `libtool' completely bungles the use of 
`--whole-archive' and `--no-whole-archive' which I see as a high 
priority issue. In several build systems I've built in the past I have 
needed to apply manual patches to `libtool' to work around this, and I 
plan on merging changes to fix this soon ( I'm taking my time to create 
test cases ).


To answer your question about why `--whole-archive' and 
`--no-whole-archive' are useful flags in general : some code-bases 
create libraries with multiple builds of objects for differing 
configurations, which rely the "cherry picking" behavior of the linker 
in combination with dummy symbols to automatically select the 
appropriate form on an object for a given link. This is sometimes 
combined with `--as-needed' and `--no-as-needed' for more fine grained 
control. These use cases are niche, but they can be incredibly useful - 
and if I remember correctly I believe that `libpthread.so', GNU 
`libc.so.6', `libgcc', and several other "core" libraries use this 
behavior in combination with linker-scripts to "choose the right 
objects" ( the nitty gritty of how this is accomplished is an 
interesting rabbit hole to explore ). This reliance on "cherry picking" 
hopefully sheds light on why `--whole-archive' is not the default 
behavior, and why those flags exist. Now, having said that I agree that 
99.9% of the time when I link a `libfoo.a' into a binary ( especially a 
shared object ), I usually want the entire archive to link - so 
`--whole-archive' is a flag I use frequently and it's a major pain that 
`libtool' doesn't handle it correctly at the moment.


We could got back and forth on whether `-export-dynamic' should imply 
`--whole-archive' or not, but where I think we absolutely have agreement 
is that `libtool' needs to start handling `--whole-archive' and 
`--no-whole-archive' properly. Do you think if those flags were fixed, 
would invoking `libtool --mode=link -export-dynamic OBJECTS 
--whole-archive libfoo.a --no-whole-archive libbar.a ...;' be a suitable 
solution to your issue? Having thought about it more I think my gut 
concern was basically backwards-compatibility with existing 
`-export-dynamic' usage.


On your last comment about all of the linking junk being super complex : 
I totally agree with you, and despite exploring niche `ld' junk for 
years now I am always learning. I think at bottom this is why I'm 
cautious about having `-export-dynamic' imply `--whole-archive' 
automatically; I don't personally know every possible use case for these 
flags and I would hate to unknowingly break someone's code-base by 
silently changing the behavior of `-export-dynamic'.



On 11/20/21 10:28 AM, David Lamparter wrote:

Follow-up Comment #2, patch #9687 (project libtool):

Well, it's now 3 years later, and my memory of details on this is pretty much
gone, but let's try.  No warranty on anything here, I'm trying to
reconstruct.

[comment #1 comment #1:]

Could you maybe elaborate on your use case? Perhaps there's something that

I'm missing.

Any executable linked with `-export-dynamic` that links static libraries may
end up missing globally visible symbols if they are not used by the executable
itself.  If the program then tries to load a module that needs these symbols,
it fails.


In my opinion : the existing behavior both by `ld' and `libtool' is

appropriate. Implying `--whole-archive' for dependency libraries in with
`-export-dynamic' will prevent users from intentionally localizing symbols.

`--whole-archive` has no impact on symbol visibility; I don't follow at all
what you're trying to say here.  A binary linked with `-export-dynamic` is
essentially a shared library, and the `ld` docs state:

"This is normally used to turn an archive file into a shared library, forcing
every object to be included in the resulting shared library."

I agree `ld` is doing the appropriate thing here, but `libtool` isn't.  A
binary with `-export-dynamic` needs to export ALL of its global symbols from
ALL of its files for dynamically loaded modules to use.  It is essentially a
shared library at the same time and needs to be treated as such.

NB: `--whole-archive` does NOT make all symbols globally visible! This is
about *objects*, i.e. files.  If you have a file getting linked in that has
some global symbols, but none of them are used in the binary itself - `ld`
will drop the entire file.  That's the wrong thing to do for both creating a
shared library as well as for creating an executable with `-export-dynamic`.


I see these flags as having distinct use cases. I'll note that, the need for

using `--whole-archive' with `ld' isn't necessarily intuitive to users, so I
see the appeal of using it in many situations - but adding additional variance
between `libtool' and 

[patch #9687] bugfix: make -export-dynamic imply --whole-archive

[David Lamparter]

Follow-up Comment #2, patch #9687 (project libtool):

Well, it's now 3 years later, and my memory of details on this is pretty much
gone, but let's try.  No warranty on anything here, I'm trying to
reconstruct.

[comment #1 comment #1:]
> Could you maybe elaborate on your use case? Perhaps there's something that
I'm missing.

Any executable linked with `-export-dynamic` that links static libraries may
end up missing globally visible symbols if they are not used by the executable
itself.  If the program then tries to load a module that needs these symbols,
it fails.

> In my opinion : the existing behavior both by `ld' and `libtool' is
appropriate. Implying `--whole-archive' for dependency libraries in with
`-export-dynamic' will prevent users from intentionally localizing symbols.

`--whole-archive` has no impact on symbol visibility; I don't follow at all
what you're trying to say here.  A binary linked with `-export-dynamic` is
essentially a shared library, and the `ld` docs state:

"This is normally used to turn an archive file into a shared library, forcing
every object to be included in the resulting shared library."

I agree `ld` is doing the appropriate thing here, but `libtool` isn't.  A
binary with `-export-dynamic` needs to export ALL of its global symbols from
ALL of its files for dynamically loaded modules to use.  It is essentially a
shared library at the same time and needs to be treated as such.

NB: `--whole-archive` does NOT make all symbols globally visible! This is
about *objects*, i.e. files.  If you have a file getting linked in that has
some global symbols, but none of them are used in the binary itself - `ld`
will drop the entire file.  That's the wrong thing to do for both creating a
shared library as well as for creating an executable with `-export-dynamic`.

> I see these flags as having distinct use cases. I'll note that, the need for
using `--whole-archive' with `ld' isn't necessarily intuitive to users, so I
see the appeal of using it in many situations - but adding additional variance
between `libtool' and `ld' does not seem justified to me.

If I remember correctly, I couldn't find a way to manually add
`--whole-archive` to the linker invocation that libtool does, because the
option is ordering-sensitive to the file names on the ld command line.  I also
do not understand what you mean with "variance" between `libtool` and `ld` -
there is no equivalency here.  `libtool` is a user/wrapper of `ld` that has
the express purpose of figuring out the "right" flags for `ld`, and it's
failing at its job.

Also: what is that distinct use case for `--whole-archive`?  I think it's
exactly this situation right here... the ld docs even hint at it.

> If the intention is to export symbols which are defined in statically linked
`libtool' libraries a "convenience library" ( `noinst_' ) might be what you're
actually looking for?

Barring my memory failing me, I think we tried that and it simply has the same
problem.


The patch posted here has been shipped in FRRouting for 3 years now, it has
not caused any breakage in any OS/distribution (prominently: Debian, Ubuntu,
RHEL, *BSD, …) - as such I would urge you to just merge it and move on.  I
would happily help you in understanding the issue here, but I can't do that
with having more or less lost my own understanding of it.  Sorry about that,
but there's a boatload of other stuff I need to understand, and my brain is
not infinite :(.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[patch #9687] bugfix: make -export-dynamic imply --whole-archive

[Alex Ameen]

Update of patch #9687 (project libtool):

  Status:None => Need Info  
 Assigned to:None => growpotkin 

___

Follow-up Comment #1:

Could you maybe elaborate on your use case? Perhaps there's something that I'm
missing.

In my opinion : the existing behavior both by `ld' and `libtool' is
appropriate. Implying `--whole-archive' for dependency libraries in with
`-export-dynamic' will prevent users from intentionally localizing symbols.

I see these flags as having distinct use cases. I'll note that, the need for
using `--whole-archive' with `ld' isn't necessarily intuitive to users, so I
see the appeal of using it in many situations - but adding additional variance
between `libtool' and `ld' does not seem justified to me.

If the intention is to export symbols which are defined in statically linked
`libtool' libraries a "convenience library" ( `noinst_' ) might be what you're
actually looking for?

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[patch #9687] bugfix: make -export-dynamic imply --whole-archive

[David Lamparter]

URL:
  

 Summary: bugfix: make -export-dynamic imply --whole-archive
 Project: GNU Libtool
Submitted by: equinox
Submitted on: Tue 04 Sep 2018 01:56:34 PM CEST
Category: None
Priority: 6
  Status: None
 Privacy: Public
 Assigned to: None
Originator Email: 
 Open/Closed: Open
 Discussion Lock: Any

___

Details:

libtool: -export-dynamic implies --whole-archive

The usage of -export-dynamic implies that the program is going to load DSOs
that need some of the program's symbols.  However, the linker will still drop
anything that it doesn't see a reference to when linking archives.  This then
breaks when the DSO tries to use something from an archive that the linker
didn't bother to include.

* build-aux/ltmain.in:  if -export-dynamic is given, put
  -Wl,--whole-archive around any archives we link in.

This changeset is placed in the public domain.




___

File Attachments:


---
Date: Tue 04 Sep 2018 01:56:34 PM CEST  Name:
0001-libtool-export-dynamic-implies-whole-archive.patch  Size: 2KiB   By:
equinox



___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/