Re: [patch #9687] bugfix: make -export-dynamic imply --whole-archive

[Alex Ameen]

Thanks for the follow up, this gave me a much clearer idea of the 
underlying issue that you're trying to solve. I really do appreciate you 
taking the time to help improve `libtool'.


You're absolutely right that `libtool' completely bungles the use of 
`--whole-archive' and `--no-whole-archive' which I see as a high 
priority issue. In several build systems I've built in the past I have 
needed to apply manual patches to `libtool' to work around this, and I 
plan on merging changes to fix this soon ( I'm taking my time to create 
test cases ).


To answer your question about why `--whole-archive' and 
`--no-whole-archive' are useful flags in general : some code-bases 
create libraries with multiple builds of objects for differing 
configurations, which rely the "cherry picking" behavior of the linker 
in combination with dummy symbols to automatically select the 
appropriate form on an object for a given link. This is sometimes 
combined with `--as-needed' and `--no-as-needed' for more fine grained 
control. These use cases are niche, but they can be incredibly useful - 
and if I remember correctly I believe that `libpthread.so', GNU 
`libc.so.6', `libgcc', and several other "core" libraries use this 
behavior in combination with linker-scripts to "choose the right 
objects" ( the nitty gritty of how this is accomplished is an 
interesting rabbit hole to explore ). This reliance on "cherry picking" 
hopefully sheds light on why `--whole-archive' is not the default 
behavior, and why those flags exist. Now, having said that I agree that 
99.9% of the time when I link a `libfoo.a' into a binary ( especially a 
shared object ), I usually want the entire archive to link - so 
`--whole-archive' is a flag I use frequently and it's a major pain that 
`libtool' doesn't handle it correctly at the moment.


We could got back and forth on whether `-export-dynamic' should imply 
`--whole-archive' or not, but where I think we absolutely have agreement 
is that `libtool' needs to start handling `--whole-archive' and 
`--no-whole-archive' properly. Do you think if those flags were fixed, 
would invoking `libtool --mode=link -export-dynamic OBJECTS 
--whole-archive libfoo.a --no-whole-archive libbar.a ...;' be a suitable 
solution to your issue? Having thought about it more I think my gut 
concern was basically backwards-compatibility with existing 
`-export-dynamic' usage.


On your last comment about all of the linking junk being super complex : 
I totally agree with you, and despite exploring niche `ld' junk for 
years now I am always learning. I think at bottom this is why I'm 
cautious about having `-export-dynamic' imply `--whole-archive' 
automatically; I don't personally know every possible use case for these 
flags and I would hate to unknowingly break someone's code-base by 
silently changing the behavior of `-export-dynamic'.



On 11/20/21 10:28 AM, David Lamparter wrote:

Follow-up Comment #2, patch #9687 (project libtool):

Well, it's now 3 years later, and my memory of details on this is pretty much
gone, but let's try.  No warranty on anything here, I'm trying to
reconstruct.

[comment #1 comment #1:]

Could you maybe elaborate on your use case? Perhaps there's something that

I'm missing.

Any executable linked with `-export-dynamic` that links static libraries may
end up missing globally visible symbols if they are not used by the executable
itself.  If the program then tries to load a module that needs these symbols,
it fails.


In my opinion : the existing behavior both by `ld' and `libtool' is

appropriate. Implying `--whole-archive' for dependency libraries in with
`-export-dynamic' will prevent users from intentionally localizing symbols.

`--whole-archive` has no impact on symbol visibility; I don't follow at all
what you're trying to say here.  A binary linked with `-export-dynamic` is
essentially a shared library, and the `ld` docs state:

"This is normally used to turn an archive file into a shared library, forcing
every object to be included in the resulting shared library."

I agree `ld` is doing the appropriate thing here, but `libtool` isn't.  A
binary with `-export-dynamic` needs to export ALL of its global symbols from
ALL of its files for dynamically loaded modules to use.  It is essentially a
shared library at the same time and needs to be treated as such.

NB: `--whole-archive` does NOT make all symbols globally visible! This is
about *objects*, i.e. files.  If you have a file getting linked in that has
some global symbols, but none of them are used in the binary itself - `ld`
will drop the entire file.  That's the wrong thing to do for both creating a
shared library as well as for creating an executable with `-export-dynamic`.


I see these flags as having distinct use cases. I'll note that, the need for

using `--whole-archive' with `ld' isn't necessarily intuitive to users, so I
see the appeal of using it in many situations - but adding additional variance
between `libtool' and 

[patch #9687] bugfix: make -export-dynamic imply --whole-archive

[David Lamparter]

Follow-up Comment #2, patch #9687 (project libtool):

Well, it's now 3 years later, and my memory of details on this is pretty much
gone, but let's try.  No warranty on anything here, I'm trying to
reconstruct.

[comment #1 comment #1:]
> Could you maybe elaborate on your use case? Perhaps there's something that
I'm missing.

Any executable linked with `-export-dynamic` that links static libraries may
end up missing globally visible symbols if they are not used by the executable
itself.  If the program then tries to load a module that needs these symbols,
it fails.

> In my opinion : the existing behavior both by `ld' and `libtool' is
appropriate. Implying `--whole-archive' for dependency libraries in with
`-export-dynamic' will prevent users from intentionally localizing symbols.

`--whole-archive` has no impact on symbol visibility; I don't follow at all
what you're trying to say here.  A binary linked with `-export-dynamic` is
essentially a shared library, and the `ld` docs state:

"This is normally used to turn an archive file into a shared library, forcing
every object to be included in the resulting shared library."

I agree `ld` is doing the appropriate thing here, but `libtool` isn't.  A
binary with `-export-dynamic` needs to export ALL of its global symbols from
ALL of its files for dynamically loaded modules to use.  It is essentially a
shared library at the same time and needs to be treated as such.

NB: `--whole-archive` does NOT make all symbols globally visible! This is
about *objects*, i.e. files.  If you have a file getting linked in that has
some global symbols, but none of them are used in the binary itself - `ld`
will drop the entire file.  That's the wrong thing to do for both creating a
shared library as well as for creating an executable with `-export-dynamic`.

> I see these flags as having distinct use cases. I'll note that, the need for
using `--whole-archive' with `ld' isn't necessarily intuitive to users, so I
see the appeal of using it in many situations - but adding additional variance
between `libtool' and `ld' does not seem justified to me.

If I remember correctly, I couldn't find a way to manually add
`--whole-archive` to the linker invocation that libtool does, because the
option is ordering-sensitive to the file names on the ld command line.  I also
do not understand what you mean with "variance" between `libtool` and `ld` -
there is no equivalency here.  `libtool` is a user/wrapper of `ld` that has
the express purpose of figuring out the "right" flags for `ld`, and it's
failing at its job.

Also: what is that distinct use case for `--whole-archive`?  I think it's
exactly this situation right here... the ld docs even hint at it.

> If the intention is to export symbols which are defined in statically linked
`libtool' libraries a "convenience library" ( `noinst_' ) might be what you're
actually looking for?

Barring my memory failing me, I think we tried that and it simply has the same
problem.


The patch posted here has been shipped in FRRouting for 3 years now, it has
not caused any breakage in any OS/distribution (prominently: Debian, Ubuntu,
RHEL, *BSD, …) - as such I would urge you to just merge it and move on.  I
would happily help you in understanding the issue here, but I can't do that
with having more or less lost my own understanding of it.  Sorry about that,
but there's a boatload of other stuff I need to understand, and my brain is
not infinite :(.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

Re: New libtool maintainer

[Ozkan Sezer]

On 10/27/21, Alex Ameen  wrote:
> Howdy!
>
> This is Alex Ameen reporting in from Austin, Texas. I'm a long time GNU
> and `autotools' user who specializes in ELF linking and loading. I'm
> writing you today to introduce myself and announce that I was recently
> approved as the new maintainer of `libtool'!
>
> I'm excited to bring some updates to the tool and want to thank everyone
> for their patience while I work my through the pending patch/bug lists
> and get familiar with some of the maintenance infrastructure.
>
> In the near future I look forward to extending `libtool' support for
> modern ELF shared objects, and improved integration with the rest of
> `autotools'. I'll share a more detailed roadmap after I've worked my
> through the pending tasks in the mailing lists.
>
> I want to express my appreciation for all of the `autotools' and
> `libtool' maintainers/contributors before me. I understand that
> `libtool' is an important piece of infrastructure for a number of
> important pieces of software, and I aim to treat modifications and
> extensions appropriately with that responsibility in mind ( don't worry
> I'm not going to break legacy behavior with reckless abandon haha ).
>
> Feel free to reach out if you have pending patches/issues you want to
> "bump", ideas for improvements, general advice for a new GNU maintainer
> - and above all if you'd like to lend a hand toward getting `libtool' up
> and running again.
>
> Thank you for your time,
>
> -Alex Ameen

It would be nice if the patches for the following long-standing
bugs were applied:


https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21137 (also reported
duplicately as 22895, 31900 and 36762)
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22373
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23348
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38305
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46559
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=44605 (also see 44684)

And maybe this?
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41837

--
O.S.