Re: [EXTERNAL] Re: Unhelpful automatic 3rd-party library linkage
On Tue, 29 Jun 2021, Oleg Smolsky wrote: Are you absolutely sure that the above is true? You specified c++17 when compiling your application. Are the libstdc++ ABI's the same across GCC versions and C++ language versions? Well, I want to claim that I am absolutely sure :) My understanding is that there have been no ABI breaks in the GCC/libstdc++ ever (even noting the 5.x move to the Standard-compliant std::string). The general principle is to let people/distros upgrade gcc/libstdc++ in the OS and let the old apps continue running. That is a good thing. :) I am the person who maintains the compilers (installed into /opt/gcc-xx) and 3rd-party libs (installed into /opt/3p) at our shop. I don't care to update the system's compiler or libs as we don't use them at all. That is, our build system uses our compiler and only links to the 3rd-party dependencies from /opt. Been there. Done that. :-) It is possible that GCC itself is pre-programmed (e.g. via the spec file) to record this information when it links with the C++ standard library. Right, I figured this very point out just a couple hours ago - the extra flags/libs (along with the -lzmq transformation) come from the ".la" file. I've rebuilt the lib, purged the file and things look good now for my build. Could you shed some light on how this .la file is supposed to be used? I see that it tries to be helpful by capturing the dependencies... but it seems to destroy the standard `-lfoo` contract. IE it appears that it reduces the level of abstraction needlessly for artifacts that are distributed/stored. Is this ".la" thing meant only for build systems where the whole tree is built from scratch at the same time? It is true that the .la files encode all of the library dependencies which were current at the time. This is the most common complaint regarding libtool. If this was not supported for static compilation, then static compilation would not be possible. Systems which do not support implicit dependencies in shared libraries also need this. A model whereby libtool can know which explicit dependencies are desired and which should not not be recorded/replayed was never proposed. As a result, it has become common for ".la" files to be deleted or edited (they are simple text files) in order to achieve the goals of binary distributions. In your case it seems you are in complete control. You could put the most recent libstdc++ library in a common area and modify the ".la" files to refer to that one instead. If necessary provide a modifed GCC spec file to assure that the compiler itself does what you want while linking. Unfortunately, existing libraries may have implicit dependencies baked into them which will not go away without rebuilding them. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
Re: [EXTERNAL] Re: Unhelpful automatic 3rd-party library linkage
On Tue, Jun 29, 2021 at 3:30 PM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Tue, 29 Jun 2021, Oleg Smolsky wrote: > > > ...and I have figured out the source of the mystery linker flags: zmq > build > > leaves libzmq.la file which contains this: > > > > # Libraries that this one depends upon. > > dependency_libs=' -lrt -lpthread /opt/gcc-10/lib/../lib64/libstdc++.la' > > > > It looks like automake/libtool finds this file (BTW, when is it found?) > and > > transforms `-lzmq` into a whole bunch of things (with explicit .so names > > and dependencies)... > > Yes, that is part of the function of libtool. In fact (as you can > see), the libstdc++.la file was provided with the compiler. > > These are features that you may love or hate depending on what you are > doing. > Right, I use these features for tracking dependencies in our product build... yet it breaks things when a 3rd-party lib is shipped. My current solution is to strip the ".la" file from all automake/libtool-based 3rd-party libs that we build/ship. Is that resonable? > > [...snip...] > The compilation toolchain you are using is set up to not put its > libraries in the default system directories. As a result, the > libstc++.so.6 file needs to be found somehow. > > Indeed! And I do that with explicit -L and -Wl,--rpath options when I link the application. The 3rd-party libs use the same technique... but I don't rebuild 'em on every compiler upgrade (because I can get away with that). The arrangement only broke when the product build (which uses automake/libtool) found an ".la" file from a 3rd-party project and I moved between compiler versions... which to me looks like a leaky abstraction in this case. Oleg.
Re: [EXTERNAL] Re: Unhelpful automatic 3rd-party library linkage
On Tue, Jun 29, 2021 at 3:22 PM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Tue, 29 Jun 2021, Oleg Smolsky wrote: > > > > It looks like GCC9 references come from libzmq: > > > > $ ldd /opt/3p/lib/libzmq.so | grep libstd > >libstdc++.so.6 => /opt/gcc-9/lib/../lib64/libstdc++.so.6 > > (0x7f95f8d9f000) > > > > Obviously the 3rd-party library was built a while ago with GCC9. At the > > time it was linked to the compiler's runtime... but now the main > > application has moved to GCC11 and I'm linking to the runtime that is > > correct right now. > > > > It looks like automake/libtool try to be helpful and check the library's > > dependencies... but that gets in the way as the new libstdc++ is a strict > > superset of the old one. They maintain ABI compatibility and so scenarios > > like these are supported. > > Are you absolutely sure that the above is true? You specified c++17 > when compiling your application. Are the libstdc++ ABI's the same > across GCC versions and C++ language versions? > Well, I want to claim that I am absolutely sure :) My understanding is that there have been no ABI breaks in the GCC/libstdc++ ever (even noting the 5.x move to the Standard-compliant std::string). The general principle is to let people/distros upgrade gcc/libstdc++ in the OS and let the old apps continue running. > > Is there a way to suppress dependency tracking for the 3rd-party > libraries? > > I wish Libtool/automake was not trying to be smart and simply passed > > "-lzmq" directly to the linker. Yet instead, the actual .so file is > > discovered and then its libstdc++.so is linked. This is just wrong for > the > > scenario at hand. > > Assuming that the whole system does not have these directories in the > default search path (e.g. via ldconfig), it appears that this is a > recorded implicit dependency which is encoded in the library itself. > The only way to remove such an implicit dependency is to rebuild the > library (e.g. libzmq.so) with different options. > > If the persons who delivered the compilers to you expected that the > C++ library was truely reusable, then they would not have have put > everything under /opt/gcc-foo directories (also suggesting that these > directories are removable). Instead they would have put the C++ > run-time libraries in a standard system location. For example, under > Ubuntu Linux, I see that libstdc++.so.6 is at > /usr/lib/x86_64-linux-gnu/libstdc++.so.6 which is a common system > directory. > :) I am the person who maintains the compilers (installed into /opt/gcc-xx) and 3rd-party libs (installed into /opt/3p) at our shop. I don't care to update the system's compiler or libs as we don't use them at all. That is, our build system uses our compiler and only links to the 3rd-party dependencies from /opt. > As far as I am aware, there is no option to request that libtool > not perform the full linkage that it does. A common work-around is to > remove the ".la" files that libtool produces and installs. > > It is possible that GCC itself is pre-programmed (e.g. via the spec > file) to record this information when it links with the C++ standard > library. > Right, I figured this very point out just a couple hours ago - the extra flags/libs (along with the -lzmq transformation) come from the ".la" file. I've rebuilt the lib, purged the file and things look good now for my build. Could you shed some light on how this .la file is supposed to be used? I see that it tries to be helpful by capturing the dependencies... but it seems to destroy the standard `-lfoo` contract. IE it appears that it reduces the level of abstraction needlessly for artifacts that are distributed/stored. Is this ".la" thing meant only for build systems where the whole tree is built from scratch at the same time? Thanks! Oleg.
