Re: [EXTERNAL] Re: Unhelpful automatic 3rd-party library linkage
On Tue, Jun 29, 2021 at 3:30 PM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Tue, 29 Jun 2021, Oleg Smolsky wrote: > > > ...and I have figured out the source of the mystery linker flags: zmq > build > > leaves libzmq.la file which contains this: > > > > # Libraries that this one depends upon. > > dependency_libs=' -lrt -lpthread /opt/gcc-10/lib/../lib64/libstdc++.la' > > > > It looks like automake/libtool finds this file (BTW, when is it found?) > and > > transforms `-lzmq` into a whole bunch of things (with explicit .so names > > and dependencies)... > > Yes, that is part of the function of libtool. In fact (as you can > see), the libstdc++.la file was provided with the compiler. > > These are features that you may love or hate depending on what you are > doing. > Right, I use these features for tracking dependencies in our product build... yet it breaks things when a 3rd-party lib is shipped. My current solution is to strip the ".la" file from all automake/libtool-based 3rd-party libs that we build/ship. Is that resonable? > > [...snip...] > The compilation toolchain you are using is set up to not put its > libraries in the default system directories. As a result, the > libstc++.so.6 file needs to be found somehow. > > Indeed! And I do that with explicit -L and -Wl,--rpath options when I link the application. The 3rd-party libs use the same technique... but I don't rebuild 'em on every compiler upgrade (because I can get away with that). The arrangement only broke when the product build (which uses automake/libtool) found an ".la" file from a 3rd-party project and I moved between compiler versions... which to me looks like a leaky abstraction in this case. Oleg.
Re: [EXTERNAL] Re: Unhelpful automatic 3rd-party library linkage
On Tue, Jun 29, 2021 at 3:22 PM Bob Friesenhahn < bfrie...@simple.dallas.tx.us> wrote: > On Tue, 29 Jun 2021, Oleg Smolsky wrote: > > > > It looks like GCC9 references come from libzmq: > > > > $ ldd /opt/3p/lib/libzmq.so | grep libstd > >libstdc++.so.6 => /opt/gcc-9/lib/../lib64/libstdc++.so.6 > > (0x7f95f8d9f000) > > > > Obviously the 3rd-party library was built a while ago with GCC9. At the > > time it was linked to the compiler's runtime... but now the main > > application has moved to GCC11 and I'm linking to the runtime that is > > correct right now. > > > > It looks like automake/libtool try to be helpful and check the library's > > dependencies... but that gets in the way as the new libstdc++ is a strict > > superset of the old one. They maintain ABI compatibility and so scenarios > > like these are supported. > > Are you absolutely sure that the above is true? You specified c++17 > when compiling your application. Are the libstdc++ ABI's the same > across GCC versions and C++ language versions? > Well, I want to claim that I am absolutely sure :) My understanding is that there have been no ABI breaks in the GCC/libstdc++ ever (even noting the 5.x move to the Standard-compliant std::string). The general principle is to let people/distros upgrade gcc/libstdc++ in the OS and let the old apps continue running. > > Is there a way to suppress dependency tracking for the 3rd-party > libraries? > > I wish Libtool/automake was not trying to be smart and simply passed > > "-lzmq" directly to the linker. Yet instead, the actual .so file is > > discovered and then its libstdc++.so is linked. This is just wrong for > the > > scenario at hand. > > Assuming that the whole system does not have these directories in the > default search path (e.g. via ldconfig), it appears that this is a > recorded implicit dependency which is encoded in the library itself. > The only way to remove such an implicit dependency is to rebuild the > library (e.g. libzmq.so) with different options. > > If the persons who delivered the compilers to you expected that the > C++ library was truely reusable, then they would not have have put > everything under /opt/gcc-foo directories (also suggesting that these > directories are removable). Instead they would have put the C++ > run-time libraries in a standard system location. For example, under > Ubuntu Linux, I see that libstdc++.so.6 is at > /usr/lib/x86_64-linux-gnu/libstdc++.so.6 which is a common system > directory. > :) I am the person who maintains the compilers (installed into /opt/gcc-xx) and 3rd-party libs (installed into /opt/3p) at our shop. I don't care to update the system's compiler or libs as we don't use them at all. That is, our build system uses our compiler and only links to the 3rd-party dependencies from /opt. > As far as I am aware, there is no option to request that libtool > not perform the full linkage that it does. A common work-around is to > remove the ".la" files that libtool produces and installs. > > It is possible that GCC itself is pre-programmed (e.g. via the spec > file) to record this information when it links with the C++ standard > library. > Right, I figured this very point out just a couple hours ago - the extra flags/libs (along with the -lzmq transformation) come from the ".la" file. I've rebuilt the lib, purged the file and things look good now for my build. Could you shed some light on how this .la file is supposed to be used? I see that it tries to be helpful by capturing the dependencies... but it seems to destroy the standard `-lfoo` contract. IE it appears that it reduces the level of abstraction needlessly for artifacts that are distributed/stored. Is this ".la" thing meant only for build systems where the whole tree is built from scratch at the same time? Thanks! Oleg.
Re: Unhelpful automatic 3rd-party library linkage
On Tue, 29 Jun 2021, Oleg Smolsky wrote: ...and I have figured out the source of the mystery linker flags: zmq build leaves libzmq.la file which contains this: # Libraries that this one depends upon. dependency_libs=' -lrt -lpthread /opt/gcc-10/lib/../lib64/libstdc++.la' It looks like automake/libtool finds this file (BTW, when is it found?) and transforms `-lzmq` into a whole bunch of things (with explicit .so names and dependencies)... Yes, that is part of the function of libtool. In fact (as you can see), the libstdc++.la file was provided with the compiler. These are features that you may love or hate depending on what you are doing. For example, if the information necessary to find the library is missing (and it is not in the default system library search path), then the program would fail to run! If a program attempts to link with a library which depends on this library, then it would fail to link, or fail to run if the library can not be found. The compilation toolchain you are using is set up to not put its libraries in the default system directories. As a result, the libstc++.so.6 file needs to be found somehow. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
Re: Unhelpful automatic 3rd-party library linkage
On Tue, 29 Jun 2021, Oleg Smolsky wrote: It looks like GCC9 references come from libzmq: $ ldd /opt/3p/lib/libzmq.so | grep libstd libstdc++.so.6 => /opt/gcc-9/lib/../lib64/libstdc++.so.6 (0x7f95f8d9f000) Obviously the 3rd-party library was built a while ago with GCC9. At the time it was linked to the compiler's runtime... but now the main application has moved to GCC11 and I'm linking to the runtime that is correct right now. It looks like automake/libtool try to be helpful and check the library's dependencies... but that gets in the way as the new libstdc++ is a strict superset of the old one. They maintain ABI compatibility and so scenarios like these are supported. Are you absolutely sure that the above is true? You specified c++17 when compiling your application. Are the libstdc++ ABI's the same across GCC versions and C++ language versions? Is there a way to suppress dependency tracking for the 3rd-party libraries? I wish Libtool/automake was not trying to be smart and simply passed "-lzmq" directly to the linker. Yet instead, the actual .so file is discovered and then its libstdc++.so is linked. This is just wrong for the scenario at hand. Assuming that the whole system does not have these directories in the default search path (e.g. via ldconfig), it appears that this is a recorded implicit dependency which is encoded in the library itself. The only way to remove such an implicit dependency is to rebuild the library (e.g. libzmq.so) with different options. If the persons who delivered the compilers to you expected that the C++ library was truely reusable, then they would not have have put everything under /opt/gcc-foo directories (also suggesting that these directories are removable). Instead they would have put the C++ run-time libraries in a standard system location. For example, under Ubuntu Linux, I see that libstdc++.so.6 is at /usr/lib/x86_64-linux-gnu/libstdc++.so.6 which is a common system directory. As far as I am aware, there is no option to request that libtool not perform the full linkage that it does. A common work-around is to remove the ".la" files that libtool produces and installs. It is possible that GCC itself is pre-programmed (e.g. via the spec file) to record this information when it links with the C++ standard library. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt
Re: Unhelpful automatic 3rd-party library linkage
...and I have figured out the source of the mystery linker flags: zmq build leaves libzmq.la file which contains this: # Libraries that this one depends upon. dependency_libs=' -lrt -lpthread /opt/gcc-10/lib/../lib64/libstdc++.la' It looks like automake/libtool finds this file (BTW, when is it found?) and transforms `-lzmq` into a whole bunch of things (with explicit .so names and dependencies)... On Tue, Jun 29, 2021 at 10:57 AM Oleg Smolsky wrote: > Hello there! I'm using automake+libtool for building C++ code and hitting > a peculiar issue with 3rd-party libraries. It comes up when a library is > built with one version of GCC and then the application is built with a > newer compler. > > So, given the following wrapper lib that pulls libzmq: > > noinst_LTLIBRARIES += libnsevent.la > nodist_libnsevent_la_SOURCES = libs/nsevent/wrapper.cpp > libnsevent_la_LIBADD = -lzmq > > Libtool results in the following g++ invocation when linking the > application: > > /opt/gcc-11/bin/g++ -std=c++17 -Wl,-rpath -Wl,/opt/gcc-11/lib64 > -Wl,-rpath -Wl,/opt/3p/lib -o ns_conn_test > libs/netsvc/test/ns_conn_test/src/cpp/NsConnTest.o -L/opt/3p/lib > ./.libs/libnsevent.a /opt/3p/lib/libzmq.so > /opt/gcc-9/lib/../lib64/libstdc++.so /opt/gcc-11/lib/../lib64/libstdc++.so > -lpthread -lrt -ldl -lm -lz -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath > -Wl,/opt/gcc-9/lib/../lib64 -Wl,-rpath -Wl,/opt/gcc-11/lib/../lib64 > -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath -Wl,/opt/gcc-9/lib/../lib64 > -Wl,-rpath -Wl,/opt/gcc-11/lib/../lib64 > > It looks like GCC9 references come from libzmq: > > $ ldd /opt/3p/lib/libzmq.so | grep libstd > libstdc++.so.6 => /opt/gcc-9/lib/../lib64/libstdc++.so.6 > (0x7f95f8d9f000) > > Obviously the 3rd-party library was built a while ago with GCC9. At the > time it was linked to the compiler's runtime... but now the main > application has moved to GCC11 and I'm linking to the runtime that is > correct right now. > > It looks like automake/libtool try to be helpful and check the library's > dependencies... but that gets in the way as the new libstdc++ is a strict > superset of the old one. They maintain ABI compatibility and so scenarios > like these are supported. > > Is there a way to suppress dependency tracking for the 3rd-party > libraries? I wish Libtool/automake was not trying to be smart and simply > passed "-lzmq" directly to the linker. Yet instead, the actual .so file is > discovered and then its libstdc++.so is linked. This is just wrong for the > scenario at hand. > > Thanks in advance, > Oleg. >
Unhelpful automatic 3rd-party library linkage
Hello there! I'm using automake+libtool for building C++ code and hitting a peculiar issue with 3rd-party libraries. It comes up when a library is built with one version of GCC and then the application is built with a newer compler. So, given the following wrapper lib that pulls libzmq: noinst_LTLIBRARIES += libnsevent.la nodist_libnsevent_la_SOURCES = libs/nsevent/wrapper.cpp libnsevent_la_LIBADD = -lzmq Libtool results in the following g++ invocation when linking the application: /opt/gcc-11/bin/g++ -std=c++17 -Wl,-rpath -Wl,/opt/gcc-11/lib64 -Wl,-rpath -Wl,/opt/3p/lib -o ns_conn_test libs/netsvc/test/ns_conn_test/src/cpp/NsConnTest.o -L/opt/3p/lib ./.libs/libnsevent.a /opt/3p/lib/libzmq.so /opt/gcc-9/lib/../lib64/libstdc++.so /opt/gcc-11/lib/../lib64/libstdc++.so -lpthread -lrt -ldl -lm -lz -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath -Wl,/opt/gcc-9/lib/../lib64 -Wl,-rpath -Wl,/opt/gcc-11/lib/../lib64 -Wl,-rpath -Wl,/opt/3p/lib -Wl,-rpath -Wl,/opt/gcc-9/lib/../lib64 -Wl,-rpath -Wl,/opt/gcc-11/lib/../lib64 It looks like GCC9 references come from libzmq: $ ldd /opt/3p/lib/libzmq.so | grep libstd libstdc++.so.6 => /opt/gcc-9/lib/../lib64/libstdc++.so.6 (0x7f95f8d9f000) Obviously the 3rd-party library was built a while ago with GCC9. At the time it was linked to the compiler's runtime... but now the main application has moved to GCC11 and I'm linking to the runtime that is correct right now. It looks like automake/libtool try to be helpful and check the library's dependencies... but that gets in the way as the new libstdc++ is a strict superset of the old one. They maintain ABI compatibility and so scenarios like these are supported. Is there a way to suppress dependency tracking for the 3rd-party libraries? I wish Libtool/automake was not trying to be smart and simply passed "-lzmq" directly to the linker. Yet instead, the actual .so file is discovered and then its libstdc++.so is linked. This is just wrong for the scenario at hand. Thanks in advance, Oleg.
