Re: Stop "relink" searching host directory when installation prefix provided
On 16 January 2011 17:13, Ralf Wildenhues wrote:
> I haven't looked into this in detail yet, but thanks for the good
> writeup and the many references. We need testsuite exposure for
> this.
Hello again
I've made a test case for my issue (attached patch file). It fails
with current Libtool and if you remove the XFAIL line it passes with
my solution applied. I'm just guessing the syntax etc based on the
other tests, so I wouldn't say it's perfect, but hopefully you can see
what I'm trying to do from the comments.
When I ran all the tests with "make -k check" there are no issues with
the rest of the tests, with or without my patch.
I understand this relinking (at least on Linux) is only needed to
remove hard-coded build directory paths to library dependencies that
Libtool stores during the initial link, which are only put there as a
convenience before things are properly "installed". So perhaps a less
controversial solution or workaround to my problem would be an easy
way to disable these "convenient" build paths and thus the relinking
completely? But I still don't see a problem with my original solution.
-Martin
Index: libtool-2.4+host-relink/tests/destdir.at
===
--- libtool-2.4+host-relink.orig/tests/destdir.at
+++ libtool-2.4+host-relink/tests/destdir.at
@@ -139,4 +139,54 @@ fi
AT_CLEANUP
+
+AT_SETUP([Not relinking against host library])
+AT_KEYWORDS([libtool])
+
+_LT_DIRSETUP
+trickydir="$(pwd)/trickydir"
+rm -rf "$trickydir"
+$mkdir_p "$trickydir"
+cd src
+
+# Create real libtricky and install directly into $trickydir
+echo "int tricky() { return 0; }" > tricky.c
+$LIBTOOL --mode=compile $CC $CPPFLAGS $CFLAGS -c tricky.c
+$LIBTOOL --mode=link $CC $CFLAGS $LDFLAGS -o libtricky.la tricky.lo -rpath
"$trickydir"
+$LIBTOOL --mode=install cp libtricky.la "$trickydir/libtricky.la"
+$LIBTOOL --mode=clean rm -f libtricky.la
+
+# Create libdep, to be eventually installed into target's /$prefix, that
+# depends on libtricky
+echo "extern int tricky(); int dep() { return tricky(); }" > dep.c
+$LIBTOOL --mode=compile $CC $CPPFLAGS $CFLAGS -c -o dep.lo dep.c
+$LIBTOOL --mode=link $CC $CFLAGS $LDFLAGS -L"$trickydir" -ltricky -o libdep.la
-rpath "$prefix" dep.lo
+
+# Create libtest, to also be eventually installed into target's /$prefix,
+# that depends on libdep
+echo "extern int tricky(), dep(); int test() { return dep() + tricky(); }" >
test.c
+$LIBTOOL --mode=compile $CC $CPPFLAGS $CFLAGS -c -o test.lo test.c
+$LIBTOOL --mode=link $CC $CFLAGS $LDFLAGS libdep.la -o libtest.la -rpath
"$prefix" test.lo
+
+# Install libdep into $DESTDIR/$prefix
+$LIBTOOL --mode=install cp libdep.la "$DESTDIR$prefix"
+
+# Create false libtricky and install directly into host's /$prefix
+# Not sure of a perfectly portable way of doing this; currently using an
+# empty lib.so file. With GCC this fails for me with the message
+# libtricky.so: file not recognized: File truncated
+: > "$prefix/libtricky.so"
+
+# Install libtest to trigger a "relink". Relink command should be like
+# $CC -o libtest.so test.o -L"$DESTDIR$prefix" -ldep -L"$trickydir" -ltricky
+# not like
+# $CC -o libtest.so test.o -L"$DESTDIR$prefix" -L"$prefix" -ldep
-L"$trickydir" -ltricky
+# Verify that the relink succeeds and it didn't search $prefix or find the
+# false libtricky there
+AT_XFAIL_IF([true])
+AT_CHECK([$LIBTOOL --mode=install cp libtest.la "$DESTDIR$prefix"],
+ [0], [ignore], [ignore])
+
+AT_CLEANUP
+
m4_popdef([_LT_DIRSETUP])
Re: Stop "relink" searching host directory when installation prefix provided
Hi Charles On 18 January 2011 02:38, Charles Wilson wrote: > Right, but if you remove the $DESTDIR/$libdir from the relink command, > then you'd break these native builds (because, e.g. /usr/lib is still in > the compiler's default search path). > > For instance, suppose you're building gettext: it installs several so's > (with interesting dependencies between them) AND a bunch of executables > that depend on THOSE so's. If you use $DESTDIR, and need to relink, but > don't include $DESTDIR/$libdir in the -L spec, then you'd relink the > executables (and so's with internal dependencies) against the versions > installed by gettext-$OLDVER in /usr/lib. I'm not trying to remove the $DESTDIR/$libdir reference; I'm trying to remove the OS's /$libdir reference. Compare these relink commands, adapted from previous email: Current behaviour: gcc *.o -L$DESTDIR/$libdir -L/$libdir -lgettextlib . . . -o .libs/libgettextsrc-0.18.so With my change: gcc *.o -L$DESTDIR/$libdir -lgettextlib . . . -o .libs/libgettextsrc-0.18.so The current behaviour already means it would find the newer gettextlib dependency (because it gets installed in $DESTDIR/$libdir before gettextsrc does). If anything I think my change would make it less likely to find the old version of gettextlib. Hope we are understanding each other :) > If the new apps depend on any added/new interfaces...boom. It sounds like you're describing something like what's going on in this report, thought I haven't quite figured out how it happens: "Relink with a DESTDIR install mistakenly links against old installed libraries rather than those in DESTDIR" http://savannah.gnu.org/support/?107416 -Martin
Re: Stop "relink" searching host directory when installation prefix provided
On 1/17/2011 8:23 AM, Martin Panter wrote: > On 16 January 2011 17:23, Charles Wilson wrote: >> Actually, Ralf's example (or one very similar to it) is the *primary* >> use of DESTDIR. It's how many packaging tools -- like rpm, or cygport >> on cygwin -- create installable binary packages. > > Agreed, that example already tends to work with unmodified Libtool. I > want to get cross-compiled installable binary packages working as > well. Right, but if you remove the $DESTDIR/$libdir from the relink command, then you'd break these native builds (because, e.g. /usr/lib is still in the compiler's default search path). For instance, suppose you're building gettext: it installs several so's (with interesting dependencies between them) AND a bunch of executables that depend on THOSE so's. If you use $DESTDIR, and need to relink, but don't include $DESTDIR/$libdir in the -L spec, then you'd relink the executables (and so's with internal dependencies) against the versions installed by gettext-$OLDVER in /usr/lib. If the new apps depend on any added/new interfaces...boom. -- Chuck
Re: Stop "relink" searching host directory when installation prefix provided
On 16 January 2011 17:13, Ralf Wildenhues wrote: > Hello Martin, > > * Martin Panter wrote on Sun, Jan 16, 2011 at 01:04:00PM CET: >> Don't search host directory during "relink" if $inst_prefix is provided > >> --- libtool-2.4.orig/libltdl/config/ltmain.m4sh >> +++ libtool-2.4/libltdl/config/ltmain.m4sh >> @@ -6122,12 +6122,14 @@ func_mode_link () >> fi >> else >> # We cannot seem to hardcode it, guess we'll fake it. >> - add_dir="-L$libdir" >> - # Try looking first in the location we're being installed to. >> + >> + # Default if $libdir is not relative to the prefix: >> + add_dir="-L$libdir" >> + >> if test -n "$inst_prefix_dir"; then >> case $libdir in >> [\\/]*) >> - func_append add_dir " -L$inst_prefix_dir$libdir" >> + add_dir="-L$inst_prefix_dir$libdir" >> ;; >> esac >> fi > > Wouldn't it also suffice to just prepend instead of append > -L$inst_prefix_dir$libdir? If no, why not? I did some experimenting cross-compiling with Open embedded's "gettext-0.18-r4" package using "angstrom-2008.1" distro. With original Libtool and all relevant patches not present: (I abbreviated the output to make it more readable) | arm-angstrom-linux-gnueabi-libtool: relink: arm-angstrom-linux-gnueabi-gcc . . . [*.o] . . . -L/. . ./image/usr/lib -L/usr/lib -lgettextlib -L/. . ./armv7a-angstrom-linux-gnueabi/usr/lib -lncurses -lc . . . -o .libs/libgettextsrc-0.18.so | /usr/lib/libncursesw.so: file not recognized: File format not recognized With both $add_dir -L options, but swapped around as Ralf suggested: | arm-angstrom-linux-gnueabi-libtool: relink: arm-angstrom-linux-gnueabi-gcc . . . [*.o] . . . -L/usr/lib -L/. . ./image/usr/lib -lgettextlib -L/. . ./armv7a-angstrom-linux-gnueabi/usr/lib -lncurses -lc . . . -o .libs/libgettextsrc-0.18.so | /usr/lib/libgettextlib.so: file not recognized: File format not recognized Now that I've done the experiment I vaguely remember seeing code inside Libtool that reversed the order of all those $add_dir -L options, just after the part identified in my patch. That would explain what happened here, so I think the current unpatched situation is already doing what Ralf intended. Just for kicks, with my patch applied for only one $add_dir -L option, a successful relink looks like: arm-angstrom-linux-gnueabi-libtool: relink: arm-angstrom-linux-gnueabi-gcc . . . [*.o] . . . -L/. . ./image/usr/lib -lgettextlib -L/. . ./armv7a-angstrom-linux-gnueabi/usr/lib -lncurses -lc . . . -o .libs/libgettextsrc-0.18.so > Asking because I'm fairly sure not everybody uses DESTDIR for cross > compilation and assumes that the target directory is no-go land for > us. (I for one often do 'make install DESTDIR=/tmp/dest' merely to > tar up the installation tree to be scp'ed to another machine where > the NFS share is mounted rw.) I hadn't really considered that some people would expect to implicitly link against libraries found in the $libdir directory. I'd tend to argue that you should have to explicitly specify a "-L$libdir" option if you really want to link against libraries from there. (It seems $libdir comes straight from the target ".la" file?) But I'm not an expert on how to use Libtool and maybe common usage or other people would disagree with me. I also tried natively (Arch Linux) compiling gettext 0.18. Original Libtool behaviour installing into staging directory: libtool: relink: gcc . . . [*.o] . . . -L/. . ./stage/usr/local/lib -L/usr/local/lib -lgettextlib -lacl -lcroco-0.6 -lglib-2.0 -lxml2 -lncurses -lc . . . -o .libs/libgettextsrc-0.18.so I would argue that it shouldn't search the OS's /usr/local/lib for any of the dependencies even though the target library is supposed to eventually exist there. If any of the target's dependencies did exist in /usr/local/lib, I'd expect I'd have to provide that directory to the "configure" script or linker. In my gettext case, the initial link doesn't search any usr/local/lib directory at all, and the gettextlib dependency is already installed into the staging directory before the above relink happens. With my proposed patch: libtool: relink: gcc . . . [*.o] . . . -L/. . ./stage/usr/local/lib -lgettextlib -lacl -lcroco-0.6 -lglib-2.0 -lxml2 -lncurses -lc . . . -o .libs/libgettextsrc-0.18.so On 16 January 2011 17:23, Charles Wilson wrote: > Actually, Ralf's example (or one very similar to it) is the *primary* > use of DESTDIR. It's how many packaging tools -- like rpm, or cygport > on cygwin -- create installable binary packages. Agreed, that example already tends to work with unmodified Libtool. I want to get cross-compiled installable binary packages working as well. -Martin
Re: Stop "relink" searching host directory when installation prefix provided
On 1/16/2011 12:13 PM, Ralf Wildenhues wrote: > (I for one often do 'make install DESTDIR=/tmp/dest' merely to > tar up the installation tree to be scp'ed to another machine where > the NFS share is mounted rw.) Actually, Ralf's example (or one very similar to it) is the *primary* use of DESTDIR. It's how many packaging tools -- like rpm, or cygport on cygwin -- create installable binary packages. -- Chuck
Re: Stop "relink" searching host directory when installation prefix provided
Hello Martin, * Martin Panter wrote on Sun, Jan 16, 2011 at 01:04:00PM CET: > Don't search host directory during "relink" if $inst_prefix is provided > --- libtool-2.4.orig/libltdl/config/ltmain.m4sh > +++ libtool-2.4/libltdl/config/ltmain.m4sh > @@ -6122,12 +6122,14 @@ func_mode_link () > fi > else > # We cannot seem to hardcode it, guess we'll fake it. > - add_dir="-L$libdir" > - # Try looking first in the location we're being installed to. > + > + # Default if $libdir is not relative to the prefix: > + add_dir="-L$libdir" > + > if test -n "$inst_prefix_dir"; then > case $libdir in > [\\/]*) > - func_append add_dir " -L$inst_prefix_dir$libdir" > + add_dir="-L$inst_prefix_dir$libdir" > ;; > esac > fi Wouldn't it also suffice to just prepend instead of append -L$inst_prefix_dir$libdir? If no, why not? Asking because I'm fairly sure not everybody uses DESTDIR for cross compilation and assumes that the target directory is no-go land for us. (I for one often do 'make install DESTDIR=/tmp/dest' merely to tar up the installation tree to be scp'ed to another machine where the NFS share is mounted rw.) I haven't looked into this in detail yet, but thanks for the good writeup and the many references. We need testsuite exposure for this. Cheers, Ralf
Stop "relink" searching host directory when installation prefix provided
I've been compiling programs with Open embedded and came across an issue that has apparently been around for a long time. I think the symptoms have been described many times before; see list of links further down. But briefly, when using Libtool to cross compile and install programs into a target "staging" file system rather than directly into the build host's file system, I found Libtool would do a "relink", which would pass a "-L/usr/lib" (host directory) option to GCC, which would then fail with a message like this: /usr/lib/libgcc_s.so: file not recognized: File format not recognized I understand earlier versions of GCC's linker would warn about this but continue and presumably find the right library with the right file format in a different directory. These days GCC just aborts instead. I suggest the attached patch as a solution. I think this problem has always been present, ever since the "installation prefix" (target staging) feature was added in the following commit: http://git.savannah.gnu.org/cgit/libtool.git/commit/?id=d2c4f8f8fc04e4fe282953800b913dc7c5595c59 My patch simply avoids passing the host directory to GCC when the directory with the installation prefix is also passed. Previously it would pass both directories. == Existing alternative patches Original patches kicking around in Open embedded just blindly delete the "add_dir" assignment. See the middle hunk of the three hunks in this patch: http://cgit.openembedded.org/cgit.cgi/openembedded/tree/recipes/libtool/libtool-2.2.6b/cross_compile.patch?id=release-2010.12 However I suspect that is only valid when cross compiling and not installing directly to the target root filesystem (without a "prefix"). There's also a newer one for a "sysroot" feature: http://cgit.openembedded.org/cgit.cgi/openembedded/tree/recipes/libtool/libtool-2.4/use-sysroot-in-libpath.patch?id=release-2010.12 I haven't learnt what this feature is, and I think this change doesn't apply if the "sysroot" feature is not used. == Other reports of this issue that I came across: "Relink with a DESTDIR install mistakenly links against old installed libraries rather than those in DESTDIR" http://savannah.gnu.org/support/?107416 My initial ramblings about this issue on the Open embedded mailing list: http://lists.linuxtogo.org/pipermail/openembedded-devel/2010-December/027868.html "Issue with Libtool when cross-compiling". See reply from Richard Purdie referring to patches from Poky. http://lists.gnu.org/archive/html/libtool/2010-02/msg0.html 'Re: Issues with "relink" and cross-compilation'. Again with reply from Richard Purdie. http://lists.gnu.org/archive/html/libtool/2010-07/msg00018.html "Relinking and cross-compiling" http://lists.gnu.org/archive/html/libtool/2010-03/msg00013.html "Can't cross-compile with same libdir as the host one" http://lists.gnu.org/archive/html/bug-libtool/2010-10/msg00036.html Found a paragraph added to documentation which might be related to my issue: http://git.savannah.gnu.org/cgit/libtool.git/commit/?id=e34d7507d63faeb4ddaa75b1ecf87ee5aad1aa37 To me that suggests cross compiling many programs would generally only be possible by installing third-party target architecture libraries into the host file system. (I imagine this is kind of how Scratchbox works.) -Martin Don't search host directory during "relink" if $inst_prefix is provided Index: libtool-2.4/libltdl/config/ltmain.m4sh === --- libtool-2.4.orig/libltdl/config/ltmain.m4sh +++ libtool-2.4/libltdl/config/ltmain.m4sh @@ -6122,12 +6122,14 @@ func_mode_link () fi else # We cannot seem to hardcode it, guess we'll fake it. - add_dir="-L$libdir" - # Try looking first in the location we're being installed to. + + # Default if $libdir is not relative to the prefix: + add_dir="-L$libdir" + if test -n "$inst_prefix_dir"; then case $libdir in [\\/]*) - func_append add_dir " -L$inst_prefix_dir$libdir" + add_dir="-L$inst_prefix_dir$libdir" ;; esac fi
