Re: GCC only on OpenBSD adds -L/usr/lib as prefix, why? Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?
On Wednesday, 3 March 2021 19:20, Marc Espie wrote: > On Wed, Mar 03, 2021 at 06:10:22PM +, Bob wrote: > > > Does that -L/usr/lib really need to be in the leading position??? > > I have zero idea how to do that purely in specs. Have fun tinkering. > > This is probably something we'll adopt but low priority. Hi Marc, Thank you a lot for confirming. I am talking to Jonathan at the GCC mailing list, and hope to have figured out soon how to have the -L/usr/lib moved from the beginning to the end through a change to the spec file. Will share outcome here. Ref. https://gcc.gnu.org/pipermail/gcc/2021-March/234927.html > > > - Where is GCC's default specs file say for AMD64/i386? > > somewhere under /usr/lib/gcc-lib or /usr/local/lib/gcc-lib > > you can get gcc to spew it out with -dumpspecs. > > > - Using what environment variable or GCC command line argument do > > I specify an alternative one? > > > > Oh come on, just read the man page and /spec :) > > -specs=file is fairly prominent.
Re: GCC only on OpenBSD adds -L/usr/lib as prefix, why? Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?
On Wednesday, 3 March 2021 16:21, Marc Espie wrote: > Do you have some actual reason to use gcc for that project instead of > clang ?... ... > But again: why gcc ? Hi Marc, Thank you very much for responding. I am in a special code project that is centered around unique GCC features and I can absolutely not move to clang now without enduring great pains. I hope that clang will have some tweaks so that I can move to it, however expectably it will take clang 2-3 years to do that. Indeed my situation is slightly uncommon. > it's been a long time since I've last looked at gcc, we've moved to clang > a few years ago for the most part. gcc is mostly there for the legacy > architectures that do not have clang support. > > Oh, I remember now, it's because of ld.ldd, the linker from clang. > see, that one does not link with /usr/lib by default, which tends to break > everything. Thank you very much for clarifying why OpenBSD GCC adds a -L/usr/lib to LD that GCC on other platforms does not have. Here an immediate followup question: Does that -L/usr/lib really need to be in the leading position??? I'm asking myself if adding it in the leading position was even deliberate. Also was it maybe just a convenient choice due to some technicality of spec file syntax, such. What I see clang do both on OpenBSD and other platforms, and GCC do on other platforms, is that they do add -L/usr/lib to LD, but they do it in the trailing position. Would doing so be fine for the purpose you just described, also? If it would be fine, what about actually moving OpenBSD's GCC (bundled & port) to do that? For completeness, here example output of clang on OpenBSD: $ echo "int main(){}">t.c; clang -c -o t.o t.c ; clang -o t t.o -LMYDIRTEST -Wl,-v --verbose OpenBSD clang version 8.0.1 (tags/RELEASE_801/final) (based on LLVM 8.0.1) Target: amd64-unknown-openbsd6.7 Thread model: posix InstalledDir: /usr/bin "/usr/bin/ld" -e __start --eh-frame-hdr -Bdynamic -dynamic-linker /usr/libexec/ld.so -o t /usr/bin/../lib/crt0.o /usr/bin/../lib/crtbegin.o -LMYDIRTEST -L/usr/bin/../lib -L/usr/lib t.o -v -lcompiler_rt -lc -lcompiler_rt /usr/bin/../lib/crtend.o LLD 8.0.1 (compatible with GNU linkers) As you see, clang adds -L/usr/lib too (in the form "-L/usr/bin/../lib"), however does so in trailing position. Here is what GCC on Debian does: $ gcc -v [..] gcc version 8.3.0 (Debian 8.3.0-6) $ echo "int main(){}">t.c; gcc -c -o t.o t.c ; gcc -o t t.o -LMYDIRTEST -Wl,-v collect2 version 8.3.0 /usr/bin/ld -plugin /usr/lib/gcc/x86_64-linux-gnu/8/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/8/lto-wrapper -plugin-opt=-fresolution=/tmp/ccd7waom.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -o t /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o -LMYDIRTEST -L/usr/lib/gcc/x86_64-linux-gnu/8 -L/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/8/../../.. t.o -v -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/8/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o GNU ld (GNU Binutils for Debian) 2.31.1 As you see it does just the same as clang: it adds -L/usr/lib , however in trailing position. > Note that you don't have to recompile gcc to change that: the specs file > is where the magic happens, and hey, you can specify a new one on the command > line, so you just need to copy and change. Brilliant, also thanks for pointing out. To navigate this question, * Where is GCC's default specs file say for AMD64/i386? * Using what environment variable or GCC command line argument do I specify an alternative one? > as far as -L goes you've got a lot of choices, between linking directly to > the .so, linking with --nostdlib and putting back the pieces manually. I agree there are alternatives however they come with great pains: * Switching GCC/LD argument from "-lz" to "libz.so" indeed works, however it's very difficult to do this in other people's code, projects generally are hardwired to the "-lz" form, e.g. Boost. * -nostdlib does many things and skipping the -L/usr/lib is only one of them. It appears an unnecessarily strong measure for what I want to achieve. If I used it, I'd need to figure out a set of additional arguments to counter all effects -nostdlib has, that I don't want. Thus in summary altering my local GCC spec file (or if this feedback makes all
Re: GCC only on OpenBSD adds -L/usr/lib as prefix, why? Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?
On Wed, Mar 03, 2021 at 06:10:22PM +, Bob wrote: > Does that -L/usr/lib really need to be in the leading position??? I have zero idea how to do that purely in specs. Have fun tinkering. This is probably something we'll adopt but low priority. > * Where is GCC's default specs file say for AMD64/i386? somewhere under /usr/lib/gcc-lib or /usr/local/lib/gcc-lib you can get gcc to spew it out with -dumpspecs. > * Using what environment variable or GCC command line argument do >I specify an alternative one? Oh come on, just read the man page and /spec :) -specs=file is fairly prominent.
Re: GCC only on OpenBSD adds -L/usr/lib as prefix, why? Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?
Do you have some actual reason to use gcc for that project instead of clang ?... as far as -L goes you've got a lot of choices, between linking directly to the .so, linking with --nostdlib and putting back the pieces manually. it's been a long time since I've last looked at gcc, we've moved to clang a few years ago for the most part. gcc is mostly there for the legacy architectures that do not have clang support. Oh, I remember now, it's because of ld.ldd, the linker from clang. see, that one does not link with /usr/lib by default, which tends to break everything. Note that you don't have to recompile gcc to change that: the specs file is where the magic happens, and hey, you can specify a new one on the command line, so you just need to copy and change. But again: why gcc ?
GCC only on OpenBSD adds -L/usr/lib as prefix, why? Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?
Hi Stuart and Marc, Thanks a lot for responding. Debugging the problem brought me to realize that GCC (both bundled gcc/g++ and the egcc/eg++ port) adds a "-L/usr/lib" argument *in the leading position* to LD. Example: $ echo "int main(){}">t.c; gcc -c -o t.o t.c; gcc -o t t.o -LMYDIRTEST -Wl,-v collect2 version 4.2.1 20070719 (OpenBSD/x86-64 ELF) /usr/bin/ld --eh-frame-hdr -e __start -Bdynamic -dynamic-linker /usr/libexec/ld.so -L/usr/lib -o t /usr/lib/crt0.o /usr/lib/crtbegin.o -LMYDIRTEST -L/usr/lib/gcc-lib/amd64-unknown-openbsd6.7/4.2.1 t.o -v -lgcc -lc -lgcc /usr/lib/crtend.o LLD 8.0.1 (compatible with GNU linkers) As you see here, my "-LMYDIRTEST" argument ends up AFTER the -L/usr/lib argument which gcc/egcc adds by itself. This then has the effect that "-L/path/to/my/zlib/build/lib -lz" will NOT attempt to pick my custom build EVER, but instead ALWAYS pick the /usr/lib/libz.so . This is a great headache for any attempt to make a local build of software where zlib is a component! And makes me curious, is this deliberate? I tested clang on OpenBSD and on Linux and it does not have this behavior. Also tested GCC on Linux, and it does not have this behavior. I could imagine that the LEADING instead of TRAILING -L/usr/lib could be some kind of security measure, however if that was the case, I'm sure OpenBSD clang would have had the same, and it does not. What is the motivation? The likely follow-up question will then be, how do I disable this behavior (and get -L/usr/lib as last LD argument as is the case on clang everywhere and GCC on other platforms). Below here for completeness response to Stuart's and Marc's previously shared thoughts on the topic. Overall after having given this question a couple of days of work, I feel there is a certain asymmetry to this -L/usr/lib prefix behavior and it could need a bit of discussion or at the very least clarification. Either OpenBSD's port changes this, or I need to build a local patched port myself. Also, and this is a bit beyond my GCC skills - the OpenBSD GCC port contains patches that introduces this behavior in the first place, does it not - if anyone has in depth understanding of GCC, please feel free to point out which OpenBSD port patch file(s) it is that causes this behavior, if so I could just "rm" that one in my local ports repo and voillas my eg++ and egcc will work the way I want out of the box. Before ending this email I should also mention a partial mitigation of the -L/usr/lib prefix problem: Namely, link in the zlib .so file by mentioning it by .so filename on the GCC (LD) command line, instead of via the "-lz" argument. This works, however it will only work within a fully home-built custom codebase, as any other code project that depends on zlib out there, guaranteedly is hardcoded to link to zlib by "-lz". One would need to patch all such projects' makefiles to link to it by "libz.so" instead of "-lz", and that would be tedious and error-prone, good only as a hack. Thanks! Bob (I wrote:) > > > I'll start with the detail problem, and discuss the reproduction at the > > > bottom: > > > This has brought me to the bizarre issue that the libpng build plainly > > > refuses to link to my custom libz file /home/myuser/lib/libz.so.1 . > > > Instead, it insists with linking to the OS' global > > > /usr/lib/libz.so.5.0 . (Stuart wrote:) > > This is as expected really, it looks for the higher library version > > number. I evaluated the version number hypothesis, for instance by bumping my zlib build's SONAME version number to 999.0 , and my build would not get preference. I also talked to a GCC guy and he said there is no such thing as version preference in respect of gcc -lNAME . Have you actually seen this "pick highest version number" behavior on any platform? (Marc wrote:) > Not really, it's more that you have to make sure to put your own directory > before the system directory, which requires a bit more magic than just -L. > > (remember that linking will stop at the first directory with a satisfying > library, and link with the highest version number found in there, which > is fortunate for stuff like libtool!) Marc, what you suggest here is correct and not correct - because GCC on OpenBSD *always* adds -L/usr/lib in the leading position to LD's arguments, then it does not matter in what order you as user add any -L yourself. (Re. libtool I don't know what you had on your mind.) (I wrote:) > > Hi, > > I am trying to make a custom build of libpng in my home directory, > > using a libz build that I made in my home directory also. > > Both are latest version, libpng 1.6.37 same as OpenBSD's port > > https://cvsweb.openbsd.org/ports/graphics/png/Makefile?rev=1.125=text/x-cvsweb-markup > > and libz the latest one they published. For zlib it's 1.2.11 > > from http://zlib.net/ from 2017, OpenBSD does not have a port but > > base bundles a 2009 version. Since 2009, significantly an export > > "inflateReset2" has
Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?
Hi Stuart and Marc, Thank you very much for responding. I have given some more days to debugging this issue and narrowed it down very much now. I'll changing name on this thread to "GCC only on OpenBSD adds -L/usr/lib as prefix, why" and follow up there. Bob On Wednesday, 24 February 2021 15:40, Marc Espie wrote: > On Wed, Feb 24, 2021 at 02:17:14PM -, Stuart Henderson wrote: > > > On 2021-02-23, Bob obs...@protonmail.com wrote: > > > > > Hi, > > > I am trying to make a custom build of libpng in my home directory, > > > using a libz build that I made in my home directory also. > > > Both are latest version, libpng 1.6.37 same as OpenBSD's port > > > https://cvsweb.openbsd.org/ports/graphics/png/Makefile?rev=1.125=text/x-cvsweb-markup > > > and libz the latest one they published. For zlib it's 1.2.11 > > > from http://zlib.net/ from 2017, OpenBSD does not have a port but > > > base bundles a 2009 version. Since 2009, significantly an export > > > "inflateReset2" has been added. > > > > As you have seen it is difficult to have library functions in one > > version in base and in another version built elsewhere (whether that's > > in $HOME or in ports). Ports only does this for libraries where there's > > really no other choice and where that has been done they're used very > > rarely (the port can then not depend on any libraries which pull in the > > library from base). Currently that is libbind (used only by asdig and > > zeek) and openssl (used as a static library by sslscan, and dynamic > > for nrpe and nsca-ng). > > > > > I'll start with the detail problem, and discuss the reproduction at the > > > bottom: > > > This has brought me to the bizarre issue that the libpng build plainly > > > refuses to link to my custom libz file /home/myuser/lib/libz.so.1 . > > > Instead, it insists with linking to the OS' global > > > /usr/lib/libz.so.5.0 . > > > > This is as expected really, it looks for the higher library version > > number. > > Not really, it's more that you have to make sure to put your own directory > before the system directory, which requires a bit more magic than just -L. > > (remember that linking will stop at the first directory with a satisfying > library, and link with the highest version number found in there, which > is fortunate for stuff like libtool!)
Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?
On Wed, Feb 24, 2021 at 02:17:14PM -, Stuart Henderson wrote: > On 2021-02-23, Bob wrote: > > Hi, > > > > I am trying to make a custom build of libpng in my home directory, > > using a libz build that I made in my home directory also. > > > > Both are latest version, libpng 1.6.37 same as OpenBSD's port > > https://cvsweb.openbsd.org/ports/graphics/png/Makefile?rev=1.125=text/x-cvsweb-markup > > and libz the latest one they published. For zlib it's 1.2.11 > > from http://zlib.net/ from 2017, OpenBSD does not have a port but > > base bundles a 2009 version. Since 2009, significantly an export > > "inflateReset2" has been added. > > As you have seen it is difficult to have library functions in one > version in base and in another version built elsewhere (whether that's > in $HOME or in ports). Ports only does this for libraries where there's > really no other choice and where that has been done they're used *very* > rarely (the port can then not depend on any libraries which pull in the > library from base). Currently that is libbind (used only by asdig and > zeek) and openssl (used as a static library by sslscan, and dynamic > for nrpe and nsca-ng). > > > I'll start with the detail problem, and discuss the reproduction at the > > bottom: > > > > > > This has brought me to the bizarre issue that the libpng build plainly > > refuses to link to my custom libz file /home/myuser/lib/libz.so.1 . > > Instead, it insists with linking to the OS' global > > /usr/lib/libz.so.5.0 . > > This is as expected really, it looks for the higher library version > number. Not really, it's more that you have to make sure to put your own directory before the system directory, which requires a bit more magic than just -L. (remember that linking will stop at the first directory with a satisfying library, and link with the highest version number found in there, which is fortunate for stuff like libtool!)
Re: OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?
On 2021-02-23, Bob wrote: > Hi, > > I am trying to make a custom build of libpng in my home directory, > using a libz build that I made in my home directory also. > > Both are latest version, libpng 1.6.37 same as OpenBSD's port > https://cvsweb.openbsd.org/ports/graphics/png/Makefile?rev=1.125=text/x-cvsweb-markup > and libz the latest one they published. For zlib it's 1.2.11 > from http://zlib.net/ from 2017, OpenBSD does not have a port but > base bundles a 2009 version. Since 2009, significantly an export > "inflateReset2" has been added. As you have seen it is difficult to have library functions in one version in base and in another version built elsewhere (whether that's in $HOME or in ports). Ports only does this for libraries where there's really no other choice and where that has been done they're used *very* rarely (the port can then not depend on any libraries which pull in the library from base). Currently that is libbind (used only by asdig and zeek) and openssl (used as a static library by sslscan, and dynamic for nrpe and nsca-ng). > I'll start with the detail problem, and discuss the reproduction at the > bottom: > > > This has brought me to the bizarre issue that the libpng build plainly > refuses to link to my custom libz file /home/myuser/lib/libz.so.1 . > Instead, it insists with linking to the OS' global > /usr/lib/libz.so.5.0 . This is as expected really, it looks for the higher library version number. > I have reduced the issue to the libpng build step where it produces > libpng.so: > > /usr/local/bin/egcc -shared -fPIC -DPIC -o .libs/libpng.so.16.37 > .libs/png.o .libs/pngerror.o .libs/pngget.o .libs/pngmem.o > .libs/pngpread.o .libs/pngread.o .libs/pngrio.o .libs/pngrtran.o > .libs/pngrutil.o .libs/pngset.o .libs/pngtrans.o .libs/pngwio.o > .libs/pngwrite.o .libs/pngwtran.o .libs/pngwutil.o -L/home/myuser/lib > -lm -lz -Og -g -fstack-protector-all > [with or without: -Wl,--version-script=libpng.vers] > [with or without: -Wl,-t] You might get somewhere further with something like "-nostdlib -L/home/myuser/lib -L/usr/lib". You might need either rpath or change to using static libs. (not relevant to libpng which is in C, but if you try to build C++ software with GCC rather than Clang you will often run into problems with incompatible standard libraries too). > What appears is that GCC ignores the "-L" search path given to it. -L adds, it doesn't overwrite. Hopefully you're just doing this to use pngfix; if you try and replace the installed libpng with one built like this you're likely to end up with all sorts of problems.
OpenBSD: Failing to link custom libpng to custom libz, any thoughts how fix?
Hi, I am trying to make a custom build of libpng in my home directory, using a libz build that I made in my home directory also. Both are latest version, libpng 1.6.37 same as OpenBSD's port https://cvsweb.openbsd.org/ports/graphics/png/Makefile?rev=1.125=text/x-cvsweb-markup and libz the latest one they published. For zlib it's 1.2.11 from http://zlib.net/ from 2017, OpenBSD does not have a port but base bundles a 2009 version. Since 2009, significantly an export "inflateReset2" has been added. I'll start with the detail problem, and discuss the reproduction at the bottom: This has brought me to the bizarre issue that the libpng build plainly refuses to link to my custom libz file /home/myuser/lib/libz.so.1 . Instead, it insists with linking to the OS' global /usr/lib/libz.so.5.0 . /home/myuser/lib/ contains libz.so , libz.a, libz.so.1 and libz.so.1.2.11 . I have reduced the issue to the libpng build step where it produces libpng.so: /usr/local/bin/egcc -shared -fPIC -DPIC -o .libs/libpng.so.16.37 .libs/png.o .libs/pngerror.o .libs/pngget.o .libs/pngmem.o .libs/pngpread.o .libs/pngread.o .libs/pngrio.o .libs/pngrtran.o .libs/pngrutil.o .libs/pngset.o .libs/pngtrans.o .libs/pngwio.o .libs/pngwrite.o .libs/pngwtran.o .libs/pngwutil.o -L/home/myuser/lib -lm -lz -Og -g -fstack-protector-all [with or without: -Wl,--version-script=libpng.vers] [with or without: -Wl,-t] When provided with "-Wl,-t", LD run by GCC prints out: "/usr/lib/crtbeginS.o .libs/png.o .libs/pngerror.o .libs/pngget.o .libs/pngmem.o .libs/pngpread.o .libs/pngread.o .libs/pngrio.o .libs/pngrtran.o .libs/pngrutil.o .libs/pngset.o .libs/pngtrans.o .libs/pngwio.o .libs/pngwrite.o .libs/pngwtran.o .libs/pngwutil.o /usr/lib/libm.so.10.1 /usr/lib/libz.so.5.0 /usr/lib/crtendS.o" which shows that already at link time it picks the OS global one (that is /usr/lib/libz.so.5.0). What appears is that GCC ignores the "-L" search path given to it. "readelf -d .libs/libpng.so.16.37" shows: "Dynamic section at offset 0x40050 contains 25 entries: Tag Type Name/Value 0x001d (RUNPATH) Library runpath: [/home/myuser/lib] 0x0001 (NEEDED) Shared library: [libm.so.10.1] 0x0001 (NEEDED) Shared library: [libz.so.5.0]" Again providing receipt that the wrong LibZ is linked. "objdump -p .libs/libpng.so.16.37" shows the same: ".libs/libpng.so.16.37: file format elf64-x86-64 Program Header: ... Dynamic Section: RUNPATH /home/myuser/lib NEEDED libm.so.10.1 NEEDED libz.so.5.0" The way I got to this issue, was that the libpng "make" would crash with: "ld: error: undefined symbol: inflateReset2 >>> referenced by pngfix.c:2184 (contrib/tools/pngfix.c:2184) >>> contrib/tools/pngfix.o:(zlib_reset) collect2: error: ld returned 1 exit status" This turned out to be that the libpng build process' linking of its "pngfix" accessory tool, would crash because inflateReset2 was not found in the OS' libz (but it is in my custom one). "ldd" showed that libpng.so.17.37 already linked to the OS provided one as discussed above. Whole console interaction leading to the crash below. This whole experience makes me suspicious that maybe for instance "libz" is a reserved filename where only the OS' installation is accepted, but that would make no sense at all? Any thoughts for how to successfully make GCC link, please let me know, Thanks, Bob $ CC=/usr/local/bin/egcc \ > CXX=/usr/local/bin/eg++ \ > LD_LIBRARY_PATH=/home/myuser/lib \ > LDFLAGS="-L/home/myuser/lib -lz " \ > LIBS="-L/home/myuser/lib -I/home/myuser/include -lz " \ > CFLAGS="-Og -g -fstack-protector-all -fpic \ > -I/home/myuser/include -L/home/myuser/lib -lz" \ > CPPFLAGS="-Og -g -fstack-protector-all -fpic \ > -I/home/myuser/include -L/home/myuser/lib -lz " \ > ./configure \ > --prefix=/home/myuser \ > --enable-hardware-optimizations=no \ > --with-pic=yes \ > --enable-unversioned-links \ > --with-zlib-prefix=/home/myuser/lib checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... ./install-sh -c -d checking for gawk... no checking for mawk... no checking for nawk... no checking for awk... awk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether to enable maintainer-specific portions of Makefiles... no checking for gcc... /usr/local/bin/egcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether /usr/local/bin/egcc accepts -g... yes checking for /usr/local/bin/egcc option to accept ISO C89... none needed checking whether /usr/local/bin/egcc understands -c and -o