Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-23 Sebastian Andrzej Siewior wrote: > 3x to be exact: > - 1x shared with threads > - 1x static with threads > - 1x non-shared, no threads, no encoders, just xzdec. > > There are three build folder in the end. The full gets a make install, > the other get xzdec/liblzma.a extracted. Thanks! I remember the details now, it's excellent. I figured out a way to make everything just work in the common case. If --with-pic or --without-pic is used then building both shared and static liblzma at the same time isn't possible (configure will fail). That is, --with-pic or --without-pic requires that also --disable-shared or --disable-static is used on GNU/Linux. It's in xz.git now and will be in the next releases (5.2.9 is needed to fix other bugs) so I hope any workarounds can be removed from distros after that. Thanks to Adrian for reporting the bug! -- Lasse Collin
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
Hi Sebastian! On 11/23/22 21:40, John Paul Adrian Glaubitz wrote: Adrian, could you please remove the -dev package from the buildd? Then it should work (eitherway I'm going to disable the versions for static builds). The chroots are regularly regenerated by a cron job using debootstrap. We don't have any particular setting to explicitly pull in liblzma-dev on ia64, see [1]. So, I have honestly no clue why it was installed. It's pulled in by libunwind-dev which is AFAIK always installed when building C/C++ code on ia64. The external libunwind library is a hard requirement on ia64. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-23 21:12:53 [+0100], To Lasse Collin wrote: > > It is fine to build *static* liblzma with --disable-symbol-versions on > > all archs. Debian-specific workaround is fine in the short term but > > this should be fixed upstream. One method would be to disable the extra > > symbols on ia64 but that is not a real fix. Perhaps it's not really > > possible as long as the main build system is Autotools, I don't > > currently know. > > I'm not sure what other do but it might be reasonable to disable symbol > versions for static linking/ compile since there should be no need for > them. > I kicked a mariadb build on amd64 with liblzma-dev as an addititional > dependency just to see if it fails. Just for the protocol: The mariadb build on amd64 with liblzma-dev installed passed. So this was not it⦠Sebastian
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-23 22:26:39 [+0200], Lasse Collin wrote: > On 2022-11-23 John Paul Adrian Glaubitz wrote: > > Well, Debian builds both the static and dynamic libraries in separate > > steps, so I'm not sure whether the autotools build system would be > > able to detect that. > > I would assume the separate steps means running configure twice, once > to disable static build and once to disable shared build. 3x to be exact: - 1x shared with threads - 1x static with threads - 1x non-shared, no threads, no encoders, just xzdec. There are three build folder in the end. The full gets a make install, the other get xzdec/liblzma.a extracted. Sebastian
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
Hi! On 11/23/22 21:12, Sebastian Andrzej Siewior wrote: Can it build against liblzma.so if liblzma.a isn't available? mariadb does not depend on liblzma-dev for building. The build log says: | -- The following features have been disabled: | * INNODB_LZMA, LZMA compression in the InnoDB storage engine The amd64 buildd has liblzma5 installed - not liblzma-dev. So it can't compile against it nor link statically. The ia64 buildd however has liblzma-dev installed so the options are there. I *think* only the testsuite (or whatever these few binaries were) link statically against it and not the software package as a whole. Adrian, could you please remove the -dev package from the buildd? Then it should work (eitherway I'm going to disable the versions for static builds). The chroots are regularly regenerated by a cron job using debootstrap. We don't have any particular setting to explicitly pull in liblzma-dev on ia64, see [1]. So, I have honestly no clue why it was installed. One could run # debootstrap --foreign --no-check-gpg --arch=ia64 --variant=buildd and see if that creates a chroot with liblzma-dev pre-installed. Let me try a native run on yttrium. Adrian [1] https://salsa.debian.org/debian-ports-team/dsa-puppet -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-23 John Paul Adrian Glaubitz wrote: > Well, Debian builds both the static and dynamic libraries in separate > steps, so I'm not sure whether the autotools build system would be > able to detect that. I would assume the separate steps means running configure twice, once to disable static build and once to disable shared build. > I would make --enable-static and --enable-symbol-versions mutually > exclusive so that the configure fails if both are enabled. I was thinking of a slightly friendlier approach so that the combination --disable-shared --enable-static would imply --disable-symbol-versions on GNU/Linux (it doesn't matter elsewhere for now). It's good if people never need to use the option *-symbol-versions. The defaults need to be as good as easily possible. Using --disable-symbol-versions as a temporary workaround is fine but if it is needed in the long term then something is broken. -- Lasse Collin
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
Hi Lasse! On 11/23/22 20:52, Lasse Collin wrote: On 2022-11-23 John Paul Adrian Glaubitz wrote: So, for now, we should build the static library with "--disable-symbol-versions". An ugly workaround in upstream could be to make configure fail on GNU/Linux if both shared and static libs are about to be built. That is, show an error message describing that one thing has to be built at a time. It's not pretty but with Autotools I don't see any other way except dropping the RHEL/CentOS 7 compat symbols completely. Static libs shouldn't have symbol versions (no matter which arch), somehow it just doesn't always create problems. Well, Debian builds both the static and dynamic libraries in separate steps, so I'm not sure whether the autotools build system would be able to detect that. That is, it would be mandatory to use either --disable-static or --disable-shared to make configure pass. Or would it be less bad to default to shared-only build and require the use of both --disable-shared --enable-static to get static build? I don't like any of these but I don't have better ideas. I would make --enable-static and --enable-symbol-versions mutually exclusive so that the configure fails if both are enabled. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-23 John Paul Adrian Glaubitz wrote: > So, for now, we should build the static library with > "--disable-symbol-versions". An ugly workaround in upstream could be to make configure fail on GNU/Linux if both shared and static libs are about to be built. That is, show an error message describing that one thing has to be built at a time. It's not pretty but with Autotools I don't see any other way except dropping the RHEL/CentOS 7 compat symbols completely. Static libs shouldn't have symbol versions (no matter which arch), somehow it just doesn't always create problems. That is, it would be mandatory to use either --disable-static or --disable-shared to make configure pass. Or would it be less bad to default to shared-only build and require the use of both --disable-shared --enable-static to get static build? I don't like any of these but I don't have better ideas. Thoughts? -- Lasse Collin
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
Hi Sebastian! On 11/23/22 20:29, Sebastian Andrzej Siewior wrote: On 2022-11-23 14:33:39 [+0100], John Paul Adrian Glaubitz wrote: @Sebastian: Can you do that? Does anything speak against that? No, let me do that. Great, thank you! Don't forget to reference Debian bug #1024516 in the debian/changelog ;-). Thanks, Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-23 14:33:39 [+0100], John Paul Adrian Glaubitz wrote: > @Sebastian: Can you do that? Does anything speak against that? No, let me do that. > Adrian Sebastian
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-23 John Paul Adrian Glaubitz wrote: > On 11/23/22 12:31, Lasse Collin wrote: > > (1) Does this make the problem go away? > > Yes, that fixes the linker problem for me. At least in the case of > mariadb-10.6. Why does it want static liblzma.a in the first place? It sounds weird to require rebuilding of mariadb-10.6 every time liblzma is updated. Can it build against liblzma.so if liblzma.a isn't available? It is fine to build *static* liblzma with --disable-symbol-versions on all archs. Debian-specific workaround is fine in the short term but this should be fixed upstream. One method would be to disable the extra symbols on ia64 but that is not a real fix. Perhaps it's not really possible as long as the main build system is Autotools, I don't currently know. I'm still curious why exactly one symbol (lzma_get_progress) looks special in the readelf output. For some reason no other symbols with the symver declarations are there. Does it happen because of something in XZ Utils or is it weird behavior in the toolchain that creates the static lib. One can wonder if it was a mistake to try to clean up the issues that started from the RHEL/CentOS 7 patch since now it has created a new problem. On the other hand, the same could have happened if this kind of symbol versioning had been done to avoid bumping the soname (which hopefully will never happen though). -- Lasse Collin
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
Hi Lasse! On 11/23/22 12:31, Lasse Collin wrote: On 2022-11-23 John Paul Adrian Glaubitz wrote: I guess the additional unwind section breaks your workaround, so the best might be to just disable this workaround on ia64 using the configure flag, no? There currently is no configure option to only disable the CentOS 7 workaround symbols. They are enabled if $host_os matches linux* and --disable-symbol-versions wasn't used. Disabling symbol versions from liblzma.so.5 will cause problems as they have been used since 5.2.0 and many programs and libraries will expect to find XZ_5.0 and XZ_5.2. Having the symbol versions in a static library doesn't make much sense though. Perhaps this is a bug in XZ Utils. As a test, the static liblzma.a could be built without symbol versions with --disable-shared --disable-symbol-versions: (1) Does this make the problem go away? Yes, that fixes the linker problem for me. At least in the case of mariadb-10.6. So, for now, we should build the static library with "--disable-symbol-versions". @Sebastian: Can you do that? Does anything speak against that? Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-23 John Paul Adrian Glaubitz wrote: > I guess the additional unwind section breaks your workaround, so the > best might be to just disable this workaround on ia64 using the > configure flag, no? There currently is no configure option to only disable the CentOS 7 workaround symbols. They are enabled if $host_os matches linux* and --disable-symbol-versions wasn't used. Disabling symbol versions from liblzma.so.5 will cause problems as they have been used since 5.2.0 and many programs and libraries will expect to find XZ_5.0 and XZ_5.2. Having the symbol versions in a static library doesn't make much sense though. Perhaps this is a bug in XZ Utils. As a test, the static liblzma.a could be built without symbol versions with --disable-shared --disable-symbol-versions: (1) Does this make the problem go away? (2) Do the failing builds even require that liblzma.a is present on the system? I don't now how to avoid symvers in a static library as, to my understanding, GNU Libtool doesn't add any -DBUILDING_SHARED_LIBRARY kind of flag which would allow using a #ifdef to know when to use the symbol versions. Libtool does add -DDLL_EXPORT when building a shared library on Windows but that's not useful here. (Switching to another build system would avoid some other Libtool problems too like wrong shared library versioning on some OSes. However, Autotools-based build system is able to produce usable xz on quite a few less-common systems that some other build systems don't support.) A workaround to this workaround could be to disable the CentOS 7 symbols on ia64 by default. Adding an explicit configure option is possible too, if needed. But the first step should be to understand what is going on since the same problem could appear in the future if symbol versions are used for providing compatibility with an actual ABI change (hopefully not needed but still). > Older versions are available through Debian Snapshots: > > > http://snapshot.debian.org/package/xz-utils/ liblzma.a in liblzma-dev_5.2.5-2.1_ia64.deb doesn't have any "@XZ" in it which is expected. This looks normal: : [0x18c0-0x1990], info at +0x100 > > Many other functions are listed in those .IA_64.unwind > > sections too but lzma_get_progress is the only one that has "@XZ" > > as part of the function name. > > Hmm, that definitely seems the problem. Could it be that the symbols > that are exported on ia64 need some additional naming? It seems weird why only one symbol is affected. Perhaps it's a bug in the toolchain creating liblzma.a. However, perhaps the main bug is that XZ Utils build puts symbol versions into a static liblzma. :-( > I think we can waive for CentOS 7 compatibility on Debian unstable > ia64 ο. There is no official CentOS 7 for ia64 but that isn't the whole story as the broken patch has been used elsewhere too. Not having those extra symbols would still be fine in practice. :-) -- Lasse Collin
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
(resent because I got Sergei's email address wrong) Hello Lasse! On 11/23/22 00:11, Lasse Collin wrote: On 2022-11-22 Sebastian Andrzej Siewior wrote: This looks like it is staticaly linked against liblzma. The shared libs in Debian seem to be correct as you managed to answer right before my email. Thanks! ο But the above comment made me look at Debian's liblzma.a. The output of readelf -aW usr/lib/ia64-linux-gnu/liblzma.a includes the following two lines in both 5.2.7 and 5.3.4alpha: Unwind section '.IA_64.unwind' at offset 0x2000 contains 15 entries: [...] : [0x1980-0x1a50], info at +0x108 I guess the additional unwind section breaks your workaround, so the best might be to just disable this workaround on ia64 using the configure flag, no? There are no older versions on the mirror so I didn't check what pre-5.2.7 would have. But .IA_64.unwind is a ia64-specific thing. Older versions are available through Debian Snapshots: http://snapshot.debian.org/package/xz-utils/ Many other functions are listed in those .IA_64.unwind sections too but lzma_get_progress is the only one that has "@XZ" as part of the function name. Hmm, that definitely seems the problem. Could it be that the symbols that are exported on ia64 need some additional naming? I don't understand these details but I wanted let you know anyway in case it isn't a coincidence why lzma_get_progress appears in a special form in both liblzma.a and in the linker error messages. The error has @@XZ_5.2 (which even 5.2.0 has in shared liblzma.so.5) but here the static lib has @XZ_5.2.2 which exists solely for CentOS 7 compatibility. I think we can waive for CentOS 7 compatibility on Debian unstable ia64 ο. Let me CC Sergei Trofimovich from Gentoo who has a more in-depth knowledge on the ia64 architecture. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
Hello Lasse! On 11/23/22 00:11, Lasse Collin wrote: On 2022-11-22 Sebastian Andrzej Siewior wrote: This looks like it is staticaly linked against liblzma. The shared libs in Debian seem to be correct as you managed to answer right before my email. Thanks! :-) But the above comment made me look at Debian's liblzma.a. The output of readelf -aW usr/lib/ia64-linux-gnu/liblzma.a includes the following two lines in both 5.2.7 and 5.3.4alpha: Unwind section '.IA_64.unwind' at offset 0x2000 contains 15 entries: [...] : [0x1980-0x1a50], info at +0x108 I guess the additional unwind section breaks your workaround, so the best might be to just disable this workaround on ia64 using the configure flag, no? There are no older versions on the mirror so I didn't check what pre-5.2.7 would have. But .IA_64.unwind is a ia64-specific thing. Older versions are available through Debian Snapshots: http://snapshot.debian.org/package/xz-utils/ Many other functions are listed in those .IA_64.unwind sections too but lzma_get_progress is the only one that has "@XZ" as part of the function name. Hmm, that definitely seems the problem. Could it be that the symbols that are exported on ia64 need some additional naming? I don't understand these details but I wanted let you know anyway in case it isn't a coincidence why lzma_get_progress appears in a special form in both liblzma.a and in the linker error messages. The error has @@XZ_5.2 (which even 5.2.0 has in shared liblzma.so.5) but here the static lib has @XZ_5.2.2 which exists solely for CentOS 7 compatibility. I think we can waive for CentOS 7 compatibility on Debian unstable ia64 ;-). Let me CC Sergei Trofimovich from Gentoo who has a more in-depth knowledge on the ia64 architecture. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-22 Sebastian Andrzej Siewior wrote: > This looks like it is staticaly linked against liblzma. The shared libs in Debian seem to be correct as you managed to answer right before my email. Thanks! :-) But the above comment made me look at Debian's liblzma.a. The output of readelf -aW usr/lib/ia64-linux-gnu/liblzma.a includes the following two lines in both 5.2.7 and 5.3.4alpha: Unwind section '.IA_64.unwind' at offset 0x2000 contains 15 entries: [...] : [0x1980-0x1a50], info at +0x108 There are no older versions on the mirror so I didn't check what pre-5.2.7 would have. But .IA_64.unwind is a ia64-specific thing. Many other functions are listed in those .IA_64.unwind sections too but lzma_get_progress is the only one that has "@XZ" as part of the function name. I don't understand these details but I wanted let you know anyway in case it isn't a coincidence why lzma_get_progress appears in a special form in both liblzma.a and in the linker error messages. The error has @@XZ_5.2 (which even 5.2.0 has in shared liblzma.so.5) but here the static lib has @XZ_5.2.2 which exists solely for CentOS 7 compatibility. lzma_cputhreads doesn't show the same special behavior in ia64 liblzma.a even though lzma_cputhreads is handled exactly like lzma_get_progress in the liblzma C code and linker script. -- Lasse Collin
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-22 John Paul Adrian Glaubitz wrote: > Does anyone have a clue why this particular change may have broken > the linking on ia64? Thanks for your report. This is important to fix. What do these commands print? Fix the path to liblzma.so.5 if needed. readelf --dyn-syms -W /lib/liblzma.so.5 \ | grep lzma_get_progress readelf --dyn-syms -W /lib/liblzma.so.5 \ | grep lzma_stream_encoder_mt_memusage The first should print 2 lines and the second 3 lines. The rightmost columns should be like these: FUNCGLOBAL DEFAULT 11 lzma_get_progress@@XZ_5.2 FUNCGLOBAL DEFAULT 11 lzma_get_progress@XZ_5.2.2 FUNCGLOBAL DEFAULT 11 lzma_stream_encoder_mt_memusage@@XZ_5.2 FUNCGLOBAL DEFAULT 11 lzma_stream_encoder_mt_memusage@XZ_5.1.2alpha FUNCGLOBAL DEFAULT 11 lzma_stream_encoder_mt_memusage@XZ_5.2.2 Pay close attention to @ vs. @@. The XZ_5.2 must be the ones with @@. If you see the same as above then I don't have a clue. By any chance, was XZ Utils built with GCC older than 10 using link-time optimization (LTO, -flto)? As my commit message describes and NEWS warns, GCC < 10 and LTO will not produce correct results due to the symbol versions. It should work fine with GCC >= 10 or Clang. For what it is worth, when I wrote the patch I tested it on on Slackware 10.1 (32-bit x86) that has GCC 3.3.4 and it worked perfectly there. This symbol version stuff isn't a new thing so it really should work. -- Lasse Collin
Re: [xz-devel] RHEL7 ABI patch (913ddc5) breaks linking on ia64
On 2022-11-22 18:51:49 [+0100], John Paul Adrian Glaubitz wrote: > Hello! Hi, > [ 36%] Linking CXX shared module ha_archive.so > cd /<>/builddir/storage/archive && /usr/bin/cmake -E > cmake_link_script CMakeFiles/archive.dir/link.txt --verbose=1 > /usr/bin/c++ -fPIC -g -O2 -ffile-prefix-map=/<>=. > -specs=/usr/share/dpkg/pie-compile.specs -Wformat -Werror=format-security > -Wdate-time -D_FORTIFY_SOURCE=2 -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC > -fstack-protector --param=ssp-buffer-size=4 -O2 -g -static-libgcc > -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized > -fno-omit-frame-pointer -D_FORTIFY_SOURCE=2 -DDBUG_OFF -Wall -Wenum-compare > -Wenum-conversion -Wextra -Wformat-security -Wmissing-braces > -Wno-format-truncation -Wno-init-self -Wno-nonnull-compare > -Wno-unused-parameter -Woverloaded-virtual -Wnon-virtual-dtor -Wvla > -Wwrite-strings -specs=/usr/share/dpkg/pie-link.specs -Wl,-z,relro,-z,now > -shared -o ha_archive.so CMakeFiles/archive.dir/azio.c.o > CMakeFiles/archive.dir/ha_archive.cc.o ../../libservices/libmysqlservices.a > -lz > /usr/bin/ld: warning: -z relro ignored > /usr/bin/ld: ha_archive.so: version node not found for symbol > lzma_get_progress@@XZ_5.2 > /usr/bin/ld: failed to set dynamic section sizes: bad value > collect2: error: ld returned 1 exit status > make[4]: *** [storage/archive/CMakeFiles/archive.dir/build.make:118: > storage/archive/ha_archive.so] Error 1 > make[4]: Leaving directory '/<>/builddir' > make[3]: *** [CMakeFiles/Makefile2:4913: > storage/archive/CMakeFiles/archive.dir/all] Error 2 I'm not sure if this an ia64 issue or something else is missing. Looking at the symbols: | bigeasy@yttrium:~$ readelf -W --dyn-syms /lib/ia64-linux-gnu/liblzma.so.5|grep lzma_get_progress |160: 7480 208 FUNCGLOBAL DEFAULT 12 lzma_get_progress@@XZ_5.2 |161: 7480 208 FUNCGLOBAL DEFAULT 12 lzma_get_progress@XZ_5.2.2 | bigeasy@yttrium:~$ readelf -W --dyn-syms /usr/bin/xz|grep progress | 45: 0 FUNCGLOBAL DEFAULT UND lzma_get_progress@XZ_5.2 (8) The @@ thingy is used in the library to mark the default symbol. So liblzma provides two lzma_get_progress and default is XZ_5.2. The XZ binary picked it up properly. Looking around in your build: | bigeasy@yttrium:~$ readelf -W --dyn-syms ../glaubitz/mariadb-10.6/mariadb-10.6-10.6.11/builddir/client/mariadb-conv |grep lzma_get_progress |812: 0011c140 208 FUNCGLOBAL DEFAULT 14 lzma_get_progress@@XZ_5.2 |813: 0011c140 208 FUNCGLOBAL DEFAULT 14 lzma_get_progress@XZ_5.2.2 This looks like it is staticaly linked against liblzma. I didn't find lzma_get_progress anywhere else. So it looks like this function isn't used by mariadb itself but appears due to static linking somewhere and asks for trouble. I didn't find any reference to lzma_get_progress in /lib/ia64-linux-gnu/libgcc_s.so.1, /lib/ia64-linux-gnu/libz.so.1.2.13, ha_archive.cc.o or libmysqlservices.a. This seems to be all that is passed to the compiler for linking. Sebastian
