Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[Steve McIntyre]

On Thu, Feb 02, 2017 at 02:42:26PM +0100, Jens Reyer wrote:
>On 02/02/2017 02:30 PM, Steve McIntyre wrote:
>> Dropping the -nostdlib argument to the gcc call inside sonames2elf
>> makes a difference - it'll add libc6 to the mix and force the output
>> to match the system you're building for. You may then need to filter
>> out the libc6 entry afterwards, but that's easily done.
>
>I'll do that for wine and wine-development, keeping options open for
>dpkg to handle this correctly.

Cool. :-)

>Filtering out isn't necessary: dpkg-shlibdeps already takes care of
>removing redundant entries. And we do want to depend on libc6.

Ah, of course yes...

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Into the distance, a ribbon of black
Stretched to the point of no turning back

Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[Steve McIntyre]

Hi,

On Thu, Feb 02, 2017 at 05:10:14AM +0100, Guillem Jover wrote:
>On Wed, 2017-02-01 at 15:34:04 +, Steve McIntyre wrote:
>> 
>> Please don't go down that route, the ABI flags are intended to save
>> people from that. I'm curious what's going wrong with libgsm1 here
>> such that we're not seeing consistent ABI flags.
>
>Well, I don't see any other sane option really. The problem is that
>the code involved is in perl so must be arch independent, in contrast
>to glibc, which is built against the architecture's ABI.

Nod. I feel your pain... :-/

>In this case dpkg-shlibdeps, sees an ELF object with an EM_ARM machine
>ID which links against "stuff". And it traverses the linker paths, in
>principle in the right order, but for example there is no guarantee of
>the correct order when using the ld.so.conf paths, because they are
>alpha sorted, not in host > foreign order. :(

ACK. In the particular case of wine here, I can see that there's a
genuine problem that we've found (see other mail to Jens). What other
packages are showing problems?

>So we have an ELF object that might or might not have the SOFT/HARD
>flags set, which links against SONAMEs that when found might or might
>not have the SOFT/HARD flags set. And it needs to know which one is
>ABI compatible to keep it or discard it.

Yup. By now, I'd hope that we should have a consistent set of flags in
programs and libraries, though. If there are any that are not yet
fixed, I'd like to know the details so we can fix them.

>Also inferring the architecture from the package shipping the file is
>not currently reliable, due to multilib, because those subvert the
>packaging system by shipping .deb with contents for the wrong arch. :(

Ewww. :-(

>> If you're worried about EABIv4, does the logic of the dpkg checker not
>> match the checks we added in glibc itself?
>
>One of the problems is that currently the ABI matcher is a bit
>simplistic, and it just combines various things that need to really
>match and just compares the byte-streams for equality. Because that
>allows the code to do good enough matching even when it does not know
>new ELF machine types. It does better than the previous objdump
>matcher in any case.
>
>In the future I guess I'll need to special case some architectures,
>in particular, at least ARM, MIPS and SH.
>
>And just for reference the code being discussed is:
>
>  
> 

ACK, I found that last night. It looks a little too simplistic, as you
say. It's not easy to get this right for all the arches, I totally
appreciate that!

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
There's no sensation to compare with this
Suspended animation, A state of bliss

Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[Jens Reyer]

On 02/02/2017 02:30 PM, Steve McIntyre wrote:
> Dropping the -nostdlib argument to the gcc call inside sonames2elf
> makes a difference - it'll add libc6 to the mix and force the output
> to match the system you're building for. You may then need to filter
> out the libc6 entry afterwards, but that's easily done.

I'll do that for wine and wine-development, keeping options open for
dpkg to handle this correctly.

Filtering out isn't necessary: dpkg-shlibdeps already takes care of
removing redundant entries. And we do want to depend on libc6.

Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[Steve McIntyre]

On Wed, Feb 01, 2017 at 05:03:13PM +0100, Jens Reyer wrote:
>On 02/01/2017 04:34 PM, Steve McIntyre wrote:
>> 
>> Please don't go down that route, the ABI flags are intended to save
>> people from that. I'm curious what's going wrong with libgsm1 here
>> such that we're not seeing consistent ABI flags.
>
>It's not just libgsm1, on armhf the build fails because of more than 20
>libraries:
>
>https://buildd.debian.org/status/fetch.php?pkg=wine=armhf=1.8.6-4=1485847439=0
>
>Greets
>jre
>
>
>If it helps, this is the short form of the code that triggers this (we
>compute a list of dependencies for libs that are dlopen'ed by Wine):
>
>$ sonamesDepends="libfontconfig.so.1 libfreetype.so.6 libncurses.so.5"
>$ sonamesRecommends="libcups.so.2 libdbus-1.so.3 libfontconfig.so.1
>libfreetype.so.6 libGL.so.1 libgnutls.so.30 libgsm.so.1 libjpeg.so.62
>libncurses.so.5 libodbc.so.2 libopenal.so.1 libOSMesa.so.8
>libpng16.so.16 libtiff.so.5 libX11.so.6 libXcomposite.so.1
>libXcursor.so.1 libXext.so.6 libXi.so.6 libXinerama.so.1 libXrandr.so.2
>libXrender.so.1 libxslt.so.1 libXxf86vm.so.1"
>
>$ printf 'INPUT(%s)\n' "$sonamesDepends" > libeverything.so
>$ gcc -shared -nostdlib -Wl,--no-as-needed -L. -leverything -o elf.depends
>
>$ printf 'INPUT(%s)\n' "$sonamesRecommends" > libeverything.so
>$ gcc -shared -nostdlib -Wl,--no-as-needed -L. -leverything -o
>elf.recommends
>
>$ dpkg-shlibdeps --warnings=1 \
>-pdlopen \
>-dDepends -edebian/tmp/elf.depends \
>-dRecommends -edebian/tmp/elf.recommends \
>-Tdebian/libwine.substvars

*wince* Not sure how valid the sonames2elf hackery is here, but it's
the output of that script that's causing issues. It looks like there's
a hole in the gcc pipeline around the ABI flag-handling code here, and
it's outputting *both* ABI flags, which is (IIRC) strictly invalid:

  Flags: 0x5000600, Version5 EABI, soft-float ABI, 
hard-float ABI

I'll look into that (again!), but that's going to take some time.

Dropping the -nostdlib argument to the gcc call inside sonames2elf
makes a difference - it'll add libc6 to the mix and force the output
to match the system you're building for. You may then need to filter
out the libc6 entry afterwards, but that's easily done.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Can't keep my eyes from the circling sky,
Tongue-tied & twisted, Just an earth-bound misfit, I...

Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[Guillem Jover]

On Wed, 2017-02-01 at 15:34:04 +, Steve McIntyre wrote:
> On Wed, Feb 01, 2017 at 05:08:50AM +0100, Guillem Jover wrote:
> >On Tue, 2017-01-31 at 22:44:36 +, James Cowgill wrote:
> >> 
> >> Here libgsm.so has neither HARD or SOFT flags set. Also, asking gcc to
> >> generate a library which does not link against libc (this is used by
> >> sonames2elf in some packages) causes both flags to be set (maybe
> >> because it's compatible with both?).
> >
> >Even worse, I've found at least one instance of a package containing
> >binaries with EABIv4 (on armel, paris-traceroute). So I guess I'll have
> >to remove the flags matching on EM_ARM ELF binaries for now. At least
> >this will get us back to the previous behavior with objdump, no
> >regression in that sense.
> >
> >To be able to distinguish armel from armhf I'd probably need to check
> >the arm attributes section for Tag_ABI_VFP_args, which annoyingly
> >seems to be set even when the HARD flag is not set. :/ But this will
> >be for dpkg 1.19.x.
> 
> Please don't go down that route, the ABI flags are intended to save
> people from that. I'm curious what's going wrong with libgsm1 here
> such that we're not seeing consistent ABI flags.

Well, I don't see any other sane option really. The problem is that
the code involved is in perl so must be arch independent, in contrast
to glibc, which is built against the architecture's ABI.

In this case dpkg-shlibdeps, sees an ELF object with an EM_ARM machine
ID which links against "stuff". And it traverses the linker paths, in
principle in the right order, but for example there is no guarantee of
the correct order when using the ld.so.conf paths, because they are
alpha sorted, not in host > foreign order. :(

So we have an ELF object that might or might not have the SOFT/HARD
flags set, which links against SONAMEs that when found might or might
not have the SOFT/HARD flags set. And it needs to know which one is
ABI compatible to keep it or discard it.

Also inferring the architecture from the package shipping the file is
not currently reliable, due to multilib, because those subvert the
packaging system by shipping .deb with contents for the wrong arch. :(

> If you're worried about EABIv4, does the logic of the dpkg checker not
> match the checks we added in glibc itself?

One of the problems is that currently the ABI matcher is a bit
simplistic, and it just combines various things that need to really
match and just compares the byte-streams for equality. Because that
allows the code to do good enough matching even when it does not know
new ELF machine types. It does better than the previous objdump
matcher in any case.

In the future I guess I'll need to special case some architectures,
in particular, at least ARM, MIPS and SH.

And just for reference the code being discussed is:

  


Thanks,
Guillem

Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[Jens Reyer]

On 02/01/2017 04:34 PM, Steve McIntyre wrote:
> Hey folks,
> 
> On Wed, Feb 01, 2017 at 05:08:50AM +0100, Guillem Jover wrote:
>> On Tue, 2017-01-31 at 22:44:36 +, James Cowgill wrote:
>>>
>>> Here libgsm.so has neither HARD or SOFT flags set. Also, asking gcc to
>>> generate a library which does not link against libc (this is used by
>>> sonames2elf in some packages) causes both flags to be set (maybe
>>> because it's compatible with both?).
>>
>> Even worse, I've found at least one instance of a package containing
>> binaries with EABIv4 (on armel, paris-traceroute). So I guess I'll have
>> to remove the flags matching on EM_ARM ELF binaries for now. At least
>> this will get us back to the previous behavior with objdump, no
>> regression in that sense.
>>
>> To be able to distinguish armel from armhf I'd probably need to check
>> the arm attributes section for Tag_ABI_VFP_args, which annoyingly
>> seems to be set even when the HARD flag is not set. :/ But this will
>> be for dpkg 1.19.x.
> 
> Please don't go down that route, the ABI flags are intended to save
> people from that. I'm curious what's going wrong with libgsm1 here
> such that we're not seeing consistent ABI flags.

It's not just libgsm1, on armhf the build fails because of more than 20
libraries:

https://buildd.debian.org/status/fetch.php?pkg=wine=armhf=1.8.6-4=1485847439=0

Greets
jre


If it helps, this is the short form of the code that triggers this (we
compute a list of dependencies for libs that are dlopen'ed by Wine):

$ sonamesDepends="libfontconfig.so.1 libfreetype.so.6 libncurses.so.5"
$ sonamesRecommends="libcups.so.2 libdbus-1.so.3 libfontconfig.so.1
libfreetype.so.6 libGL.so.1 libgnutls.so.30 libgsm.so.1 libjpeg.so.62
libncurses.so.5 libodbc.so.2 libopenal.so.1 libOSMesa.so.8
libpng16.so.16 libtiff.so.5 libX11.so.6 libXcomposite.so.1
libXcursor.so.1 libXext.so.6 libXi.so.6 libXinerama.so.1 libXrandr.so.2
libXrender.so.1 libxslt.so.1 libXxf86vm.so.1"

$ printf 'INPUT(%s)\n' "$sonamesDepends" > libeverything.so
$ gcc -shared -nostdlib -Wl,--no-as-needed -L. -leverything -o elf.depends

$ printf 'INPUT(%s)\n' "$sonamesRecommends" > libeverything.so
$ gcc -shared -nostdlib -Wl,--no-as-needed -L. -leverything -o
elf.recommends

$ dpkg-shlibdeps --warnings=1 \
-pdlopen \
-dDepends -edebian/tmp/elf.depends \
-dRecommends -edebian/tmp/elf.recommends \
-Tdebian/libwine.substvars

Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[Steve McIntyre]

Hey folks,

On Wed, Feb 01, 2017 at 05:08:50AM +0100, Guillem Jover wrote:
>On Tue, 2017-01-31 at 22:44:36 +, James Cowgill wrote:
>> 
>> Here libgsm.so has neither HARD or SOFT flags set. Also, asking gcc to
>> generate a library which does not link against libc (this is used by
>> sonames2elf in some packages) causes both flags to be set (maybe
>> because it's compatible with both?).
>
>Even worse, I've found at least one instance of a package containing
>binaries with EABIv4 (on armel, paris-traceroute). So I guess I'll have
>to remove the flags matching on EM_ARM ELF binaries for now. At least
>this will get us back to the previous behavior with objdump, no
>regression in that sense.
>
>To be able to distinguish armel from armhf I'd probably need to check
>the arm attributes section for Tag_ABI_VFP_args, which annoyingly
>seems to be set even when the HARD flag is not set. :/ But this will
>be for dpkg 1.19.x.

Please don't go down that route, the ABI flags are intended to save
people from that. I'm curious what's going wrong with libgsm1 here
such that we're not seeing consistent ABI flags.

If you're worried about EABIv4, does the logic of the dpkg checker not
match the checks we added in glibc itself?

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Mature Sporty Personal
  More Innovation More Adult
  A Man in Dandism
  Powered Midship Specialty

Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[Jens Reyer]

On 02/01/2017 05:08 AM, Guillem Jover wrote:
> On Tue, 2017-01-31 at 22:44:36 +, James Cowgill wrote:
>> The new ABI mismatch detector seems to be a bit too strict on armel and
>> armhf.

Thanks to both of you for quickly handling this!


>> This was first seen with wine:
>> https://buildd.debian.org/status/fetch.php?pkg=wine=armhf=1.8.6-4=1485847439=0
>> https://buildd.debian.org/status/fetch.php?pkg=wine=armel=1.8.6-4=1485847712=0
>>
>> But there seem to be some other recent build failures relating to this
>> as well.
> 
> Oh, wow, I'm not sure this is very kosher, but if it is causing build
> failures, then it does not matter much.

This sonames2elf stuff is only used to calculate dependencies on
dlopen'ed libraries.  For this we grep the sonames from config.h, create
an ELF that depends on all of them, and then use dpkg-shlibdeps against it.

It's an workaround until this feature is implemented directly in
dpkg-shlibdeps (#596715, dpkg-shlibdeps: Please allow to manually add
library dependencies via shlibdeps).

Greets
jre

Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[Guillem Jover]

Hi!

On Tue, 2017-01-31 at 22:44:36 +, James Cowgill wrote:
> Package: dpkg
> Version: 1.18.21
> Severity: serious
> X-Debbugs-CC: debian-...@lists.debian.org

> [Disclaimer: I'm not an ARM porter and I don't really know much about
> the ARM psABI]
> 
> The new ABI mismatch detector seems to be a bit too strict on armel and
> armhf. The detector forces the ELF_FLAG_ARM_HARD_FLOAT and
> ELF_FLAG_ARM_SOFT_FLOAT flags to be equal on both the library and its
> user but checking ABI comparability doesn't seem that simple.

Indeed. :(

> For example, on armel linking against libgsm.so currently gives this:
> 
> $ dpkg-shlibdeps -v -e../a.out -Ttest
> dpkg-shlibdeps: debug: >> Scanning ../a.out (for Depends field)
> dpkg-shlibdeps: debug: Skipping lib /usr/lib/arm-linux-gnueabi/libgsm.so.1, 
> libabi=0x010100280500 != objabi=0x0101002805000200
> dpkg-shlibdeps: error: cannot find library libgsm.so.1 needed by ../a.out 
> (ELF format: 'elf32-littlearm' abi: '0101002805000200'; RPATH: '')
> dpkg-shlibdeps: debug: Library libc.so.6 found in 
> /lib/arm-linux-gnueabi/libc.so.6
> dpkg-shlibdeps: debug: Using symbols file 
> /var/lib/dpkg/info/libc6:armel.symbols for libc.so.6
> dpkg-shlibdeps: warning: binaries to analyze should already be installed in 
> their package's directory
> dpkg-shlibdeps: error: cannot continue due to the error above
> Note: libraries are not searched in other binary packages that do not have 
> any shlibs or symbols file.
> To help dpkg-shlibdeps find private libraries, you might need to use -l.
> 
> Here libgsm.so has neither HARD or SOFT flags set. Also, asking gcc to
> generate a library which does not link against libc (this is used by
> sonames2elf in some packages) causes both flags to be set (maybe
> because it's compatible with both?).

Even worse, I've found at least one instance of a package containing
binaries with EABIv4 (on armel, paris-traceroute). So I guess I'll have
to remove the flags matching on EM_ARM ELF binaries for now. At least
this will get us back to the previous behavior with objdump, no
regression in that sense.

To be able to distinguish armel from armhf I'd probably need to check
the arm attributes section for Tag_ABI_VFP_args, which annoyingly
seems to be set even when the HARD flag is not set. :/ But this will
be for dpkg 1.19.x.

> This was first seen with wine:
> https://buildd.debian.org/status/fetch.php?pkg=wine=armhf=1.8.6-4=1485847439=0
> https://buildd.debian.org/status/fetch.php?pkg=wine=armel=1.8.6-4=1485847712=0
>
> But there seem to be some other recent build failures relating to this
> as well.

Oh, wow, I'm not sure this is very kosher, but if it is causing build
failures, then it does not matter much.

I'm preparing an upload right now.

Thanks,
Guillem

Bug#853793: dpkg: ABI mismatch detector is too strict on armel/armhf

[James Cowgill]

Package: dpkg
Version: 1.18.21
Severity: serious
X-Debbugs-CC: debian-...@lists.debian.org

Hi,

[Disclaimer: I'm not an ARM porter and I don't really know much about
the ARM psABI]

The new ABI mismatch detector seems to be a bit too strict on armel and
armhf. The detector forces the ELF_FLAG_ARM_HARD_FLOAT and
ELF_FLAG_ARM_SOFT_FLOAT flags to be equal on both the library and its
user but checking ABI comparability doesn't seem that simple.

For example, on armel linking against libgsm.so currently gives this:

$ dpkg-shlibdeps -v -e../a.out -Ttest
dpkg-shlibdeps: debug: >> Scanning ../a.out (for Depends field)
dpkg-shlibdeps: debug: Skipping lib /usr/lib/arm-linux-gnueabi/libgsm.so.1, 
libabi=0x010100280500 != objabi=0x0101002805000200
dpkg-shlibdeps: error: cannot find library libgsm.so.1 needed by ../a.out (ELF 
format: 'elf32-littlearm' abi: '0101002805000200'; RPATH: '')
dpkg-shlibdeps: debug: Library libc.so.6 found in 
/lib/arm-linux-gnueabi/libc.so.6
dpkg-shlibdeps: debug: Using symbols file 
/var/lib/dpkg/info/libc6:armel.symbols for libc.so.6
dpkg-shlibdeps: warning: binaries to analyze should already be installed in 
their package's directory
dpkg-shlibdeps: error: cannot continue due to the error above
Note: libraries are not searched in other binary packages that do not have any 
shlibs or symbols file.
To help dpkg-shlibdeps find private libraries, you might need to use -l.

Here libgsm.so has neither HARD or SOFT flags set. Also, asking gcc to
generate a library which does not link against libc (this is used by
sonames2elf in some packages) causes both flags to be set (maybe
because it's compatible with both?).

This was first seen with wine:
https://buildd.debian.org/status/fetch.php?pkg=wine=armhf=1.8.6-4=1485847439=0
https://buildd.debian.org/status/fetch.php?pkg=wine=armel=1.8.6-4=1485847712=0

But there seem to be some other recent build failures relating to this
as well.

Thanks,
James



signature.asc
Description: OpenPGP digital signature