Bug#956418: src:glibc: Please provide optimized builds for ARMv8.1
On 2020-05-07 13:04, Noah Meyerhans wrote:
> On Wed, May 06, 2020 at 04:15:09PM +0200, Aurelien Jarno wrote:
> > > >One solution for this would be to ship the optimized library in the same
> > > >package as the default library. Now this is not acceptable for embedded
> > > >systems as they might not need that library and can't remove it. This is
> > > >even more problematic if we need to add more optimized libraries. I guess
> > > >this might be the case for arm64 as there are many new extensions in the
> > > >pipe.
> > >
> > > ACK. It's a problem to ship the different things in separate
> > > packages. If it's really a problem for smaller systems to have all the
> > > variants because of size, is there maybe another way to do things? How
> > > about keeping the existing libc and have an extra package
> > > ("libc-optimised") with all the optimised versions *and* the basic
> > > version, and have it provide/replace/conflict libc6?
> > >
> > > (/me prepares to be ambarrassed as you point out the obvious flaw I'm
> > > missing...)
> >
> > I guess that the provide/replace/conflict libc6 will just prevent
> > installation of foreign libc6 packages, basically making this optimized
> > package useless in the multiarch context.
> >
> > OTOH, what is the drawback of having GCC defaulting to -moutline-atomics?
> > It will improve performance on many more packages than only glibc, and
> > is way easier to implement overall. It also means users has nothing to
> > do to get additional performances.
>
> For the current issue, defaulting to -moutline-atomics might be a sane
> approach. As you said earlier, though, it seems that there are many new
> extensions in the pipe for ARM. There may not be an equivalent solution
> for all of them, and even if there is, at some point the runtime
> overhead of all this conditional code is going to add up to something
> meaningful.
If we are talking about future extensions, another option for some of
them is to use ifunc. It's how the various SSE and AVX extensions are
supported on x86, and neon is supported on armv7.
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net
Bug#956418: src:glibc: Please provide optimized builds for ARMv8.1
On Wed, May 06, 2020 at 04:15:09PM +0200, Aurelien Jarno wrote:
> > >One solution for this would be to ship the optimized library in the same
> > >package as the default library. Now this is not acceptable for embedded
> > >systems as they might not need that library and can't remove it. This is
> > >even more problematic if we need to add more optimized libraries. I guess
> > >this might be the case for arm64 as there are many new extensions in the
> > >pipe.
> >
> > ACK. It's a problem to ship the different things in separate
> > packages. If it's really a problem for smaller systems to have all the
> > variants because of size, is there maybe another way to do things? How
> > about keeping the existing libc and have an extra package
> > ("libc-optimised") with all the optimised versions *and* the basic
> > version, and have it provide/replace/conflict libc6?
> >
> > (/me prepares to be ambarrassed as you point out the obvious flaw I'm
> > missing...)
>
> I guess that the provide/replace/conflict libc6 will just prevent
> installation of foreign libc6 packages, basically making this optimized
> package useless in the multiarch context.
>
> OTOH, what is the drawback of having GCC defaulting to -moutline-atomics?
> It will improve performance on many more packages than only glibc, and
> is way easier to implement overall. It also means users has nothing to
> do to get additional performances.
For the current issue, defaulting to -moutline-atomics might be a sane
approach. As you said earlier, though, it seems that there are many new
extensions in the pipe for ARM. There may not be an equivalent solution
for all of them, and even if there is, at some point the runtime
overhead of all this conditional code is going to add up to something
meaningful.
noah
Re: Bug#953235: vtkplotter: autopkgtest arm64 failure: No module named 'vtkIOFFMPEGPython'
On Thu, May 07, 2020 at 10:28:33AM +0200, Paul Gevers wrote: >... > On 07-05-2020 10:07, Adrian Bunk wrote: > > This is a toolchain problem affecting many packages: > > https://sourceware.org/bugzilla/show_bug.cgi?id=25051 > > Do you have any rough estimate how many? >... Other bugs I can quickly find are #930039 and #945878. AFAIR I have seen maintainers adding workarounds for arm64+ppc64el FTBFS caused by this bug that did not have bugs in the BTS. #951704 looks like a similar but unrelated problem with jemalloc. > Is there any way to predict which packages are effected, Anything loading a plugin that is directly or indirectly linked with libgomp might or might not have this problem. Python C extensions are the most frequent ones. Heavy usage of Python code with C extensions tends to be in the more scientific areas of the archive, I'd guess many of these packages have no users at all on ppc64el or arm64 who would report bugs (Raspbian is armhf). python3-vtkplotter -> python3-vtk7 -> libvtk -> FFmpeg libraries -> libsoxr -> libgomp > or to detect which packages are effected? "import vtk" works, but "import vtkplotter" blows up when importing vtk. This is similar to the two different OpenSSL versions in stretch, where module load order might determine whether Apache segfaults or starts. >... > > Is there a non-manual way to express that the autopkgtest must not run > > on arm64 and powerpc64el? > > There is currently not even a manual way except for marking the test as > skippable and exitting with error code 77 on unsupported architectures. > Mind you, I don't think toolchain issues should be marked at the package > level (but maybe you didn't mean that). vtkplotter: flagged for removal in 6.3 days The big hammer would be to remove the autopkgtest... > If we can detect this failure > mode (and similar ones in the future) we can of course generate hints > based on this heuristics and have the failures ignored until the > toolchain issues are fixed. A failing arm64 or ppc64el autopkgtest log containing the string "libgomp.so.1: cannot allocate memory in static TLS block" is this bug. > Paul cu Adrian
Re: Bug#953235: vtkplotter: autopkgtest arm64 failure: No module named 'vtkIOFFMPEGPython'
Dear Adrian, On 07-05-2020 10:07, Adrian Bunk wrote: > This is a toolchain problem affecting many packages: > https://sourceware.org/bugzilla/show_bug.cgi?id=25051 Do you have any rough estimate how many? Is there any way to predict which packages are effected, or to detect which packages are effected? > There is nothing a binary-all python module can do to fix > architecture-specific toolchain bugs. Ack. > Is there a non-manual way to express that the autopkgtest must not run > on arm64 and powerpc64el? There is currently not even a manual way except for marking the test as skippable and exitting with error code 77 on unsupported architectures. Mind you, I don't think toolchain issues should be marked at the package level (but maybe you didn't mean that). If we can detect this failure mode (and similar ones in the future) we can of course generate hints based on this heuristics and have the failures ignored until the toolchain issues are fixed. Paul signature.asc Description: OpenPGP digital signature
Re: Bug#953235: vtkplotter: autopkgtest arm64 failure: No module named 'vtkIOFFMPEGPython'
On Fri, Mar 06, 2020 at 10:57:20AM +0100, Paul Gevers wrote: >... > However, it fails on arm64. I copied some of the output at the bottom of > this report. > > Currently this regression is blocking the migration to testing [1]. Can > you please investigate the situation and fix it? >... > https://ci.debian.net/data/autopkgtest/testing/amd64/v/vtkplotter/4281870/log.gz > > autopkgtest [12:57:24]: test python3: [--- > Processing test_actors.py script.. > Traceback (most recent call last): > File "/usr/lib/python3/dist-packages/vtk/vtkIOFFMPEG.py", line 5, in > > from .vtkIOFFMPEGPython import * > ImportError: /lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory > in static TLS block >... This is a toolchain problem affecting many packages: https://sourceware.org/bugzilla/show_bug.cgi?id=25051 There is nothing a binary-all python module can do to fix architecture-specific toolchain bugs. Is there a non-manual way to express that the autopkgtest must not run on arm64 and powerpc64el? cu Adrian
Bug#956418: src:glibc: Please provide optimized builds for ARMv8.1
On Wed, May 06, 2020 at 01:56:24PM +0100, Steve McIntyre wrote:
>...
> On Sun, May 03, 2020 at 11:53:35PM +0200, Aurelien Jarno wrote:
> >
> >One solution for this would be to ship the optimized library in the same
> >package as the default library. Now this is not acceptable for embedded
> >systems as they might not need that library and can't remove it. This is
> >even more problematic if we need to add more optimized libraries. I guess
> >this might be the case for arm64 as there are many new extensions in the
> >pipe.
>
> ACK. It's a problem to ship the different things in separate
> packages. If it's really a problem for smaller systems to have all the
> variants because of size, is there maybe another way to do things? How
> about keeping the existing libc and have an extra package
> ("libc-optimised") with all the optimised versions *and* the basic
> version, and have it provide/replace/conflict libc6?
>...
What Noah mentioned for a similar proposal also applies here:
On Mon, May 04, 2020 at 02:45:41PM -0400, Noah Meyerhans wrote:
>...
> I don't know how well dpkg would cope with transitioning
> between providers, which seems like the riskiest side of this kind of
> thing.
I'd guess you could make this an installation-only change with
a few hacks here and there, but once you think that through
with all the followup-hacks required it doesn't sound like
a good idea.
cu
Adrian
