Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Paul Gevers]

Hi all,

On Sat, 4 Aug 2018 18:48:44 +0200 Aurelien Jarno 
wrote:
> On 2018-07-23 22:17, Paul Gevers wrote:
> > On Sun, 01 Apr 2018 21:56:33 +0200 Florian Weimer  
> > wrote:
> > > > I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours
> > > > built on amd64, the glibc takes around 20 minutes to build and the
> > > > testsuite around 2h to run.
> > > 
> > > This is still rather slow.  I see native builds on relatively current
> > > hardware taking 2 minutes, plus 12 to 15 minutes to build and run the
> > > test suite (all with parallel make, although parallel make for tests
> > > is disabled automatically for some subdirectories).  200 minutes on
> > > current (amd64) hardware sounds quite excessive.
> > 
> > I just did a retry on our infrastructure and it ran in 57 minutes. But
> > it ran on one of the two big workers (8 cores and 30 GB memory). We want
> > to make all workers equal and we are going down to 2 cores and 7.2 GB.
> > 
> > Could it be that the memory is the actual problem and/or also an issue?
> 
> I don't think think the memory is really a problem, at least not for the
> values you give. A few tests might be memory hungry, but 4GB should be
> enough.

I retried another time, now that all the workers are equivalent. The
current test suite runs in around 2:10 on our infrastructure. Albeit
long, I will remove glibc from the blacklist.

However, we would still appreciate it when the test can be made smarter.

Paul



signature.asc
Description: OpenPGP digital signature

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Aurelien Jarno]

On 2018-07-23 22:17, Paul Gevers wrote:
> On Sun, 01 Apr 2018 21:56:33 +0200 Florian Weimer  wrote:
> > > I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours
> > > built on amd64, the glibc takes around 20 minutes to build and the
> > > testsuite around 2h to run.
> > 
> > This is still rather slow.  I see native builds on relatively current
> > hardware taking 2 minutes, plus 12 to 15 minutes to build and run the
> > test suite (all with parallel make, although parallel make for tests
> > is disabled automatically for some subdirectories).  200 minutes on
> > current (amd64) hardware sounds quite excessive.
> 
> I just did a retry on our infrastructure and it ran in 57 minutes. But
> it ran on one of the two big workers (8 cores and 30 GB memory). We want
> to make all workers equal and we are going down to 2 cores and 7.2 GB.
> 
> Could it be that the memory is the actual problem and/or also an issue?

I don't think think the memory is really a problem, at least not for the
values you give. A few tests might be memory hungry, but 4GB should be
enough.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net


signature.asc
Description: PGP signature

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Aurelien Jarno]

On 2018-07-30 23:14, Florian Weimer wrote:
> * Paul Gevers:
> 
> > Hi Florian,
> >
> > On 29-07-18 13:26, Florian Weimer wrote:
> >> I'm not sure why it is necessary to build glibc three times (unless
> >> it's impossible to get multi-arch packages into the buildroot).
> >
> > I am not sure if I understand what you mean, but currently having
> > multiple arches available in the autopkgtest testbed isn't supported. I
> > have seen packages try (gnupg2), but this goes easily wrong considering
> > the unstable-to-testing migration setup. If there is a real need for
> > this, it should come from autopkgtest.
> 
> Sorry, I never worked on the Debian toolchain, so my phrasing was
> poor.
> 
> In concrete terms, what I meant was: Why build libc6-i386 on amd64
> when there is a libc6:i386 package as well?

This package is basically only used by gcc to support multilib.
Therefore it's indeed possible to not include those builds in the
autopkgtests.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Paul Gevers]

Hi Florian,

On 30-07-18 23:14, Florian Weimer wrote:
> * Paul Gevers:
>> On 29-07-18 13:26, Florian Weimer wrote:
>>> I'm not sure why it is necessary to build glibc three times (unless
>>> it's impossible to get multi-arch packages into the buildroot).
>>
>> I am not sure if I understand what you mean, but currently having
>> multiple arches available in the autopkgtest testbed isn't supported. I
>> have seen packages try (gnupg2), but this goes easily wrong considering
>> the unstable-to-testing migration setup. If there is a real need for
>> this, it should come from autopkgtest.

> In concrete terms, what I meant was: Why build libc6-i386 on amd64
> when there is a libc6:i386 package as well?

I have no idea.

> In Fedora, there's a restriction that buildds cannot install foreign
> architecture packages.  Some packages need a 32-bit glibc on a 64-bit
> builder, too.  (Typical gcc flags, for example, or fake amd64 packages
> such as amd64).  That made me wonder if Debian has a similar
> restriction for its buildds.

I don't know what the restrictions are on buildds. But autopkgtest
doesn't run on buildds, so I'm not sure if you question is relevant
(unless the above paragraph answers your own question). In autopkgtest
installing foreign architecture packages isn't supported yet as I
mentioned in my previous e-mail.

Paul



signature.asc
Description: OpenPGP digital signature

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Florian Weimer]

* Paul Gevers:

> Hi Florian,
>
> On 29-07-18 13:26, Florian Weimer wrote:
>> I'm not sure why it is necessary to build glibc three times (unless
>> it's impossible to get multi-arch packages into the buildroot).
>
> I am not sure if I understand what you mean, but currently having
> multiple arches available in the autopkgtest testbed isn't supported. I
> have seen packages try (gnupg2), but this goes easily wrong considering
> the unstable-to-testing migration setup. If there is a real need for
> this, it should come from autopkgtest.

Sorry, I never worked on the Debian toolchain, so my phrasing was
poor.

In concrete terms, what I meant was: Why build libc6-i386 on amd64
when there is a libc6:i386 package as well?

In Fedora, there's a restriction that buildds cannot install foreign
architecture packages.  Some packages need a 32-bit glibc on a 64-bit
builder, too.  (Typical gcc flags, for example, or fake amd64 packages
such as amd64).  That made me wonder if Debian has a similar
restriction for its buildds.

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Paul Gevers]

Hi Florian,

On 29-07-18 13:26, Florian Weimer wrote:
> I'm not sure why it is necessary to build glibc three times (unless
> it's impossible to get multi-arch packages into the buildroot).

I am not sure if I understand what you mean, but currently having
multiple arches available in the autopkgtest testbed isn't supported. I
have seen packages try (gnupg2), but this goes easily wrong considering
the unstable-to-testing migration setup. If there is a real need for
this, it should come from autopkgtest.

Paul



signature.asc
Description: OpenPGP digital signature

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Florian Weimer]

* Paul Gevers:

> On Sun, 01 Apr 2018 21:56:33 +0200 Florian Weimer  wrote:
>> > I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours
>> > built on amd64, the glibc takes around 20 minutes to build and the
>> > testsuite around 2h to run.
>> 
>> This is still rather slow.  I see native builds on relatively current
>> hardware taking 2 minutes, plus 12 to 15 minutes to build and run the
>> test suite (all with parallel make, although parallel make for tests
>> is disabled automatically for some subdirectories).  200 minutes on
>> current (amd64) hardware sounds quite excessive.
>
> I just did a retry on our infrastructure and it ran in 57 minutes. But
> it ran on one of the two big workers (8 cores and 30 GB memory). We want
> to make all workers equal and we are going down to 2 cores and 7.2 GB.
>
> Could it be that the memory is the actual problem and/or also an issue?

I looked at the build process, and the amd64 package actually builds
glibc three times (for amd64, i386 and x32).  So 57 minutes is
actually very close to the numbers I gave.

I'm not sure why it is necessary to build glibc three times (unless
it's impossible to get multi-arch packages into the buildroot).  If
you disable kernel support for the i386 and x32 subarchitectures, at
those test suites will not run, which will speed up the build
somewhat.

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Paul Gevers]

On Sun, 01 Apr 2018 21:56:33 +0200 Florian Weimer  wrote:
> > I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours
> > built on amd64, the glibc takes around 20 minutes to build and the
> > testsuite around 2h to run.
> 
> This is still rather slow.  I see native builds on relatively current
> hardware taking 2 minutes, plus 12 to 15 minutes to build and run the
> test suite (all with parallel make, although parallel make for tests
> is disabled automatically for some subdirectories).  200 minutes on
> current (amd64) hardware sounds quite excessive.

I just did a retry on our infrastructure and it ran in 57 minutes. But
it ran on one of the two big workers (8 cores and 30 GB memory). We want
to make all workers equal and we are going down to 2 cores and 7.2 GB.

Could it be that the memory is the actual problem and/or also an issue?

Paul



signature.asc
Description: OpenPGP digital signature

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Florian Weimer]

* Aurelien Jarno:

> I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours
> built on amd64, the glibc takes around 20 minutes to build and the
> testsuite around 2h to run.

This is still rather slow.  I see native builds on relatively current
hardware taking 2 minutes, plus 12 to 15 minutes to build and run the
test suite (all with parallel make, although parallel make for tests
is disabled automatically for some subdirectories).  200 minutes on
current (amd64) hardware sounds quite excessive.

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Aurelien Jarno]

On 2018-03-14 07:40, Paul Gevers wrote:
> Hi Aurelien,
> 
> On 14-03-18 00:15, Aurelien Jarno wrote:
> > On 2018-03-13 21:13, Paul Gevers wrote:
> >> On Mon, 18 May 2015 15:47:34 -0300 Antonio Terceiro
> >>  wrote:
> >>> The glibc test runs times out at ci.debian.net after running for ~3h,
> >>> apparently since they were introduced:
> >>> https://ci.debian.net/packages/g/glibc/unstable/amd64/
> >>
> >> Is there any hope to have this fixed?
> > 
> > I can drop the autopkg tests entries in the next upload if that can help.
> 
> No, that doesn't help anything, as the current situation already
> achieves the same thing.
> 
> We are on the verge of enabling autopkgtest influence on
> unstable-to-testing migration and only a working autopkgtest changes
> anything there. Having glibc autopkgtest appears to me like a
> very-nice-to-have.

Ok.

> Do you know why the autopkgtest runs so long? As Antonio mentioned in
> other similar bug reports, the time out only counts for individual test
> cases, so if you can break down the autopkgtest into smaller fragments,
> the time out may not be an issue.

I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours
built on amd64, the glibc takes around 20 minutes to build and the
testsuite around 2h to run. I am not sure the testsuite is ran in
parallel on ci.debian.net.

> I haven't checked if glibc does, but remember that the idea is to test
> as-installed packages, so rebuilding should be avoided unless needed for
> the test itself. Even then, when test need binaries/artifacts that
> aren't build by default, one could create a binary package containing
> these artifacts in the regular build and depend on it while
> autopkgtesting. E.g. MariaDB/MySQL do that.

This is not something possible with the current upstream build system.
Patches are welcomed though.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net


signature.asc
Description: PGP signature

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Paul Gevers]

Hi Aurelien,

On 14-03-18 00:15, Aurelien Jarno wrote:
> On 2018-03-13 21:13, Paul Gevers wrote:
>> On Mon, 18 May 2015 15:47:34 -0300 Antonio Terceiro
>>  wrote:
>>> The glibc test runs times out at ci.debian.net after running for ~3h,
>>> apparently since they were introduced:
>>> https://ci.debian.net/packages/g/glibc/unstable/amd64/
>>
>> Is there any hope to have this fixed?
> 
> I can drop the autopkg tests entries in the next upload if that can help.

No, that doesn't help anything, as the current situation already
achieves the same thing.

We are on the verge of enabling autopkgtest influence on
unstable-to-testing migration and only a working autopkgtest changes
anything there. Having glibc autopkgtest appears to me like a
very-nice-to-have.

Do you know why the autopkgtest runs so long? As Antonio mentioned in
other similar bug reports, the time out only counts for individual test
cases, so if you can break down the autopkgtest into smaller fragments,
the time out may not be an issue.

I haven't checked if glibc does, but remember that the idea is to test
as-installed packages, so rebuilding should be avoided unless needed for
the test itself. Even then, when test need binaries/artifacts that
aren't build by default, one could create a binary package containing
these artifacts in the regular build and depend on it while
autopkgtesting. E.g. MariaDB/MySQL do that.

Paul



signature.asc
Description: OpenPGP digital signature

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Aurelien Jarno]

On 2018-03-13 21:13, Paul Gevers wrote:
> On Mon, 18 May 2015 15:47:34 -0300 Antonio Terceiro
>  wrote:
> > The glibc test runs times out at ci.debian.net after running for ~3h,
> > apparently since they were introduced:
> > https://ci.debian.net/packages/g/glibc/unstable/amd64/
> 
> Is there any hope to have this fixed?

I can drop the autopkg tests entries in the next upload if that can help.

Aurelien

-- 
Aurelien Jarno  GPG: 4096R/1DDD8C9B
aurel...@aurel32.net http://www.aurel32.net


signature.asc
Description: PGP signature

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Paul Gevers]

On Mon, 18 May 2015 15:47:34 -0300 Antonio Terceiro
 wrote:
> The glibc test runs times out at ci.debian.net after running for ~3h,
> apparently since they were introduced:
> https://ci.debian.net/packages/g/glibc/unstable/amd64/

Is there any hope to have this fixed?

> I am blacklisting glibc for now, and will revisit that when this bug gets
> closed.

Soon, we want to use autopkgtest results to influence
unstable-to-testing migration. It would be a shame to have glibc missing.

Paul



signature.asc
Description: OpenPGP digital signature

Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time

[Antonio Terceiro]

Source: glibc
Severity: normal
User: autopkgtest-de...@lists.alioth.debian.org
Usertags: autopkgtest

Hi,

The glibc test runs times out at ci.debian.net after running for ~3h,
apparently since they were introduced:
http://ci.debian.net/packages/g/gutenprint/unstable/amd64/

It seems that it's just the build that is taking forever on that hardware.
Would it be possible to not rebuild every time, and just run tests against the
installed libc?

I am blacklisting glibc for now, and will revisit that when this bug gets
closed.

-- System Information:
Debian Release: stretch/sid
  APT prefers buildd-unstable
  APT policy: (500, 'buildd-unstable'), (500, 'unstable'), (500, 'testing'), 
(1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.0.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=pt_BR.UTF-8, LC_CTYPE=pt_BR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

-- 
Antonio Terceiro terce...@debian.org


signature.asc
Description: Digital signature