Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
Hi all, On Sat, 4 Aug 2018 18:48:44 +0200 Aurelien Jarno wrote: > On 2018-07-23 22:17, Paul Gevers wrote: > > On Sun, 01 Apr 2018 21:56:33 +0200 Florian Weimer > > wrote: > > > > I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours > > > > built on amd64, the glibc takes around 20 minutes to build and the > > > > testsuite around 2h to run. > > > > > > This is still rather slow. I see native builds on relatively current > > > hardware taking 2 minutes, plus 12 to 15 minutes to build and run the > > > test suite (all with parallel make, although parallel make for tests > > > is disabled automatically for some subdirectories). 200 minutes on > > > current (amd64) hardware sounds quite excessive. > > > > I just did a retry on our infrastructure and it ran in 57 minutes. But > > it ran on one of the two big workers (8 cores and 30 GB memory). We want > > to make all workers equal and we are going down to 2 cores and 7.2 GB. > > > > Could it be that the memory is the actual problem and/or also an issue? > > I don't think think the memory is really a problem, at least not for the > values you give. A few tests might be memory hungry, but 4GB should be > enough. I retried another time, now that all the workers are equivalent. The current test suite runs in around 2:10 on our infrastructure. Albeit long, I will remove glibc from the blacklist. However, we would still appreciate it when the test can be made smarter. Paul signature.asc Description: OpenPGP digital signature
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
On 2018-07-23 22:17, Paul Gevers wrote: > On Sun, 01 Apr 2018 21:56:33 +0200 Florian Weimer wrote: > > > I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours > > > built on amd64, the glibc takes around 20 minutes to build and the > > > testsuite around 2h to run. > > > > This is still rather slow. I see native builds on relatively current > > hardware taking 2 minutes, plus 12 to 15 minutes to build and run the > > test suite (all with parallel make, although parallel make for tests > > is disabled automatically for some subdirectories). 200 minutes on > > current (amd64) hardware sounds quite excessive. > > I just did a retry on our infrastructure and it ran in 57 minutes. But > it ran on one of the two big workers (8 cores and 30 GB memory). We want > to make all workers equal and we are going down to 2 cores and 7.2 GB. > > Could it be that the memory is the actual problem and/or also an issue? I don't think think the memory is really a problem, at least not for the values you give. A few tests might be memory hungry, but 4GB should be enough. Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: PGP signature
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
On 2018-07-30 23:14, Florian Weimer wrote: > * Paul Gevers: > > > Hi Florian, > > > > On 29-07-18 13:26, Florian Weimer wrote: > >> I'm not sure why it is necessary to build glibc three times (unless > >> it's impossible to get multi-arch packages into the buildroot). > > > > I am not sure if I understand what you mean, but currently having > > multiple arches available in the autopkgtest testbed isn't supported. I > > have seen packages try (gnupg2), but this goes easily wrong considering > > the unstable-to-testing migration setup. If there is a real need for > > this, it should come from autopkgtest. > > Sorry, I never worked on the Debian toolchain, so my phrasing was > poor. > > In concrete terms, what I meant was: Why build libc6-i386 on amd64 > when there is a libc6:i386 package as well? This package is basically only used by gcc to support multilib. Therefore it's indeed possible to not include those builds in the autopkgtests. Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
Hi Florian, On 30-07-18 23:14, Florian Weimer wrote: > * Paul Gevers: >> On 29-07-18 13:26, Florian Weimer wrote: >>> I'm not sure why it is necessary to build glibc three times (unless >>> it's impossible to get multi-arch packages into the buildroot). >> >> I am not sure if I understand what you mean, but currently having >> multiple arches available in the autopkgtest testbed isn't supported. I >> have seen packages try (gnupg2), but this goes easily wrong considering >> the unstable-to-testing migration setup. If there is a real need for >> this, it should come from autopkgtest. > In concrete terms, what I meant was: Why build libc6-i386 on amd64 > when there is a libc6:i386 package as well? I have no idea. > In Fedora, there's a restriction that buildds cannot install foreign > architecture packages. Some packages need a 32-bit glibc on a 64-bit > builder, too. (Typical gcc flags, for example, or fake amd64 packages > such as amd64). That made me wonder if Debian has a similar > restriction for its buildds. I don't know what the restrictions are on buildds. But autopkgtest doesn't run on buildds, so I'm not sure if you question is relevant (unless the above paragraph answers your own question). In autopkgtest installing foreign architecture packages isn't supported yet as I mentioned in my previous e-mail. Paul signature.asc Description: OpenPGP digital signature
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
* Paul Gevers: > Hi Florian, > > On 29-07-18 13:26, Florian Weimer wrote: >> I'm not sure why it is necessary to build glibc three times (unless >> it's impossible to get multi-arch packages into the buildroot). > > I am not sure if I understand what you mean, but currently having > multiple arches available in the autopkgtest testbed isn't supported. I > have seen packages try (gnupg2), but this goes easily wrong considering > the unstable-to-testing migration setup. If there is a real need for > this, it should come from autopkgtest. Sorry, I never worked on the Debian toolchain, so my phrasing was poor. In concrete terms, what I meant was: Why build libc6-i386 on amd64 when there is a libc6:i386 package as well? In Fedora, there's a restriction that buildds cannot install foreign architecture packages. Some packages need a 32-bit glibc on a 64-bit builder, too. (Typical gcc flags, for example, or fake amd64 packages such as amd64). That made me wonder if Debian has a similar restriction for its buildds.
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
Hi Florian, On 29-07-18 13:26, Florian Weimer wrote: > I'm not sure why it is necessary to build glibc three times (unless > it's impossible to get multi-arch packages into the buildroot). I am not sure if I understand what you mean, but currently having multiple arches available in the autopkgtest testbed isn't supported. I have seen packages try (gnupg2), but this goes easily wrong considering the unstable-to-testing migration setup. If there is a real need for this, it should come from autopkgtest. Paul signature.asc Description: OpenPGP digital signature
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
* Paul Gevers: > On Sun, 01 Apr 2018 21:56:33 +0200 Florian Weimer wrote: >> > I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours >> > built on amd64, the glibc takes around 20 minutes to build and the >> > testsuite around 2h to run. >> >> This is still rather slow. I see native builds on relatively current >> hardware taking 2 minutes, plus 12 to 15 minutes to build and run the >> test suite (all with parallel make, although parallel make for tests >> is disabled automatically for some subdirectories). 200 minutes on >> current (amd64) hardware sounds quite excessive. > > I just did a retry on our infrastructure and it ran in 57 minutes. But > it ran on one of the two big workers (8 cores and 30 GB memory). We want > to make all workers equal and we are going down to 2 cores and 7.2 GB. > > Could it be that the memory is the actual problem and/or also an issue? I looked at the build process, and the amd64 package actually builds glibc three times (for amd64, i386 and x32). So 57 minutes is actually very close to the numbers I gave. I'm not sure why it is necessary to build glibc three times (unless it's impossible to get multi-arch packages into the buildroot). If you disable kernel support for the i386 and x32 subarchitectures, at those test suites will not run, which will speed up the build somewhat.
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
On Sun, 01 Apr 2018 21:56:33 +0200 Florian Weimer wrote: > > I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours > > built on amd64, the glibc takes around 20 minutes to build and the > > testsuite around 2h to run. > > This is still rather slow. I see native builds on relatively current > hardware taking 2 minutes, plus 12 to 15 minutes to build and run the > test suite (all with parallel make, although parallel make for tests > is disabled automatically for some subdirectories). 200 minutes on > current (amd64) hardware sounds quite excessive. I just did a retry on our infrastructure and it ran in 57 minutes. But it ran on one of the two big workers (8 cores and 30 GB memory). We want to make all workers equal and we are going down to 2 cores and 7.2 GB. Could it be that the memory is the actual problem and/or also an issue? Paul signature.asc Description: OpenPGP digital signature
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
* Aurelien Jarno: > I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours > built on amd64, the glibc takes around 20 minutes to build and the > testsuite around 2h to run. This is still rather slow. I see native builds on relatively current hardware taking 2 minutes, plus 12 to 15 minutes to build and run the test suite (all with parallel make, although parallel make for tests is disabled automatically for some subdirectories). 200 minutes on current (amd64) hardware sounds quite excessive.
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
On 2018-03-14 07:40, Paul Gevers wrote: > Hi Aurelien, > > On 14-03-18 00:15, Aurelien Jarno wrote: > > On 2018-03-13 21:13, Paul Gevers wrote: > >> On Mon, 18 May 2015 15:47:34 -0300 Antonio Terceiro > >> wrote: > >>> The glibc test runs times out at ci.debian.net after running for ~3h, > >>> apparently since they were introduced: > >>> https://ci.debian.net/packages/g/glibc/unstable/amd64/ > >> > >> Is there any hope to have this fixed? > > > > I can drop the autopkg tests entries in the next upload if that can help. > > No, that doesn't help anything, as the current situation already > achieves the same thing. > > We are on the verge of enabling autopkgtest influence on > unstable-to-testing migration and only a working autopkgtest changes > anything there. Having glibc autopkgtest appears to me like a > very-nice-to-have. Ok. > Do you know why the autopkgtest runs so long? As Antonio mentioned in > other similar bug reports, the time out only counts for individual test > cases, so if you can break down the autopkgtest into smaller fragments, > the time out may not be an issue. I have no idea. On a fast 4-cores amd64 machine and for the 3 flavours built on amd64, the glibc takes around 20 minutes to build and the testsuite around 2h to run. I am not sure the testsuite is ran in parallel on ci.debian.net. > I haven't checked if glibc does, but remember that the idea is to test > as-installed packages, so rebuilding should be avoided unless needed for > the test itself. Even then, when test need binaries/artifacts that > aren't build by default, one could create a binary package containing > these artifacts in the regular build and depend on it while > autopkgtesting. E.g. MariaDB/MySQL do that. This is not something possible with the current upstream build system. Patches are welcomed though. Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: PGP signature
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
Hi Aurelien, On 14-03-18 00:15, Aurelien Jarno wrote: > On 2018-03-13 21:13, Paul Gevers wrote: >> On Mon, 18 May 2015 15:47:34 -0300 Antonio Terceiro >> wrote: >>> The glibc test runs times out at ci.debian.net after running for ~3h, >>> apparently since they were introduced: >>> https://ci.debian.net/packages/g/glibc/unstable/amd64/ >> >> Is there any hope to have this fixed? > > I can drop the autopkg tests entries in the next upload if that can help. No, that doesn't help anything, as the current situation already achieves the same thing. We are on the verge of enabling autopkgtest influence on unstable-to-testing migration and only a working autopkgtest changes anything there. Having glibc autopkgtest appears to me like a very-nice-to-have. Do you know why the autopkgtest runs so long? As Antonio mentioned in other similar bug reports, the time out only counts for individual test cases, so if you can break down the autopkgtest into smaller fragments, the time out may not be an issue. I haven't checked if glibc does, but remember that the idea is to test as-installed packages, so rebuilding should be avoided unless needed for the test itself. Even then, when test need binaries/artifacts that aren't build by default, one could create a binary package containing these artifacts in the regular build and depend on it while autopkgtesting. E.g. MariaDB/MySQL do that. Paul signature.asc Description: OpenPGP digital signature
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
On 2018-03-13 21:13, Paul Gevers wrote: > On Mon, 18 May 2015 15:47:34 -0300 Antonio Terceiro > wrote: > > The glibc test runs times out at ci.debian.net after running for ~3h, > > apparently since they were introduced: > > https://ci.debian.net/packages/g/glibc/unstable/amd64/ > > Is there any hope to have this fixed? I can drop the autopkg tests entries in the next upload if that can help. Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: PGP signature
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
On Mon, 18 May 2015 15:47:34 -0300 Antonio Terceiro wrote: > The glibc test runs times out at ci.debian.net after running for ~3h, > apparently since they were introduced: > https://ci.debian.net/packages/g/glibc/unstable/amd64/ Is there any hope to have this fixed? > I am blacklisting glibc for now, and will revisit that when this bug gets > closed. Soon, we want to use autopkgtest results to influence unstable-to-testing migration. It would be a shame to have glibc missing. Paul signature.asc Description: OpenPGP digital signature
Bug#785651: glibc: test run times out on ci.debian.net; maybe don't force a build every time
Source: glibc Severity: normal User: autopkgtest-de...@lists.alioth.debian.org Usertags: autopkgtest Hi, The glibc test runs times out at ci.debian.net after running for ~3h, apparently since they were introduced: http://ci.debian.net/packages/g/gutenprint/unstable/amd64/ It seems that it's just the build that is taking forever on that hardware. Would it be possible to not rebuild every time, and just run tests against the installed libc? I am blacklisting glibc for now, and will revisit that when this bug gets closed. -- System Information: Debian Release: stretch/sid APT prefers buildd-unstable APT policy: (500, 'buildd-unstable'), (500, 'unstable'), (500, 'testing'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 4.0.0-1-amd64 (SMP w/4 CPU cores) Locale: LANG=pt_BR.UTF-8, LC_CTYPE=pt_BR.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) -- Antonio Terceiro signature.asc Description: Digital signature