Re: FTBFS with parallel make
On Fri, 26 Jan 2018 18:21:00 + Niels Thykier wrote: > Please keep in mind that compat 9 and earlier could use --parallel (and > compat 10+ can still use --no-parallel or --max-parallel), so these > numbers are at best rough guesstimates for the number of packages with > parallel enabled/disabled. Maybe adding lintian info "using --no-parallel" is better to track. -- Regards, Hideki Yamane henrich @ debian.org/iijmio-mail.jp
Re: FTBFS with parallel make
Jeremy Bicha: > On Fri, Jan 26, 2018 at 9:38 AM, Andrey Rahmatullin wrote: >> I wonder what percent of the packages has compat < 10. > > Well start with > https://lintian.debian.org/tags/package-uses-deprecated-debhelper-compat-version.html > > Thanks, > Jeremy Bicha > For the curious: * Deprecated compat levels: ~5300 packages * Compat 9: ~14800 packages * Compat 10: ~6700 packages * Compat 11: Unknown (but at most 1132) Please keep in mind that compat 9 and earlier could use --parallel (and compat 10+ can still use --no-parallel or --max-parallel), so these numbers are at best rough guesstimates for the number of packages with parallel enabled/disabled. Thanks, ~Niels Data sources: * https://lintian.debian.org/tags/package-uses-deprecated-debhelper-compat-version.html * https://lintian.debian.org/tags/package-uses-old-debhelper-compat-version.html
Re: FTBFS with parallel make
On Fri, Jan 26, 2018 at 07:38:21PM +0500, Andrey Rahmatullin wrote: > On Fri, Jan 26, 2018 at 03:07:25PM +0100, Adam Borowski wrote: > > So this is _already_ done; at most individual packages might not use this > > option, either via an explicit opt-out, or by having old crufty packaging. > I wonder what percent of the packages has compat < 10. For packages in jessie (as in the original post), I believe the percent is quite high. -- ⢀⣴⠾⠻⢶⣦⠀ The bill with 3 years prison for mentioning Polish concentration ⣾⠁⢰⠒⠀⣿⡁ camps is back. What about KL Warschau (operating until 1956)? ⢿⡄⠘⠷⠚⠋⠀ Zgoda? Łambinowice? Most ex-German KLs? If those were "soviet ⠈⠳⣄ puppets", Bereza Kartuska? Sikorski's camps in UK (thanks Brits!)?
Re: FTBFS with parallel make
On Fri, Jan 26, 2018 at 9:38 AM, Andrey Rahmatullin wrote: > I wonder what percent of the packages has compat < 10. Well start with https://lintian.debian.org/tags/package-uses-deprecated-debhelper-compat-version.html Thanks, Jeremy Bicha
Re: FTBFS with parallel make
On Fri, Jan 26, 2018 at 03:07:25PM +0100, Adam Borowski wrote: > So this is _already_ done; at most individual packages might not use this > option, either via an explicit opt-out, or by having old crufty packaging. I wonder what percent of the packages has compat < 10. -- WBR, wRAR signature.asc Description: PGP signature
Re: FTBFS with parallel make
On Fri, Jan 26, 2018 at 09:42:05AM +0100, Philipp Hahn wrote: > we (Univention GmbH) rebuild packages (from Debian-Jessie or newer) > using "-j8". Several builds failed, but work when I use "-j1": > IMHO with todays multi-core CPUs -jX is a must: > - If I rebuild the full distribution, building multiple packages each > with -j1 is okay. > - But if I need a single package not being able to use -jX is a NO-GO: > For Spectre I re-build gcc-4.9 and it took 13h15m with -j1, because -j8 > failed with the above error. > > What do you thing: If parallel build a worthy goal? Well, it is worthy, but let's take a look at buildd options: * amd64: parallel=4 * arm64: parallel=6 * armel: parallel=4 * armhf: parallel=4 * i386: parallel=4 * mips: parallel=4 * mips64el: parallel=4 * mipsel: parallel=4 * ppc64el:parallel=4 * s390x: parallel=2 * alpha: parallel=2 * hppa: parallel=1 * hurd-i386: parallel=1 * ia64: parallel=2 * kfreebsd-amd64: (buildd down) * kfreebsd-i386: (buildd down) * m68k: nobench nocheck * powerpc:parallel=4 * powerpcspe: nobench nocheck * ppc64: parallel=32 * sh4:nobench nocheck * sparc64:parallel=32 * x32:parallel=2 (May differ per-buildd, I listed only one per arch.) So this is _already_ done; at most individual packages might not use this option, either via an explicit opt-out, or by having old crufty packaging. Most of those packages are too small to bother; if you find one that fails despite having a lengthy build, it'd be worth reporting. However: > * perl-5.20.2-3+deb8u9 > * systemd-215-17+deb8u7 > * exim-4.84.2-2+deb8u4 > * freeradius-2.2.5+dfsg-0.2+deb8u1 > * mysql-5.5.58-0+deb8u1: Bug#857059 > * libpaper: Bug#857058 > * isc-dhcp: Bug#651922 > * e2fsprogs: (attachment) You're not expecting wishlist bugs to be fixed in oldstable, are you? > if someone™ can also donate CPU time to check that -j`nproc` works. As others in the thread have already mentioned, -j is an old broken option that probably should be removed, -J is the one to use; the preferred way is to set DEB_BUILD_OPTIONS=parallel=auto, although I think this is already the default these days. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ The bill with 3 years prison for mentioning Polish concentration ⣾⠁⢰⠒⠀⣿⡁ camps is back. What about KL Warschau (operating until 1956)? ⢿⡄⠘⠷⠚⠋⠀ Zgoda? Łambinowice? Most ex-German KLs? If those were "soviet ⠈⠳⣄ puppets", Bereza Kartuska? Sikorski's camps in UK (thanks Brits!)?
Re: FTBFS with parallel make
On Fri, Jan 26, 2018 at 07:02:52AM -0500, Roberto C. Sánchez wrote: > That is interesting. I build using gbp/cowbuilder and so I set these > environment variables: > > DEB_BUILD_OPTIONS="parallel=`nproc`" DH_VERBOSE=1 > > I was not previously aware of the distinction between -j and -J for > dpkg-buildpackage. However, looking at the dpkg-buildpackage man page > there does not appear to be a DEB_BUILD_OPTIONS setting to trigger the > use of -J in place of -j. At least, that is the case on stretch. Is > there an easy way (preferrably via environment variables) to achieve > that? AFAIK -J *is* DEB_BUILD_OPTIONS=parallel, as opposed to -j setting MAKEFLAGS which directly affects upstream makefiles. -- WBR, wRAR signature.asc Description: PGP signature
Re: FTBFS with parallel make
On Fri, Jan 26, 2018 at 02:37:08PM +0500, Andrey Rahmatullin wrote: > On Fri, Jan 26, 2018 at 09:42:05AM +0100, Philipp Hahn wrote: > > we (Univention GmbH) rebuild packages (from Debian-Jessie or newer) > > using "-j8". > Is that a dpkg-buildpackage option? It's documented to fail on certain > packages, you need to use -J instead, and maintainers need to certify that > a package can be built in parallel by bumping the debhelper compat level > or passing appropriate flags to debhelper tools. > That is interesting. I build using gbp/cowbuilder and so I set these environment variables: DEB_BUILD_OPTIONS="parallel=`nproc`" DH_VERBOSE=1 I was not previously aware of the distinction between -j and -J for dpkg-buildpackage. However, looking at the dpkg-buildpackage man page there does not appear to be a DEB_BUILD_OPTIONS setting to trigger the use of -J in place of -j. At least, that is the case on stretch. Is there an easy way (preferrably via environment variables) to achieve that? Regards, -Roberto -- Roberto C. Sánchez
Re: FTBFS with parallel make
On Fri, Jan 26, 2018 at 09:42:05AM +0100, Philipp Hahn wrote: > we (Univention GmbH) rebuild packages (from Debian-Jessie or newer) > using "-j8". Is that a dpkg-buildpackage option? It's documented to fail on certain packages, you need to use -J instead, and maintainers need to certify that a package can be built in parallel by bumping the debhelper compat level or passing appropriate flags to debhelper tools. > What do you thing: If parallel build a worthy goal? Yes. Please test packages and submit patches. > With all the reproducible build stuff going on, I think it would be nice > if someone™ can also donate CPU time to check that -j`nproc` works. No, building the package with -j`nproc` is not enough, you also need to be sure the result works correctly (and repro is not very useful for this until 100% coverage). -- WBR, wRAR signature.asc Description: PGP signature
Re: FTBFS with parallel make
On 14929 March 1977, Philipp Hahn wrote: > we (Univention GmbH) rebuild packages (from Debian-Jessie or newer) > using "-j8". Several builds failed, but work when I use "-j1": > With all the reproducible build stuff going on, I think it would be nice > if someone™ can also donate CPU time to check that -j`nproc` works. Very well volunteered, you are running the builds already, so go the next step and submit bugs and patches based on them. Not very useful for others to waste time in doing the thing you already run. -- bye, Joerg
