Bug#663762: debian-policy: default for DEB_BUILD_OPTIONS=parallel=N
On Sun, Mar 25, 2012 at 09:18:18AM +1030, Ron wrote: My feeling is that it is reasonable to expect the project buildds to explicitly set parallel=N to a value that is suitable for them We already do this. Kurt -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#663762: debian-policy: default for DEB_BUILD_OPTIONS=parallel=N
On Sun, Mar 25, 2012 at 09:18:18AM +1030, Ron wrote: Hi Jakub, Russ, On Sat, Mar 17, 2012 at 04:55:19PM -0700, Russ Allbery wrote: Jakub Wilk jw...@debian.org writes: How should packages behave if there is no explicit parallel=N in DEB_BUILD_OPTIONS? I saw two different approaches: 1) Behave (roughly) like if parallel=1 was set. 2) Be clever and try to guess the right level of parallelism, e.g. by using getconf _NPROCESSORS_ONLN or parsing /proc/cpuinfo (ugh!). On an hyperthreaded system, getconf _NPROCESSORS_ONLN return the number of threads and not the number of cores, so it is not always what the user want. Cheers, -- Bill. ballo...@debian.org Imagine a large red swirl here. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#663762: debian-policy: default for DEB_BUILD_OPTIONS=parallel=N
On Tue, Mar 13, 2012 at 23:10:46 +0100, Jakub Wilk wrote: Package: debian-policy Severity: wishlist How should packages behave if there is no explicit parallel=N in DEB_BUILD_OPTIONS? I saw two different approaches: 1) Behave (roughly) like if parallel=1 was set. 2) Be clever and try to guess the right level of parallelism, e.g. by using getconf _NPROCESSORS_ONLN or parsing /proc/cpuinfo (ugh!). Could the Policy clarify which approach is correct? Thanks! I think 1 would make most sense, and avoid breaking expectations, just like 'make' without -j runs one job at a time. Cheers, Julien signature.asc Description: Digital signature
Bug#663762: debian-policy: default for DEB_BUILD_OPTIONS=parallel=N
On Sun, Mar 25, 2012 at 04:44:23PM +0200, Bill Allombert wrote: On Sun, Mar 25, 2012 at 09:18:18AM +1030, Ron wrote: Hi Jakub, Russ, On Sat, Mar 17, 2012 at 04:55:19PM -0700, Russ Allbery wrote: Jakub Wilk jw...@debian.org writes: How should packages behave if there is no explicit parallel=N in DEB_BUILD_OPTIONS? I saw two different approaches: 1) Behave (roughly) like if parallel=1 was set. 2) Be clever and try to guess the right level of parallelism, e.g. by using getconf _NPROCESSORS_ONLN or parsing /proc/cpuinfo (ugh!). On an hyperthreaded system, getconf _NPROCESSORS_ONLN return the number of threads and not the number of cores, so it is not always what the user want. It's actually somewhat tricky to reliably separate the number of 'real' cores from the number of 'virtual' cores on such systems (though it can be done, and the ease of performing such a query probably could be improved if there was a proven need for it). In practice though, on a 4 core HT system reporting it has 8 cores, make -j8 is actually almost always what you want. I have several such systems and it it doesn't burden them any harder than -j4 on a strictly 4-core system does, and it does build many or most things ~2x as fast as -j4 would. In fact given the way that recursive make works, one quite surprising point is that some things will actually benefit with a speedup of notably greater than N for a given -jN. Witness a factor of 8 walltime speedup on a non-HT 4 core machine: (with hot cache and both tests run several times) $ time make clean real0m4.194s user0m3.980s sys 0m1.944s $ time make -j4 clean real0m0.498s user0m1.780s sys 0m0.460s I've seen larger multipliers than that too, but obviously this depends more on the work that make is doing at each of its levels, than on the degree of CPU loading. The time saving is very real just the same though. Best, Ron -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#663762: debian-policy: default for DEB_BUILD_OPTIONS=parallel=N
On Sun, 25 Mar 2012, Bill Allombert wrote: On Sun, Mar 25, 2012 at 09:18:18AM +1030, Ron wrote: On Sat, Mar 17, 2012 at 04:55:19PM -0700, Russ Allbery wrote: Jakub Wilk jw...@debian.org writes: How should packages behave if there is no explicit parallel=N in DEB_BUILD_OPTIONS? I saw two different approaches: 1) Behave (roughly) like if parallel=1 was set. 2) Be clever and try to guess the right level of parallelism, e.g. by using getconf _NPROCESSORS_ONLN or parsing /proc/cpuinfo (ugh!). On an hyperthreaded system, getconf _NPROCESSORS_ONLN return the number of threads and not the number of cores, so it is not always what the user want. IMO, this is best left for the local admin. Guessing this kind of stuff right in every arch is a very hard problem, and the safe choice [which doesn't break any package builds] is to not do a parallel build at all. For the record: if you ever thought about using crap like getconf _NPROCESSORS_ONLN for anything, you will probably benefit from looking at the hwloc and libhwloc* packages. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#663762: debian-policy: default for DEB_BUILD_OPTIONS=parallel=N
Hi Jakub, Russ, On Sat, Mar 17, 2012 at 04:55:19PM -0700, Russ Allbery wrote: Jakub Wilk jw...@debian.org writes: How should packages behave if there is no explicit parallel=N in DEB_BUILD_OPTIONS? I saw two different approaches: 1) Behave (roughly) like if parallel=1 was set. 2) Be clever and try to guess the right level of parallelism, e.g. by using getconf _NPROCESSORS_ONLN or parsing /proc/cpuinfo (ugh!). Could the Policy clarify which approach is correct? Thanks! I think status quo is generally do whatever upstream does by default, which is slightly different than either of those. It may be (2) if upstream is that smart, but it's different than (1) in that we don't actively suppress parallelization if upstream enables something by default. My conservative inclination would be towards (1). Speaking as someone who is upstream for a number of pieces of software that are both packaged for Debian and built by non-Debian users in a more 'conventional' manner -- and as someone who for quite a long time now has been defaulting any non-trivial package that would benefit from it to use _NPROCESSORS_ONLN (or something similar as appropriate for the package in question) -- I was actually very interested in being the sort of 'smart' upstream that Russ talks about, and looked at making this the default for unpackaged builds some time ago as well ... But it turns out, that's not actually such a smart thing for an upstream build system to do. The main reason that I abandoned it, despite wanting all my users to benefit from fast build times, without wasting the multi- core systems that most of them now own, is that it actually becomes quite difficult to then let those users override that build system's choice of parallel level. There is no standard equivalent in autoconf to the D_B_O parallel=N option. So the only way to do that is to invent some other non-standard thing. Which in the simple case of ./configure make -jN seemed kind of backwards in terms of both convenience and common knowledge. For packages though it's a different problem. Having to explicitly set D_B_O is not convenient (especially if you are throwing the package off to a remote build machine somewhere), and knowing what a good value to set it to for any random package you download, may not be obvious either. Especially for packages that may now be using the simplified helpers, which could permit them building in parallel if that was specified, even if the package was not actually confirmed to be safe for that. So my inclination is that it should be the *packagers* who make a best guess at this (much the same way we expect them to provide a sane default configuration when the package is installed) -- and that it be done *in* the packages, rather than encouraging upstreams to do this instead. There are some packages (such as the toolchain set) which also take into account things like available memory when deciding how deeply parallel they should run. Familiarity with the package is the key to providing good defaults for casual users. So while I'd probably agree that it would be a mistake for policy to strongly mandate that all packages should do this -- I'd also strongly object to policy outlawing it too. The future is filled with multi- core processors, and clinging to a single threaded past will just make people wonder why they wasted their money on them. My feeling is that it is reasonable to expect the project buildds to explicitly set parallel=N to a value that is suitable for them, and that it is reasonable to believe that when a 'home' user wants to build a package from source, they want that build to finish as quickly as possibly so that they can get on with the reason they were building it in the first place as soon as possible. It's not like running make -j4 on a 4 core machine will actually bring it to its knees and make it unusable for other simultaneous tasks -- any more than make -j1 does on a single core machine. One thing policy might usefully clarify though is the idea that the parallel=N setting is an *upper* bound. And that packages which know they are going to consume a lot of memory or other resources with each thread are free to use any number of threads *up to* N, but aren't strictly constrained to only using either 1 or N, with nothing between if they also have some other heuristic that may guide their choice. I expect only a fairly small number of packages might need that extra freedom though. Anyhow, thanks for bringing this question up. We discussed it at some length on #d-d several months back and never really got to a consensus. People fairly evenly split over whether N=1 or N=N should be the default, and about the only thing we were all in perfect agreement about was: if people don't like 'my' preferred default, they can set D_B_O parallel= for themselves explicitly to override it!. Which works equally well whichever side of the consensus you happened to be on :)
Bug#663762: debian-policy: default for DEB_BUILD_OPTIONS=parallel=N
Jakub Wilk jw...@debian.org writes: How should packages behave if there is no explicit parallel=N in DEB_BUILD_OPTIONS? I saw two different approaches: 1) Behave (roughly) like if parallel=1 was set. 2) Be clever and try to guess the right level of parallelism, e.g. by using getconf _NPROCESSORS_ONLN or parsing /proc/cpuinfo (ugh!). Could the Policy clarify which approach is correct? Thanks! I think status quo is generally do whatever upstream does by default, which is slightly different than either of those. It may be (2) if upstream is that smart, but it's different than (1) in that we don't actively suppress parallelization if upstream enables something by default. My conservative inclination would be towards (1). -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#663762: debian-policy: default for DEB_BUILD_OPTIONS=parallel=N
Package: debian-policy Severity: wishlist How should packages behave if there is no explicit parallel=N in DEB_BUILD_OPTIONS? I saw two different approaches: 1) Behave (roughly) like if parallel=1 was set. 2) Be clever and try to guess the right level of parallelism, e.g. by using getconf _NPROCESSORS_ONLN or parsing /proc/cpuinfo (ugh!). Could the Policy clarify which approach is correct? Thanks! -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
