Re: make -j in Debian packages
On Thu, Jul 06, 2006 at 11:58:28PM +0200, Adam Borowski wrote: program X consist of a number of C files; it seems like compiling every file takes around 24MB, Like I said, there's just too many variables. Also, I wouldn't be interested in figuring out how much RAM the build takes if I were to maintain a package like, say, X or so. No one forces you to do so. No one even said a word about making concurrency mandatory. It's just a way to make builds faster on machines that are beefy enough (ie, virtually all). Again, I don't think it's that important to improve speed on already fast machines, but to improve speed on those that aren't that fast. When a *user* wants to recompile a package on his machine, he *may* want to improve the speed as well, but I don't think that this should be a major problem. However speeding up the buildd process on buildds would be good for the whole project. Before you've proven that this is indeed possible, I don't think there's much point in this whole exercise; otherwise there *is* going to be a problem with you overloading build machines, and *you will* get bugs filed about that (from me, at the very least). Here, you got a piece of working code, is that not a good enough proof? Tell me how exactly a build using 4 processes using ~24MB each overloading a build machine which has 512+MB ram. And builds which do require 1+GB of memory, well, simply are not going to use any concurrency. Somehow I get the impression you're over-estimating the average buildd. This lists the average RAM per arch for those registered on buildd.net: buildd=# select arch, count(*) as num, round(avg(ram)) as avg RAM from status group by arch order by arch; arch | num | avg RAM +-+- alpha | 5 |1178 amd64 | 3 |2219 arm| 9 | 56 armeb | 3 | 32 hppa | 2 | 896 hurd-i386 | 1 | 248 i386 | 3 | 512 ia64 | 2 |4096 kfreebsd-amd64 | 1 | 512 kfreebsd-i386 | 2 | 512 m32r | 1 | 128 m68k | 24 | 148 mips | 11 | 135 mipsel | 5 | 96 powerpc| 4 | 336 s390 | 4 | 256 sparc | 7 | 969 (17 rows) When you look at such archs as arm, armeb and mipsel you'll see that your assumption of 512 MB RAM for a buildd is a little bit off the reality. Even the s390 buildds just have 256 MB RAM. Alpha, amd64 and ia64 are exceptional, I think, especially ia64. Additionally you have to consider that some buildds are also accessible by DDs, e.g. crest.d.o, so it may happen that on those boxes DDs are compiling their packages in parallel to the buildd, giving 3-5 parallel compilers in total. And note that the system I propose actually _limits_ concurrency in some cases. The whole thread started with gem packages choking the m68k build. In fact, gem was building just fine on that particular buildd. I know that, because it was one of my m68k buildds and I just wondered about the parallel makes and if that is allowed and asked Wouter about it, but otherwise the build went fine. It's a big package, and the maintainer rightfully thinked that it is completely useless on small arches. The When he would have thought this, he would have excluded those archs from the arch list. And it's not even a big package. When you look at http://buildd.net/cgi/ptracker.cgi?unstable_pkg=gemsearchtype=m68k you can see that even armeb built it in a reasonable time (armeb just have 32M, see above). optimization he employed was to use -j4 -- something that can't hurt a machine on arches where the package is usable. Too bad, the maintainer's fault was that he didn't protect the -j4 from arches which can't handle it. And handling this, exceptionally rare in normal use, case is what we can fix. Even you, in the initial post of this thread, proposed a way to enable concurrency in some cases, so this can't be such a bad idea :p Well, Wouter asked if usage of -jX is something mentioned in policy or such and when it's not mentioned there, how to handle those packages. Nobody objects that concurrency *can* improve the buildtime, but the big question is: how can we find a way to implement this in a good way? To achieve this, it's not that good to argue some points over and over again. I'm still in favour of my proposal to use the existing infrastructure (in lack of something better) and elaborate the pro and cons of one way or the other. When there are results, the project can decide what method to use and if it wants to use it at all. Any takers? My machines are quite busy at the moment and I'm in the progress of moving, so I'm not a good candidate, but would be willing to assist in some way or another, if I can. -- Ciao...//Fon: 0381-2744150 Ingo \X/ SIP: [EMAIL
Re: make -j in Debian packages
Adam Borowski [EMAIL PROTECTED] writes: On Sun, Jul 02, 2006 at 11:57:50AM +0200, Wouter Verhelst wrote: Additionally, it puzzles me how you think a maintainer will be able to accurately predict how much RAM a certain build is going to use. There are so many variables, that I think anything but 'this is the fastest way to build it on my machine' is going to be unfeasible. Ram usage between architectures should not vary more than a factor of 4. Much less in normal cases. Let's say: program X consist of a number of C files; it seems like compiling every file takes around 24MB, with the final link taking much more[1]. I guess this can be called typical, in C you need to store just the current file and the headers you use. Now, let's assume we use not the simple snippet I wrote [2] but a concurrency-helper with the interface Goswin described, with some unknown logic inside. The maintainer thus declares that package X takes 24MB, and says it's good to use heavy concurrency: concurrency-helper --ram-estimate 24 --more-concurrent The machine is a mid-range user box, with 512MB ram. Thus, if the helper decides to go with -j4, the safety margin is _5_ times. I guess you can trust people to be at least within _that_ error range. And even if they fail, you can always force the build to use -j1. The amount of paging can also be gathered during a build and included in the buildd log. Too bad avgtext / avgdata / maxresident accounting is broken and always gives 0 (e.g. run /usr/bin/time ls). If, let's say, the machine is a high-end one with 2GB ram, it runs 4 buildds at once and the admin didn't specify his preferences, using -j4 won't be any worse than on the user box mentioned above. And the concurrency-helper could easily know about 4 buildds being run in parallel and assume only 512MB ram and 1/4 the number of cpus being available per buildd for its calculations. The point of the helper is to remove the decision from the package alone to a central place that is easily configurable for a wide range of cases. [1]. I was once forced to do a kernel compile on a critically memory starved box. Going from .c to .o went quite smoothly, but the final link was an unholy swappeathon that took hours. [2]. My idea was to simply go with -j1 if the machine has less than X memory, or with a given constant otherwise. Cheers, MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Thu, Jul 06, 2006 at 11:58:28PM +0200, Adam Borowski wrote: On Wed, Jul 05, 2006 at 08:23:00PM +0200, Wouter Verhelst wrote: On Mon, Jul 03, 2006 at 03:04:14PM +0200, Adam Borowski wrote: On Sun, Jul 02, 2006 at 11:57:50AM +0200, Wouter Verhelst wrote: Additionally, it puzzles me how you think a maintainer will be able to accurately predict how much RAM a certain build is going to use. There are so many variables, that I think anything but 'this is the fastest way to build it on my machine' is going to be unfeasible. Let's say: program X consist of a number of C files; it seems like compiling every file takes around 24MB, Like I said, there's just too many variables. Also, I wouldn't be interested in figuring out how much RAM the build takes if I were to maintain a package like, say, X or so. No one forces you to do so. No one even said a word about making concurrency mandatory. It's just a way to make builds faster on machines that are beefy enough (ie, virtually all). You're not convincing me that you can indeed accurately predict for every compiled language out there how much RAM you're going to need during the build. So what? If you're not sure, you simply don't provide your prediction. And with a huge safety margin, you would have to be MALICIOUS to make a mistake. Before you've proven that this is indeed possible, I don't think there's much point in this whole exercise; otherwise there *is* going to be a problem with you overloading build machines, and *you will* get bugs filed about that (from me, at the very least). Here, you got a piece of working code, is that not a good enough proof? Tell me how exactly a build using 4 processes using ~24MB each overloading a build machine which has 512+MB ram. And builds which do require 1+GB of memory, well, simply are not going to use any concurrency. And note that the system I propose actually _limits_ concurrency in some cases. The whole thread started with gem packages choking the m68k build. It's a big package, and the maintainer rightfully thinked that it is completely useless on small arches. Err, AIUI, ruby gems are a way to automatically install extras to a running ruby environment, much in the same way that the CPAN module is used for Perl. I fail to see why this would be completely useless on smaller architectures. If you want to use ruby there, you may want to use gem. -- Fun will now commence -- Seven Of Nine, Ashes to Ashes, stardate 53679.4 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Fri, Jul 07, 2006 at 03:34:59PM +0200, Wouter Verhelst wrote: Err, AIUI, ruby gems are a way to automatically install extras to a running ruby environment, much in the same way that the CPAN module is used for Perl. I fail to see why this would be completely useless on smaller architectures. If you want to use ruby there, you may want to use gem. Uhm, gem seems to be a X11 package dealing with graphics rendering and animation. grep -ir ruby on its source doesn't show anything, too. Or am I missing something? Also: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 18425 kilobyte 25 0 20904 15m 5660 R 29.5 3.2 0:01.00 cc1plus 18428 kilobyte 25 0 19880 14m 5552 R 23.6 2.9 0:00.71 cc1plus 18436 kilobyte 25 0 17496 10m 3232 R 10.3 2.0 0:00.31 cc1plus 18259 kilobyte 15 0 4940 1392 1008 S 1.7 0.3 0:00.59 mpc123 18426 kilobyte 23 0 5004 2372 1468 S 1.7 0.5 0:00.06 g++ 18434 kilobyte 25 0 5004 2376 1468 S 1.7 0.5 0:00.05 g++ 18423 kilobyte 16 0 2264 1152 852 R 0.3 0.2 0:00.02 top Doesn't sound that deadly... -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Friday 07 July 2006 19:06, Adam Borowski wrote: On Fri, Jul 07, 2006 at 03:34:59PM +0200, Wouter Verhelst wrote: Err, AIUI, ruby gems are a way to automatically install extras to a running ruby environment, much in the same way that the CPAN module is used for Perl. I fail to see why this would be completely useless on smaller architectures. If you want to use ruby there, you may want to use gem. Uhm, gem seems to be a X11 package dealing with graphics rendering and animation. grep -ir ruby on its source doesn't show anything, too. Or am I missing something? He is talking about http://rubyforge.org/projects/rubygems Find the debian source package (not uploaded yet) at: http://svn.debian.org/wsvn/pkg-ruby-extras it is in packages/libgems-ruby/ This is also worth reading: http://pkg-ruby-extras.alioth.debian.org/rubygems.html but I do believe it is possible to install ruby extras (just like perl modules) as both: debian packages (packaging system domain) and via RubyGems (local admin domain) -- pub 4096R/0E4BD0AB 2003-03-18 people.fccf.net/danchev/key pgp.mit.edu fingerprint 1AE7 7C66 0A26 5BFF DF22 5D55 1C57 0C89 0E4B D0AB -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Fri, Jul 07, 2006 at 02:46:15PM +0200, Goswin von Brederlow wrote:
The point of the helper is to remove the decision from the package
alone to a central place that is easily configurable for a wide range
of cases.
Ok, here goes my stab at the helper:
(attached)
Usage:
$(MAKE) -j`debian/concurrency-helper`
(note: you'll need to either chmod +x or insert perl after the
first backtick)
Maintainer's manipulation:
on command line
User's manipulation:
ENV var
joeyh's manipulation:
upgrade clause
I see that the rules I used are probably way too cowardly memory-wise
but way too fork-happy where the number of CPUs is concerned. But oh
well, it's not me who is supposed to set the defaults anyway :p
This time an example isn't really needed, but just in case I've used
kbtin as the test package. mentors.debian.net seems to be down, so:
deb-src http://angband.pl/debian unstable main
dget http://angband.pl/debian/kbtin/kbtin_1.0.7-1.dsc
(in an uploadeable state, in case you'll want to test it against real
buildds :p)
Cheers,
--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.
#!/usr/bin/perl -w
use strict;
use integer;
#Upgrade clause:
my $dh='/usr/bin/dh_concurrency';
exec {$dh} $dh,@ARGV if -x $dh;
=head1 NAME
concurrency-helper - guess a reasonable concurrency level for make
=head1 SYNOPSIS
$(MAKE) -j`Bconcurrency-helper [SIoptions]`
=head1 DESCRIPTION
concurrency-helper is a program that will guess whether running more parallel
Imake jobs would speed up the build, and write a reasonable argument for I-j
on the standard output.
If the environment variable BCONCURRENCY_LEVEL is set, it will override any
guesses.
=head1 OPTIONS
This is the interface for the package maintainer.
=over 4
=item B--ram-estimate IX [IY]
Give some indication of ram usage, in megabytes. If the host has too
little ram the concurency will be tuned down to prevent swapping. A
broad indication +- a factor of 2 is probably sufficient. The [Y] would
be to indicate ram usage for 32bit and 64bit archs seperately. Given the
pointer size ram usage can vary between them a lot.
=item B--more-concurrent
Indicate that there are lots of small files that greatly benefit from
interleaving I/O with cpu time. Try to use more concurency than cpus.
=item B--max-concurrency IX
Limit the concurrency to no more than X. Shouldn't be ever needed, you're
either SMP-safe or not.
=back
=head1 CAVEATS
This helper can't detect multiple builds running on the same host. In
that case, just set BCONCURRENCY_LEVEL appropiately; to 1 if you want
to disable any parallelism altogether.
=head1 SEE ALSO
Lmake(1)
=head1 AUTHOR
Adam Borowski [EMAIL PROTECTED]
=cut
#Leaving no output would lead to bare -j, that is, -j INFINITY.
sub die_1($)
{
print 1\n unless -t STDOUT;
die concurrenct-helper: $_[0];
}
if (defined $ENV{'CONCURRENCY_LEVEL'})
{
$ENV{'CONCURRENCY_LEVEL'}=~/^(\d+)$/ $10
or die_1 E: CONCURRENCY_LEVEL malformed.\n;
print $1\n;
exit;
}
my ($mem,$cpus); #machine's specs
my ($ram32,$ram64,$limit,$extra); #arguments
my $ram;
open FREE, free| or die_1 W: can't run 'free'.\n;
while(FREE)
{
$mem=$1/1024 if /^Mem:\s+(\d+)/;
}
close FREE;
$mem or die_1 W: can't get the amount of memory.\n;
if (open CPUS, /proc/cpuinfo)
{
$cpus=scalar grep(/^processor\s/,CPUS);
close CPUS;
}
$cpus=1 unless $cpus;
while(@ARGV)
{
$_=shift;
if (/^--max-concur(|r)ency$/)
{
$_=shift
or die_1 E: --max-concurrency requires an argument.\n;
/^(\d+)$/ $10
or die_1 E: --max-concurrency expects a positive integer.\n;
$limit=$1;
}
elsif (/^--ram-estimate$/)
{
$_=shift
or die_1 E: --ram-estimate expects an argument or two.\n;
/^(\d+)$/
or die_1 E: --ram-estimate: ram32 malformed.\n;
$ram32=$1;
#The second argument is optional.
if (@ARGV $ARGV[0]=~/^(\d+)$/)
{
$ram64=$1;
shift;
}
else
{
$ram64=$ram32;
}
}
elsif (/^--more-concur(|r)en(cy|t)$/)
{
$extra=1;
}
else
{
die_1 E: invalid argument \$_\\n;
}
}
$ram=(length(pack('l!',0)) = 4)? $ram32:$ram64;
### Here the fun starts.
# Edit this part to fine-tune the logic.
#Default to 24MB per job.
$ram=24 unless $ram;
#Be a coward, use no more than 1/5 of the memory.
$_=$mem/5/$ram;
#Don't have too many jobs waiting for CPU.
my $cap=$cpus+1;
$cap*=2 if $extra;
$_=$cap if $_$cap;
#Obey --max-concurrency.
$_=$limit if (defined $limit $_$limit);
#But use at least _one_ job. We want to get the build done, don't we?
$_=1 if $_1;
print $_\n;
Re: make -j in Debian packages
On Wed, Jul 05, 2006 at 08:23:00PM +0200, Wouter Verhelst wrote: On Mon, Jul 03, 2006 at 03:04:14PM +0200, Adam Borowski wrote: On Sun, Jul 02, 2006 at 11:57:50AM +0200, Wouter Verhelst wrote: Additionally, it puzzles me how you think a maintainer will be able to accurately predict how much RAM a certain build is going to use. There are so many variables, that I think anything but 'this is the fastest way to build it on my machine' is going to be unfeasible. Let's say: program X consist of a number of C files; it seems like compiling every file takes around 24MB, Like I said, there's just too many variables. Also, I wouldn't be interested in figuring out how much RAM the build takes if I were to maintain a package like, say, X or so. No one forces you to do so. No one even said a word about making concurrency mandatory. It's just a way to make builds faster on machines that are beefy enough (ie, virtually all). You're not convincing me that you can indeed accurately predict for every compiled language out there how much RAM you're going to need during the build. So what? If you're not sure, you simply don't provide your prediction. And with a huge safety margin, you would have to be MALICIOUS to make a mistake. Before you've proven that this is indeed possible, I don't think there's much point in this whole exercise; otherwise there *is* going to be a problem with you overloading build machines, and *you will* get bugs filed about that (from me, at the very least). Here, you got a piece of working code, is that not a good enough proof? Tell me how exactly a build using 4 processes using ~24MB each overloading a build machine which has 512+MB ram. And builds which do require 1+GB of memory, well, simply are not going to use any concurrency. And note that the system I propose actually _limits_ concurrency in some cases. The whole thread started with gem packages choking the m68k build. It's a big package, and the maintainer rightfully thinked that it is completely useless on small arches. The optimization he employed was to use -j4 -- something that can't hurt a machine on arches where the package is usable. Too bad, the maintainer's fault was that he didn't protect the -j4 from arches which can't handle it. And handling this, exceptionally rare in normal use, case is what we can fix. Even you, in the initial post of this thread, proposed a way to enable concurrency in some cases, so this can't be such a bad idea :p Regards, -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Mon, Jul 03, 2006 at 03:04:14PM +0200, Adam Borowski wrote: On Sun, Jul 02, 2006 at 11:57:50AM +0200, Wouter Verhelst wrote: Additionally, it puzzles me how you think a maintainer will be able to accurately predict how much RAM a certain build is going to use. There are so many variables, that I think anything but 'this is the fastest way to build it on my machine' is going to be unfeasible. Let's say: program X consist of a number of C files; it seems like compiling every file takes around 24MB, In the maintainer's environment... yes, we're all running Debian Unstable to do our builds, but that's not enough to be sure that the build needs indeed so and so much RAM. For examle, you may be using ccache, or you may be doing stuff on an architecture which just happens to allocate 64 bits for a pointer instead of 32. Or the buildd may have just finished the compiler transition while you're a few days outdated on your local unstable chroot. Like I said, there's just too many variables. Also, I wouldn't be interested in figuring out how much RAM the build takes if I were to maintain a package like, say, X or so. You're not convincing me that you can indeed accurately predict for every compiled language out there how much RAM you're going to need during the build. Before you've proven that this is indeed possible, I don't think there's much point in this whole exercise; otherwise there *is* going to be a problem with you overloading build machines, and you *will* get bugs filed about that (from me, at the very least). -- Fun will now commence -- Seven Of Nine, Ashes to Ashes, stardate 53679.4 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Sun, Jul 02, 2006 at 11:57:50AM +0200, Wouter Verhelst wrote: Additionally, it puzzles me how you think a maintainer will be able to accurately predict how much RAM a certain build is going to use. There are so many variables, that I think anything but 'this is the fastest way to build it on my machine' is going to be unfeasible. Let's say: program X consist of a number of C files; it seems like compiling every file takes around 24MB, with the final link taking much more[1]. I guess this can be called typical, in C you need to store just the current file and the headers you use. Now, let's assume we use not the simple snippet I wrote [2] but a concurrency-helper with the interface Goswin described, with some unknown logic inside. The maintainer thus declares that package X takes 24MB, and says it's good to use heavy concurrency: concurrency-helper --ram-estimate 24 --more-concurrent The machine is a mid-range user box, with 512MB ram. Thus, if the helper decides to go with -j4, the safety margin is _5_ times. I guess you can trust people to be at least within _that_ error range. And even if they fail, you can always force the build to use -j1. If, let's say, the machine is a high-end one with 2GB ram, it runs 4 buildds at once and the admin didn't specify his preferences, using -j4 won't be any worse than on the user box mentioned above. [1]. I was once forced to do a kernel compile on a critically memory starved box. Going from .c to .o went quite smoothly, but the final link was an unholy swappeathon that took hours. [2]. My idea was to simply go with -j1 if the machine has less than X memory, or with a given constant otherwise. Cheers, -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Mon, 3 Jul 2006 15:04:14 +0200, Adam Borowski [EMAIL PROTECTED] said: On Sun, Jul 02, 2006 at 11:57:50AM +0200, Wouter Verhelst wrote: Additionally, it puzzles me how you think a maintainer will be able to accurately predict how much RAM a certain build is going to use. There are so many variables, that I think anything but 'this is the fastest way to build it on my machine' is going to be unfeasible. Let's say: program X consist of a number of C files; it seems like compiling every file takes around 24MB, with the final link taking much more[1]. I guess this can be called typical, in C you need to store just the current file and the headers you use. But the amount of RAM taken depends on many factors, such as (just off the top of my head) architecture, compiler, the libraries used. (e.g. I have a package that builds fine in my sid pbuilder on my 128MB box, but it eats memory like crazy in my sarge pbuilder on the same box, AFAIK mostly due to the older compiler.) -- Hubert Chan - email Jabber: [EMAIL PROTECTED] - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA (Key available at wwwkeys.pgp.net) Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Fri, Jun 30, 2006 at 12:12:10PM +0200, Adam Borowski wrote: On Fri, Jun 30, 2006 at 03:26:15AM +0200, Goswin von Brederlow wrote: Adam Borowski [EMAIL PROTECTED] writes: Still, the buildd admin has no way to estimate how much a sub-process of a package is going to use, the maintainer has at least a rough idea. Since the maintainer's action is needed anyway, he can as well provide this estimate. And then the buildd admin can set CONCURRENCY_LEVEL to 1 to centrally disable any parallelism of this kind. One could patch make to use concurency when the ram permits. This would involves checking the current ram usage and forking more builds if there is enough Oh, so you mean checking the _free_ RAM instead of the _physical_ RAM? This would be reasonable Not at all. It's not as if the amount of free RAM is a variable that is static throughout a machine's usage. If something else is being done on the machine in question, then whatever happens to be the amount of free RAM available at the time the test is ran doesn't have to be the amount of free RAM available at the time the most RAM is in use during the build. Additionally, it puzzles me how you think a maintainer will be able to accurately predict how much RAM a certain build is going to use. There are so many variables, that I think anything but 'this is the fastest way to build it on my machine' is going to be unfeasible. And since that's hardly useful... Have you thought about that? -- Fun will now commence -- Seven Of Nine, Ashes to Ashes, stardate 53679.4 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Fri, Jun 30, 2006 at 03:22:48AM +0200, Goswin von Brederlow wrote: The same can't be said for upstream makefiles though. Many sources don't build with -j option. I'm not sure if debian/rules should somehow enforce -j1 in those cases or if only packages that benefit from -jX should add support for some DEB_BUILD_OPTION or CONCURENCY_LEVEL env var. The later would always be the save option. The former would have lots of hidden bugs that prop up suddenly on rebuilds or security fixes. I'm strictly in favour of being on the safe side. If maintainers would start now to add -j4 to their makefiles, we would end in more FTBFS bugs, I guess, which is unnecessary and an extra burden for the porters and buildds. When we want to introduce some sort of concurrency builds, we should do it that way that the impact is as small as possible (opt-in). Start with some packages that are known to build fine with -jX and choose 1-2 buildds on some archs to implement this feature for a small test. Maybe someone should do a complete archive rebuild with -j1 and -j4 on a smp system and compare the amount of failures to get an overview how bad it would be. Are there any m68k SMP machines? No, just joking. ;) I guess, some amd64 machines are fast enough to get some results in a reasonable time. For some more tests it would be nice if Martin M. (tbm) would do a rebuild on his machine park. He already did some archive rebuilds in the past and is experienced with this. This should show some pitfalls across the different archs. If anything goes well, the procedure could be implemented on all buildds. On the other hand, making builds significantly faster is not something that you can shake a stick at. Typically make -jX is faster even on uniprocessor, and I don't need to tell you why it's much faster on SMP. Too bad, a C++ build where every file takes 1GB memory obviously should not be parallelized. Also, no one but the maintainer knows whether a package is SMP-clean or not. You cannot guess this in an automated way. That would point to using an env varibale I doubt that a single env variable would do the trick, as the discussion has shown. There are some showstoppers for smaller machines like huge C++ packages - at least when all processes ought to run on the local machine. This is a different matter when such processes would be distributed to other, more powerful machines with crosscompilers. Thus, my counter-proposal: Let's allow maintainers to use make -jX according to their common sense, requiring obeying an env variable to opt out. and let the maintainer pass that env variable down a -jX. How about this option: We write a tool concurency-helper that gets a set of requirements of the source and outputs a suitable concurency level for current build host. Requirements could include: --package pkg Let the tool know what we build. The tool can have overrides in place for packages in case special rules apply. --max-concurency X || --non-concurent Limit the concurency, possibly to 1, for sources that have problems with it. Although such sources probably just shouldn't support this. --ram-estimate X [Y] Give some indication of ram usage. If the host has too little ram the concurency will be tuned down to prevent swapping. A broad indication +- a factor of 2 is probably sufficient. The [Y] would be to indicate ram usage for 32bit and 64bit archs seperately. Given the pointer size ram usage can vary between them a lot. --more-concurent Indicate that there are lots of small files that greatly benefit from interleaving I/O with cpu time. Try to use more concurency than cpus. The tool would look at those indicators and the hosts resources in both ram and cpus and figure out a suitable concurency level for the package from there. What do you think? Of course this is a more complex approach, but I think it's an approach into the right direction. I would like to have settings for distcc to get added (like --use-distcc and --distcc-hosts) Rationale: Nearly every buildd and nearly every user building the packages on his own will benefit from -j2 [2], even on non-SMP. Unless it's a piece of heavily-templated code, any modern box will have enough memory to handle it. The maintainer know whether the code is heavily templated or not. Mips, mipsel, arm and m68k won't benefit. The ram requirement just leads to poor cache performance or even excessive swapping in general. Sources and gcc are growing and growing and the ram of the buildds stays the same. On the other hand any modern system will build the source fast and the buildd will be idle most of the time so -j2 or not hardly matters. Wouldn't that indicate a preference to -j1? Erm. Most modern systems are not in the need to necessarily speed up the build process because there are fast enough to keep up easily anyway. Using parallel makes on these systems are just
Re: make -j in Debian packages
On Fri, Jun 30, 2006 at 03:26:15AM +0200, Goswin von Brederlow wrote: Still, the buildd admin has no way to estimate how much a sub-process of a package is going to use, the maintainer has at least a rough idea. Since the maintainer's action is needed anyway, he can as well provide this estimate. And then the buildd admin can set CONCURRENCY_LEVEL to 1 to centrally disable any parallelism of this kind. One could patch make to use concurency when the ram permits. This would involves checking the current ram usage and forking more builds if there is enough as well as suspending submakes when swapping starts. But i guess that would greatly complicate make itself. And I doubt that upstream would implement this patch. OTOH, why not use something like weak-no-auto-build or the lists for timeouts in sbuild? That way each buildd admin could setup his own rules for each package. -- Ciao...//Fon: 0381-2744150 Ingo \X/ SIP: [EMAIL PROTECTED] gpg pubkey: http://www.juergensmann.de/ij/public_key.asc -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Fri, Jun 30, 2006 at 03:26:15AM +0200, Goswin von Brederlow wrote: Adam Borowski [EMAIL PROTECTED] writes: Still, the buildd admin has no way to estimate how much a sub-process of a package is going to use, the maintainer has at least a rough idea. Since the maintainer's action is needed anyway, he can as well provide this estimate. And then the buildd admin can set CONCURRENCY_LEVEL to 1 to centrally disable any parallelism of this kind. One could patch make to use concurency when the ram permits. This would involves checking the current ram usage and forking more builds if there is enough Oh, so you mean checking the _free_ RAM instead of the _physical_ RAM? This would be reasonable -- I didn't use this in the debian/rules snippet I proposed as the physical memory is a trivially discernable number while free RAM can be sometimes hard to tell, especially on Xen: Mem:588472k total, 574028k used,1k free,4k buffers Swap: 1048568k total, 24k used, 1048544k free, 238984k cached this is on a box where the only currently accessed process is mutt (+sshd and $EDITOR). Who cares if apache's processes which didn't handle a single request in two months (the real server is elsewhere) are in the physical memory? Who cares about idle dovecot? The box is for all practical purposes completely idle as of right now. Of course, when I say that something is tricky, it doesn't mean that someone with more clue than me can't do it. as well as suspending submakes when swapping starts. But i guess that would greatly complicate make itself. Actually, make -l already does something very similar. It stops -j from forking any new processes (above 1) once the system load exceeds a given value. And while the system load is a lousy measure, make won't ever starve off the build, it will in the worst case behave as make -j1, which is what Ingo wants. Cheers and schtuff, -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Fri, Jun 30, 2006 at 08:41:33AM +0200, Ingo Juergensmann wrote: On Fri, Jun 30, 2006 at 03:22:48AM +0200, Goswin von Brederlow wrote: The same can't be said for upstream makefiles though. Many sources don't build with -j option. Right, that's just what I said :p It's the upstream and the maintainer who know whether -j is safe or not -- the end-user or buildd are not expected to guess this. What the end-user and buildd are supposed to have to say about is whether they want a fast or a low-mem build. I'm not sure if debian/rules should somehow enforce -j1 in those cases That would be opt-out, and everyone here seems to be against it in this place. Just let's not confuse maintainer's opt-foo and the builder's opt-foo. or if only packages that benefit from -jX should add support for some DEB_BUILD_OPTION or CONCURENCY_LEVEL env var. The later would always be the save option. The former would have lots of hidden bugs that prop up suddenly on rebuilds or security fixes. Hell yeah. I'm strictly in favour of being on the safe side. If maintainers would start now to add -j4 to their makefiles, we would end in more FTBFS bugs, It's them who are supposed to have a clue whether their packages can handle -j or not. If they are wrong, well, unlike typical SMP issues, this is actually a bug which is pretty likely to pop up in the first build. This means before the upload is actually done, or, for abysmal maintainers, when the faster autobuilders get the package. When we want to introduce some sort of concurrency builds, we should do it that way that the impact is as small as possible (opt-in). Start with some packages that are known to build fine with -jX and choose 1-2 buildds on some archs to implement this feature for a small test. Good idea. That's why I started with a small but not tiny (1MB of sources) package of very little importance. It appears to build fine, and when I built it on a box with 24MB of memory, the machine didn't croak. So, here: take either this test package (kbtin), or, even better, take the snippet between the marked () lines in debian/rules and apply it to a bigger package. I claim that it will handle both small and big machines well by default, and obey CONCURRENCY_LEVEL if it's set. This is not intelligent design, there is a clear way to prove my claims false. Maybe someone should do a complete archive rebuild with -j1 and -j4 on a smp system and compare the amount of failures to get an overview how bad it would be. Ugh, wrong. A complete archive rebuild won't do anything if the change is done in debian/rules, and it is known to be a bad idea for any non-SMP-clean package already. We're talking about allowing the maintainers to enable their packages to make use of concurrency if they believe that: 1. their packages are SMP-safe 2. the amount of memory taken won't exceed like 128-256MB. Thus, my counter-proposal: Let's allow maintainers to use make -jX according to their common sense, requiring obeying an env variable to opt out. and let the maintainer pass that env variable down a -jX. How about this option: We write a tool concurency-helper that gets a set of requirements of the source and outputs a suitable concurency level for current build host. Requirements could include: --package pkg Let the tool know what we build. The tool can have overrides in place for packages in case special rules apply. --max-concurency X || --non-concurent Limit the concurency, possibly to 1, for sources that have problems with it. Although such sources probably just shouldn't support this. --ram-estimate X [Y] Give some indication of ram usage. If the host has too little ram the concurency will be tuned down to prevent swapping. A broad indication +- a factor of 2 is probably sufficient. The [Y] would be to indicate ram usage for 32bit and 64bit archs seperately. Given the pointer size ram usage can vary between them a lot. --more-concurent Indicate that there are lots of small files that greatly benefit from interleaving I/O with cpu time. Try to use more concurency than cpus. The tool would look at those indicators and the hosts resources in both ram and cpus and figure out a suitable concurency level for the package from there. What do you think? Of course this is a more complex approach, but I think it's an approach into the right direction. I would like to have settings for distcc to get added (like --use-distcc and --distcc-hosts) A good idea. My proposal was to keep it simple, at the cost of being too cowardly when distcc is in use. If the helper is stored in a common place, this would also handle duplication of code. Rationale: Nearly every buildd and nearly every user building the packages on his own will benefit from -j2 [2], even on non-SMP. Unless it's a piece of heavily-templated code, any modern box
Re: make -j in Debian packages
On Fri, Jun 30, 2006 at 12:12:10PM +0200, Adam Borowski wrote:
Oh, so you mean checking the _free_ RAM instead of the _physical_ RAM?
This would be reasonable -- I didn't use this in the debian/rules
snippet I proposed as the physical memory is a trivially discernable
number while free RAM can be sometimes hard to tell, especially on Xen:
[...]
Of course, when I say that something is tricky, it doesn't mean that
someone with more clue than me can't do it.
[...]
Just grep it out of /proc/meminfo? Something like this ugly bit of
bourne shell:
echo $(
grep ^MemTotal: /proc/meminfo | awk '{ print $2 }'
) - $(
grep ^MemFree: /proc/meminfo | awk '{ print $2 }'
) - $(
grep ^Buffers: /proc/meminfo | awk '{ print $2 }'
) - $(
grep ^Cached: /proc/meminfo | awk '{ print $2 }'
) | bc
Which would give you a count of the total kb minus what's free and
in use for buffers/cache, similar to what free(1) spits out, but
probably not identical. That might not be an option on older kernels
(I seem to remember dealing with some /proc/meminfo change a while
back that broke check_swap in nagios-plugins), or at least might
differ at particular points in the kernel's release history.
--
{ IRL(Jeremy_Stanley); PGP(9E8DFF2E4F5995F8FEADDC5829ABF7441FB84657);
SMTP([EMAIL PROTECTED]); IRC([EMAIL PROTECTED]); ICQ(114362511);
AIM(dreadazathoth); YAHOO(crawlingchaoslabs); FINGER([EMAIL PROTECTED]);
MUD([EMAIL PROTECTED]:6669); WWW(http://fungi.yuggoth.org/); }
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
Adam Borowski [EMAIL PROTECTED] writes: On Wed, Jun 28, 2006 at 03:17:27AM +0200, Henning Makholm wrote: If package maintainer wants to build it faster on their own machine, I would imagine that checking for an environment variable (DEB_MAKE_OPTS or something, perhaps?) and using that would be the way to go. By default, build with a single processor. This would affect every single package, and you can't do that. While a vast majority of C code will build correctly, not every package is SMP-safe. [1] If I understand the problem correctly, it is not even necessary to modify debian/rules to get this behavior. If the interdependencies are properly declared, $ debian/rules -j42 binary should do the trick, as GNU make is smart enough to pass the option down to sub-makes that it starts. Actually, this is a bad idea; debian/rules are specifically the kind of makefiles that typically rely on the order in which dependencies are built. This is a bug as it breaks make -k, but as -k hardly ever makes sense with regard to debian/rules, this is nearly totally untested. What relies on the order and why should it? Every target in debian/rules should have the correct dependencies listed and force make to build targets in the order required. That means binary* depends on install*, install on build, build on configure or whatever combination of targets are employed. I think it is sane to demand that all debian/rules files are concurency save and it is trivial to do so. The same can't be said for upstream makefiles though. Many sources don't build with -j option. I'm not sure if debian/rules should somehow enforce -j1 in those cases or if only packages that benefit from -jX should add support for some DEB_BUILD_OPTION or CONCURENCY_LEVEL env var. The later would always be the save option. The former would have lots of hidden bugs that prop up suddenly on rebuilds or security fixes. Maybe someone should do a complete archive rebuild with -j1 and -j4 on a smp system and compare the amount of failures to get an overview how bad it would be. On the other hand, making builds significantly faster is not something that you can shake a stick at. Typically make -jX is faster even on uniprocessor, and I don't need to tell you why it's much faster on SMP. Too bad, a C++ build where every file takes 1GB memory obviously should not be parallelized. Also, no one but the maintainer knows whether a package is SMP-clean or not. You cannot guess this in an automated way. That would point to using an env varibale Thus, my counter-proposal: Let's allow maintainers to use make -jX according to their common sense, requiring obeying an env variable to opt out. and let the maintainer pass that env variable down a -jX. How about this option: We write a tool concurency-helper that gets a set of requirements of the source and outputs a suitable concurency level for current build host. Requirements could include: --package pkg Let the tool know what we build. The tool can have overrides in place for packages in case special rules apply. --max-concurency X || --non-concurent Limit the concurency, possibly to 1, for sources that have problems with it. Although such sources probably just shouldn't support this. --ram-estimate X [Y] Give some indication of ram usage. If the host has too little ram the concurency will be tuned down to prevent swapping. A broad indication +- a factor of 2 is probably sufficient. The [Y] would be to indicate ram usage for 32bit and 64bit archs seperately. Given the pointer size ram usage can vary between them a lot. --more-concurent Indicate that there are lots of small files that greatly benefit from interleaving I/O with cpu time. Try to use more concurency than cpus. The tool would look at those indicators and the hosts resources in both ram and cpus and figure out a suitable concurency level for the package from there. What do you think? Rationale: Nearly every buildd and nearly every user building the packages on his own will benefit from -j2 [2], even on non-SMP. Unless it's a piece of heavily-templated code, any modern box will have enough memory to handle it. The maintainer know whether the code is heavily templated or not. Mips, mipsel, arm and m68k won't benefit. The ram requirement just leads to poor cache performance or even excessive swapping in general. Sources and gcc are growing and growing and the ram of the buildds stays the same. On the other hand any modern system will build the source fast and the buildd will be idle most of the time so -j2 or not hardly matters. Wouldn't that indicate a preference to -j1? MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
Adam Borowski [EMAIL PROTECTED] writes: On Wed, Jun 28, 2006 at 07:50:50PM +0300, Lars Wirzenius wrote: ke, 2006-06-28 kello 18:43 +0200, Adam Borowski kirjoitti: What do you think about going with Don Armstrong's suggestion ($CONCURRENCY_LEVEL), while handling the default (no env var) my way (decent memory = parallel, little memory = -j 1) instead of Ingo's (-j 1 unless explicitely set)? I'm in favor of Ingo's approach. I don't think it is a good idea to hardcode assumptions of what is sufficient memory size into rules files; that's the kind of thing that is best done centrally. Still, the buildd admin has no way to estimate how much a sub-process of a package is going to use, the maintainer has at least a rough idea. Since the maintainer's action is needed anyway, he can as well provide this estimate. And then the buildd admin can set CONCURRENCY_LEVEL to 1 to centrally disable any parallelism of this kind. One could patch make to use concurency when the ram permits. This would involves checking the current ram usage and forking more builds if there is enough as well as suspending submakes when swapping starts. But i guess that would greatly complicate make itself. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 03:17:27AM +0200, Henning Makholm wrote: Scripsit Lars Wirzenius [EMAIL PROTECTED] If package maintainer wants to build it faster on their own machine, I would imagine that checking for an environment variable (DEB_MAKE_OPTS or something, perhaps?) and using that would be the way to go. By default, build with a single processor. If I understand the problem correctly, it is not even necessary to modify debian/rules to get this behavior. If the interdependencies are properly declared, $ debian/rules -j42 binary should do the trick, as GNU make is smart enough to pass the option down to sub-makes that it starts. Yes and no. * It only works if debian/rules calls make as `$(MAKE)'; not if it calls it as `make' or even `/usr/bin/make'. In the latter case, it just considers it some random program (rather than a sub-make) and will not pass the flags it received down to it. * It only works if the build system of the package does indeed use GNU make, and not something else, like pmake, smake, or SCons. -- Fun will now commence -- Seven Of Nine, Ashes to Ashes, stardate 53679.4 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 03:17:27AM +0200, Henning Makholm wrote:
If package maintainer wants to build it faster on their own machine, I
would imagine that checking for an environment variable (DEB_MAKE_OPTS
or something, perhaps?) and using that would be the way to go. By
default, build with a single processor.
This would affect every single package, and you can't do that. While
a vast majority of C code will build correctly, not every package is
SMP-safe. [1]
If I understand the problem correctly, it is not even necessary to
modify debian/rules to get this behavior. If the interdependencies
are properly declared,
$ debian/rules -j42 binary
should do the trick, as GNU make is smart enough to pass the option
down to sub-makes that it starts.
Actually, this is a bad idea; debian/rules are specifically the kind
of makefiles that typically rely on the order in which dependencies
are built. This is a bug as it breaks make -k, but as -k hardly ever
makes sense with regard to debian/rules, this is nearly totally
untested.
On the other hand, making builds significantly faster is not
something that you can shake a stick at. Typically make -jX is faster
even on uniprocessor, and I don't need to tell you why it's much
faster on SMP.
Too bad, a C++ build where every file takes 1GB memory obviously
should not be parallelized. Also, no one but the maintainer knows
whether a package is SMP-clean or not. You cannot guess this in an
automated way.
Thus, my counter-proposal:
Let's allow maintainers to use make -jX according to their common
sense, requiring obeying an env variable to opt out.
Rationale:
Nearly every buildd and nearly every user building the packages on
his own will benefit from -j2 [2], even on non-SMP. Unless it's a
piece of heavily-templated code, any modern box will have enough
memory to handle it. The maintainer know whether the code is heavily
templated or not.
Giving the choice to someone who doesn't know the package (upstream
and/or the maintainer) will lead to _random_ FTBFSes.
Having DEB_MAKE_NON_PARALLEL [3] disable this behavior would make
m68k happy, while giving a reasonable default for everyone else.
And yeah, I'm guilty of using unconditional make -j4 in one of my
packages (not in the archive yet), too.
[1] A real-world example:
[ make_commands ]
#!/bin/sh
rm -f commands.h
rm -f load_commands.h
ARGS=_command(char *arg, struct session *ses)
while read FUNC
do
if [ -z $FUNC ]
then
continue
fi
case $FUNC in
\;*);;
\#*)echo $FUNC commands.h;echo $FUNC load_commands.h
;;
\**)FUNC=`echo $FUNC|sed 's/^\*//'`
echo extern struct session *$FUNC$ARGS; commands.h
echo add_command(c_commands, \$FUNC\,
(t_command)${FUNC}_command); load_commands.h
;;
*) echo extern void$FUNC$ARGS; commands.h
echo add_command(commands, \$FUNC\, ${FUNC}_command);
load_commands.h
;;
esac
done
==
This script produces two files, commands.h and load_commands.h, both
of which should be regenerated every time the input data changes.
A naive Makefile will just have both of these depend on an
invocation of make_commands. This will work correctly and be
unnoticed for years until someone gives -j to make.
Would you bet that every single of your upstreams knows to avoid such
subtle errors?
[2] I benchmarked this with an old gcc several years ago on an
uniprocessor, and -j4 (!) gave the best results. Today, with gcc-4.2
-j2 won.
[3] This name sucks. Give a better one if you care :p
--
1KB // Microsoft corollary to Hanlon's razor:
// Never attribute to stupidity what can be
// adequately explained by malice.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, 28 Jun 2006, Adam Borowski wrote: Let's allow maintainers to use make -jX according to their common sense, requiring obeying an env variable to opt out. Why not just have some ENV variable (CONCURRENCY_LEVEL?) which specifies the maximum -j; the package maintainer is free to choose any level equal to or below that. If CONCURRENCY_LEVEL is not set, the rules file must assume CONCURRENCY_LEVEL=1 (or not specify -j?); this will keep what should be the current status quo the same, and allow buildd's and other builders to set this variable to a reasonable level for their system. This has the disadvantage of not automatically using -j for every package and requiring maintainer buy in to see results... but presumably those packages where it actually makes a difference will adopt it just so maintainer builds don't take as long. Don Armstrong -- There is no mechanical problem so difficult that it cannot be solved by brute strength and ignorance. -- William's Law http://www.donarmstrong.com http://rzlab.ucr.edu -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 02:06:26AM -0700, Don Armstrong wrote: On Wed, 28 Jun 2006, Adam Borowski wrote: Let's allow maintainers to use make -jX according to their common sense, requiring obeying an env variable to opt out. Why not just have some ENV variable (CONCURRENCY_LEVEL?) which specifies the maximum -j; the package maintainer is free to choose any level equal to or below that. If CONCURRENCY_LEVEL is not set, the rules file must assume CONCURRENCY_LEVEL=1 (or not specify -j?); this will keep what should be the current status quo the same, and allow buildd's and other builders to set this variable to a reasonable level for their system. I would rather have it default to a reasonable value on an average uniprocessor box, according to the maintainer's common sense. Otherwise, nearly no one will have this variable set and thus the whole benefit will be lost. And cutting the build time by half or more is a GREAT thing to have. (or not specify -j?) no -j = 1 -j with no value = INFINITY This is certainly not what you want for a package build; unlimited -j is useful for other uses of make. -jX (or -j X, the space is BAD for an option with an optional argument) = X processes This has the disadvantage of not automatically using -j for every package and requiring maintainer buy in to see results... but presumably those packages where it actually makes a difference will adopt it just so maintainer builds don't take as long. Why would you restrict this just to maintainer builds? Speeding up buildds and user builds would be a worthy goal, too. An obscure env var hardly anyone knows about means that hardly anyone will be affected. The m68k buildd admins are here, listening, so they can add this line. Or, another approach: if CONCURRENCY_LEVEL is unset, debian/rules can check the amount of memory present and/or the number of CPUs and make a guess. -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 10:31:54AM +0200, Adam Borowski wrote: On the other hand, making builds significantly faster is not something that you can shake a stick at. Typically make -jX is faster even on uniprocessor, and I don't need to tell you why it's much faster on SMP. Well, make -jX is not everywhere faster on UPs. It depends on other factors as well. If you specify -j2 and the second make is causing lots of swapping, you won't gain much if anything at all. Too bad, a C++ build where every file takes 1GB memory obviously should not be parallelized. Also, no one but the maintainer knows whether a package is SMP-clean or not. You cannot guess this in an automated way. Correct. But you should extend the decision whether a package should be built in a parallel make to the machine admin as well. When both say it's ok to make use of -jX, then go for it. If one of these two have any objections, then leave out the -j option. Thus, my counter-proposal: Let's allow maintainers to use make -jX according to their common sense, requiring obeying an env variable to opt out. I would prefer an opt-in variable. Rationale: Nearly every buildd and nearly every user building the packages on his own will benefit from -j2 [2], even on non-SMP. Unless it's a piece of heavily-templated code, any modern box will have enough memory to handle it. The maintainer know whether the code is heavily templated or not. Giving the choice to someone who doesn't know the package (upstream and/or the maintainer) will lead to _random_ FTBFSes. Therefore a double opt-in would be nice (machine admin and package maintainer saying it's OK). Having DEB_MAKE_NON_PARALLEL [3] disable this behavior would make m68k happy, while giving a reasonable default for everyone else. I don't think it's good to define an opt-out variable (*_NON_PARALLEL). Think positive! So, it would be better to define DEB_MAKE_PARALLEL, but even better it would be to use something existing: CONCURRENCY_LEVEL as Don Amstrong suggests. If this variable exists (e.g. if [ -z CONCURRENCY_LEVEL ]) the maintainer can make use of -jX. Keep in mind that there are archs that don't have that much resources to build in parallel, and I'm not thinking of m68k. M68k has buildds with 512M RAM, but for example armeb which has only 32M. I would like to see a method to allow usage of other compile accelerators as well (distcc (for using crosscc, but testing locally), ccache, etc). -- Ciao...//Fon: 0381-2744150 Ingo \X/ SIP: [EMAIL PROTECTED] gpg pubkey: http://www.juergensmann.de/ij/public_key.asc -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 02:06:26AM -0700, Don Armstrong wrote: Why not just have some ENV variable (CONCURRENCY_LEVEL?) which specifies the maximum -j; the package maintainer is free to choose any level equal to or below that. [...] This has the disadvantage of not automatically using -j for every package and requiring maintainer buy in to see results... but presumably those packages where it actually makes a difference will adopt it just so maintainer builds don't take as long. Yes, makes sense to me. When CONCURRENCY_LEVEL already works in reality, we could adopt it and make it a standard env var. -- Ciao...//Fon: 0381-2744150 Ingo \X/ SIP: [EMAIL PROTECTED] gpg pubkey: http://www.juergensmann.de/ij/public_key.asc -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 12:38:51PM +0200, Ingo Juergensmann wrote: On Wed, Jun 28, 2006 at 10:31:54AM +0200, Adam Borowski wrote: On the other hand, making builds significantly faster is not something that you can shake a stick at. Typically make -jX is faster even on uniprocessor, and I don't need to tell you why it's much faster on SMP. Well, make -jX is not everywhere faster on UPs. It depends on other factors as well. If you specify -j2 and the second make is causing lots of swapping, you won't gain much if anything at all. Exactly, just like I said: it depends on the memory. The benefits on UP are small (~10%), but except for huge working sets, non-negative. And the maintainer knows if the package handles huge chunks at once or not. Nearly every buildd and nearly every user building the packages on his own will benefit from -j2 [2], even on non-SMP. Unless it's a piece of heavily-templated code, any modern box will have enough memory to handle it. The maintainers know whether the code is heavily templated or not. Giving the choice to someone who doesn't know the package (upstream and/or the maintainer) will lead to _random_ FTBFSes. Therefore a double opt-in would be nice (machine admin and package maintainer saying it's OK). Having DEB_MAKE_NON_PARALLEL disable this behavior would make m68k happy, while giving a reasonable default for everyone else. I don't think it's good to define an opt-out variable (*_NON_PARALLEL). Think positive! So, it would be better to define DEB_MAKE_PARALLEL, but even better it would be to use something existing: CONCURRENCY_LEVEL as Don Amstrong suggests. If this variable exists (e.g. if [ -z CONCURRENCY_LEVEL ]) the maintainer can make use of -jX. Indeed, using the existing name is better. But, did you really want to say -z? It's what I want, and you seem to argue against me :p However, it's very unlikely someone would set a random env var unless the person knows about it. Defaults are what a vast majority of users use, so I would go for sensible ones. Keep in mind that there are archs that don't have that much resources to build in parallel, and I'm not thinking of m68k. M68k has buildds with 512M RAM, but for example armeb which has only 32M. Good point, but such machines are RARE. I would rather go for either have the admin opt out or them, or, have the packages detect low-mem conditions and optimize their builds accordingly. I would like to see a method to allow usage of other compile accelerators as well (distcc (for using crosscc, but testing locally), ccache, etc). In theory, you would set CC, but not everything obeys it. I personally tend to mess with what gcc points to. Usually with more than one levels of indirection, too -- colorgcc is a nice thing for when the output goes to a terminal. -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 12:38:51PM +0200, Ingo Juergensmann wrote: I don't think it's good to define an opt-out variable (*_NON_PARALLEL). Think positive! So, it would be better to define DEB_MAKE_PARALLEL, but even better it would be to use something existing: CONCURRENCY_LEVEL as Don Amstrong suggests. If this variable exists (e.g. if [ -z CONCURRENCY_LEVEL ]) the maintainer can make use of -jX. Keep in mind that there are archs that don't have that much resources to build in parallel, and I'm not thinking of m68k. M68k has buildds with 512M RAM, but for example armeb which has only 32M. Ok, so let's try some actual code. I've used my usual playground: package kbtin on mentors.debian.net Take a look at the marked ###\\\### box: If CONCURRENCY_LEVEL is set, it will be used if it looks plausible. If it's not there, it will be set to 2 if the box has at least 128MB memory. If any problem happens, the build continues with make -j 1. How do you like this, Ingo? Cheers and schtuff, -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 01:37:37PM +0200, Adam Borowski wrote: Well, make -jX is not everywhere faster on UPs. It depends on other factors as well. If you specify -j2 and the second make is causing lots of swapping, you won't gain much if anything at all. Exactly, just like I said: it depends on the memory. Not only on memory. Take into calculation such values like context switch overhead, task scheduling overhead, etc. On SMP systems there's is even some gain in parallel processing, when you can nail certain process onto certain CPUs to prevent their migration across CPU bounds. But that's another story. The benefits on UP are small (~10%), but except for huge working sets, non-negative. And the maintainer knows if the package handles huge chunks at once or not. Right. The maintainer should know it. But when the benefits are so small, one can argue if it's worth the work to change the build system? ;) Therefore I think it's best to first go with opt-in before make opt-out the default, let's say when etch+2 is out. I don't think it's good to define an opt-out variable (*_NON_PARALLEL). Think positive! So, it would be better to define DEB_MAKE_PARALLEL, but even better it would be to use something existing: CONCURRENCY_LEVEL as Don Amstrong suggests. If this variable exists (e.g. if [ -z CONCURRENCY_LEVEL ]) the maintainer can make use of -jX. Indeed, using the existing name is better. But, did you really want to say -z? It's what I want, and you seem to argue against me :p *cough* missing ! or -n instead of -z of course ;) However, it's very unlikely someone would set a random env var unless the person knows about it. Defaults are what a vast majority of users use, so I would go for sensible ones. The most sensible value, that should work for almost every user would be: make without -j option set. That's the default now, isn't it? ;) Keep in mind that there are archs that don't have that much resources to build in parallel, and I'm not thinking of m68k. M68k has buildds with 512M RAM, but for example armeb which has only 32M. Good point, but such machines are RARE. I would rather go for either have the admin opt out or them, or, have the packages detect low-mem conditions and optimize their builds accordingly. I don't think that those machines are rare. OTOH, almost everything is rare when compared to x86 boxes. But let some porters/policy people decide whether to use opt-out or opt-in. I would like to see a method to allow usage of other compile accelerators as well (distcc (for using crosscc, but testing locally), ccache, etc). In theory, you would set CC, but not everything obeys it. I personally tend to mess with what gcc points to. Usually with more than one levels of indirection, too -- colorgcc is a nice thing for when the output goes to a terminal. Of course, but my point is: when we make a decision now about -j flag, we can have some thougts about such accelerators as well and implement both in one go instead of having two separate changes to the build system. -- Ciao...//Fon: 0381-2744150 Ingo \X/ SIP: [EMAIL PROTECTED] gpg pubkey: http://www.juergensmann.de/ij/public_key.asc -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 03:22:35PM +0200, Ingo Juergensmann wrote: On Wed, Jun 28, 2006 at 01:37:37PM +0200, Adam Borowski wrote: The benefits on UP are small (~10%), but except for huge working sets, non-negative. And the maintainer knows if the package handles huge chunks at once or not. Right. The maintainer should know it. But when the benefits are so small, one can argue if it's worth the work to change the build system? ;) Therefore I think it's best to first go with opt-in before make opt-out the default, let's say when etch+2 is out. What I mean is opt-in for the maintainer, opt-out for the builder. And the benefits are small only on UP. I would like to see a method to allow usage of other compile accelerators as well (distcc (for using crosscc, but testing locally), ccache, etc). In theory, you would set CC, but not everything obeys it. I personally tend to mess with what gcc points to. Usually with more than one levels of indirection, too -- colorgcc is a nice thing for when the output goes to a terminal. Of course, but my point is: when we make a decision now about -j flag, we can have some thougts about such accelerators as well and implement both in one go instead of having two separate changes to the build system. changes to the build system? Where? Note that in my proposal there are exactly no changes to the global build system, except for a flag that can be set to force a package to go down to -j 1. No legislation needed. And with the guessing code in my debian/rules (package kbtin on m.d.n, as per the sibling branch of the thread), even that flag is not needed. On a small box, the package will go down to -j 1 without any admin intervention. I'm not talking about etch+2, I'm talking about something that can go into the archive right now. Just opt-in by including in debian/rules the code of which v0.1 I uploaded to mentors. Whee? -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 12:01:31PM +0200, Adam Borowski wrote: On Wed, Jun 28, 2006 at 02:06:26AM -0700, Don Armstrong wrote: This has the disadvantage of not automatically using -j for every package and requiring maintainer buy in to see results... but presumably those packages where it actually makes a difference will adopt it just so maintainer builds don't take as long. Why would you restrict this just to maintainer builds? Speeding up buildds and user builds would be a worthy goal, too. Speeding up buildds through uninvited make -j is certainly _not_ a worty goal, simply because you can't do that properly. SMP buildd systems currently run multiple instances of buildd at the same time, rather than expecting a package to specify make -j itself. Having three packages that specify 'make -j 4' on a multiprocessor buildd host that _already_ runs six builds in parallel is going to be devastating. While parallel builds can indeed help speed up things even on uniprocessor systems, they do this at a price; they require a _lot_ more RAM than non-parallel builds. A uniprocessor parallel build will go faster for as long as your buffercaches can keep make, the compiler, and the most often used header files in memory; as soon as those need to be swapped out, your performance will go down dramatically. Since not all machines have the same amount of RAM, it is impossible to say in a generic way when this point is going to be. In any case, I have nothing against doing parallel builds by default (at least, not as long as it can be properly disabled and regulated), but do not claim that it's a good thing also for the buildd's, because you are sadly mistaken. -- Fun will now commence -- Seven Of Nine, Ashes to Ashes, stardate 53679.4 signature.asc Description: Digital signature
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 03:42:07PM +0200, Wouter Verhelst wrote: SMP buildd systems currently run multiple instances of buildd at the same time, rather than expecting a package to specify make -j itself. Having three packages that specify 'make -j 4' on a multiprocessor buildd host that _already_ runs six builds in parallel is going to be devastating. I admit, this is news to me. While parallel builds can indeed help speed up things even on uniprocessor systems, they do this at a price; they require a _lot_ more RAM than non-parallel builds. Modular C code can require an order of magnitude less memory than templated C++. What I propose is parallelizing only those packages which are not heavy on RAM, on the maintainer's discretion. If all packages had a similar memory profile, this would be an optimization as it would equalize buildd loads somewhat. A uniprocessor parallel build will go faster for as long as your buffercaches can keep make, the compiler, and the most often used header files in memory; as soon as those need to be swapped out, your performance will go down dramatically. Since not all machines have the same amount of RAM, it is impossible to say in a generic way when this point is going to be. Right, that is why the debian/rules snippet I suggested checks just the amount of RAM without even looking at how many CPUs are present. Indeed, it would fail to recognize several instances of buildd running at once, but this is such a rare case that it can be handled with an env variable we can agree upon. In any case, I have nothing against doing parallel builds by default (at least, not as long as it can be properly disabled and regulated), Cool! This is exactly what this thread is about. What do you think about going with Don Armstrong's suggestion ($CONCURRENCY_LEVEL), while handling the default (no env var) my way (decent memory = parallel, little memory = -j 1) instead of Ingo's (-j 1 unless explicitely set)? I think that the sample script I uploaded in kbtin would do the job; if it is not good enough, it can be improved. but do not claim that it's a good thing also for the buildd's, because you are sadly mistaken. Still, my way is already an improvement over gem: gem used unconditional -j4 and it happens to be a C++ package. (Well, well. Because this happened just days after I uploaded a version with unconditional -j4 to mentors, it's likely it's me who's the real culprit. Gem's maintainer could have looked at my package and copied -j4, the choice of 4 instead of 2 kind of suggests that.) Another idea is to use -l; it trims concurrency down to 1 whenever system load exceeds a given value. Anyway, it's possible to come to an agreement that will make everyone happy, and I believe we're close. Meep? -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
ke, 2006-06-28 kello 18:43 +0200, Adam Borowski kirjoitti: What do you think about going with Don Armstrong's suggestion ($CONCURRENCY_LEVEL), while handling the default (no env var) my way (decent memory = parallel, little memory = -j 1) instead of Ingo's (-j 1 unless explicitely set)? I'm in favor of Ingo's approach. I don't think it is a good idea to hardcode assumptions of what is sufficient memory size into rules files; that's the kind of thing that is best done centrally. (Also, any rules file snippet should be as simple as possible. Otherwise fixing it everywhere when a bug is found is going to be tedious.) -- Connection resented by peer -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, Jun 28, 2006 at 07:50:50PM +0300, Lars Wirzenius wrote: ke, 2006-06-28 kello 18:43 +0200, Adam Borowski kirjoitti: What do you think about going with Don Armstrong's suggestion ($CONCURRENCY_LEVEL), while handling the default (no env var) my way (decent memory = parallel, little memory = -j 1) instead of Ingo's (-j 1 unless explicitely set)? I'm in favor of Ingo's approach. I don't think it is a good idea to hardcode assumptions of what is sufficient memory size into rules files; that's the kind of thing that is best done centrally. Still, the buildd admin has no way to estimate how much a sub-process of a package is going to use, the maintainer has at least a rough idea. Since the maintainer's action is needed anyway, he can as well provide this estimate. And then the buildd admin can set CONCURRENCY_LEVEL to 1 to centrally disable any parallelism of this kind. (Also, any rules file snippet should be as simple as possible. Otherwise fixing it everywhere when a bug is found is going to be tedious.) Exactly, or even better, it could be first debugged until it makes everyone happy, then put into a common repository (debhelper?). This way, fixing a bug would require bothering just a single person. -- 1KB // Microsoft corollary to Hanlon's razor: // Never attribute to stupidity what can be // adequately explained by malice. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Wed, 28 Jun 2006, Adam Borowski wrote: On Wed, Jun 28, 2006 at 02:06:26AM -0700, Don Armstrong wrote: On Wed, 28 Jun 2006, Adam Borowski wrote: Let's allow maintainers to use make -jX according to their common sense, requiring obeying an env variable to opt out. Why not just have some ENV variable (CONCURRENCY_LEVEL?) which specifies the maximum -j; the package maintainer is free to choose any level equal to or below that. If CONCURRENCY_LEVEL is not set, the rules file must assume CONCURRENCY_LEVEL=1 (or not specify -j?); this will keep what should be the current status quo the same, and allow buildd's and other builders to set this variable to a reasonable level for their system. I would rather have it default to a reasonable value on an average uniprocessor box, according to the maintainer's common sense. Otherwise, nearly no one will have this variable set and thus the whole benefit will be lost. And cutting the build time by half or more is a GREAT thing to have. The only places setting the variable matters is on buildds and when people are building the package; I think the buildd people will be aware of it if it helps, and maintainers will have to know of it if they implement it. This is certainly not what you want for a package build; unlimited -j is useful for other uses of make. I was more thinking of cases where the upstream Makefile already specifies a -j if one isn't set. This has the disadvantage of not automatically using -j for every package and requiring maintainer buy in to see results... but presumably those packages where it actually makes a difference will adopt it just so maintainer builds don't take as long. Why would you restrict this just to maintainer builds? I don't mean to restrict it. Speeding up buildds and user builds would be a worthy goal, too. An obscure env var hardly anyone knows about means that hardly anyone will be affected. Because maintainers will be the one making changes to rules files to implement CONCURRENCY_LEVEL (or whatever is decided on), they'll be more likely to do so if there's a benefit for them. Don ARmstrong -- There is no such thing as social gambling. Either you are there to cut the other bloke's heart out and eat it--or you're a sucker. If you don't like this choice--don't gamble. -- Robert Heinlein _Time Enough For Love_ p250 http://www.donarmstrong.com http://rzlab.ucr.edu -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
Scripsit Lars Wirzenius [EMAIL PROTECTED] If package maintainer wants to build it faster on their own machine, I would imagine that checking for an environment variable (DEB_MAKE_OPTS or something, perhaps?) and using that would be the way to go. By default, build with a single processor. If I understand the problem correctly, it is not even necessary to modify debian/rules to get this behavior. If the interdependencies are properly declared, $ debian/rules -j42 binary should do the trick, as GNU make is smart enough to pass the option down to sub-makes that it starts. Then all one has to do is to create and submit a patch for dpkg-buildpackage that lets it specify this flag on invocation. Well, in fact also design a mechanism to share knowledge about which source packages may break if given a -j due to insufficiently specified dependencies. So perhaps using $(DEB_MAKE_J_OPTION) on the $(MAKE) all line in debian/rules is a better choice after all. (In any case there is no need to check specifically whether it exists before using it - if it doesn't, make will silently expand it to nothing). -- Henning MakholmAnd why should I talk slaves' and fools' talk? I don't want him to live for ever, and I know that he's not going to live for ever whether I want him to or not. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
Henning Makholm [EMAIL PROTECTED] wrote: Well, in fact also design a mechanism to share knowledge about which source packages may break if given a -j due to insufficiently specified dependencies. So perhaps using $(DEB_MAKE_J_OPTION) on the $(MAKE) all line in debian/rules is a better choice after all. I would like to suggest that we use $(CONCURRENCY_LEVEL) instead, because there is is prior debian art (kernel-package) that does this already. Thanks, Tyler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Sun, Jun 25, 2006 at 04:36:08PM +0200, Wouter Verhelst wrote: It has come to my attention that the gem package is currently built using 'make -j 4', to have four compiler processes running at the same time. This is a bit troublesome for the poor m68k buildd, which is now suffering under High Load And Constant Swapping (HLACS). DoS against the buildd? I was going to file a flaming bug of severity 'serious', quoting the relevant paragraph from Policy which forbids such behaviour, except it's not there. Well, at least I can't find it... There is none. But you may consider it as an attack against the infrastructure. Bastian -- There is an order of things in this universe. -- Apollo, Who Mourns for Adonais? stardate 3468.1 signature.asc Description: Digital signature
Re: make -j in Debian packages
su, 2006-06-25 kello 16:36 +0200, Wouter Verhelst kirjoitti: It has come to my attention that the gem package is currently built using 'make -j 4', to have four compiler processes running at the same time. This is a bit troublesome for the poor m68k buildd, which is now suffering under High Load And Constant Swapping (HLACS). As far as I can see, using make's -j option is only useful if you have multiple processors. Packages should not make such assumptions of the build environment. If package maintainer wants to build it faster on their own machine, I would imagine that checking for an environment variable (DEB_MAKE_OPTS or something, perhaps?) and using that would be the way to go. By default, build with a single processor. I doubt we need a policy change for this. At some point, we need to stop legislating and start assuming the package maintainers have common sense. -- Happiness is going NIH on your own stuff. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
Quoting Wouter Verhelst [EMAIL PROTECTED]: Since most packages currently do not do this, some of our infrastructure (in casu, buildd machines) assume this is not being done. Doing it anyway then might upset those machines -- not just on m68k; when there was talk of a 6-way SPARC buildd machine being set up, I understand that the plan there was to run multiple instances of the buildd software on that machine, rather than having parallel builds run[1]. Having five or six build processes all do 'make -j 4' might grind even the most powerful of machines to a halt. [overly nagging] When the talk about the hijacking of Bacula was up, the consensus was 'who cares about the m68k? If they can't keep up, get more machines'. I didn't like it then, and I don't like it any more now... And i don't even HAVE a m68k machine any more! -- spy Delta Force Kennedy cryptographic Marxist Serbian critical arrangements 747 quiche supercomputer SDI PLO Ortega AK-47 [See http://www.aclu.org/echelonwatch/index.html for more about this] [Or http://www.europarl.eu.int/tempcom/echelon/pdf/rapport_echelon_en.pdf] If neither of these works, try http://www.aclu.org and search for echelon. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Sun, Jun 25, 2006 at 05:07:16PM +0200, Turbo Fredriksson wrote: When the talk about the hijacking of Bacula was up, the consensus was 'who cares about the m68k? If they can't keep up, get more machines'. You can also get the same from the other arches if you prefer. Bastian -- There are certain things men must do to remain men. -- Kirk, The Ultimate Computer, stardate 4929.4 signature.asc Description: Digital signature
Re: make -j in Debian packages
[Lars Wirzenius] As far as I can see, using make's -j option is only useful if you have multiple processors. Packages should not make such assumptions of the build environment. Actually, I've seem speedup with -j2 on a single CPU machine. I suspect one process is compiling while the other fetches the source from disk. :) Friendly, -- Petter Reinholdtsen -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
Am Sonntag, den 25.06.2006, 18:11 +0300 schrieb Lars Wirzenius: I doubt we need a policy change for this. At some point, we need to stop legislating and start assuming the package maintainers have common sense. Agreed. However, it might be a good idea to have *one* canonical variable name for that; perhaps dev-ref is a better place for this than policy. Thomas -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Sun, 2006-06-25 at 16:56 +0200, Bastian Blank wrote: DoS against the buildd? There is none. But you may consider it as an attack against the infrastructure. You on the other hand, might consider that developers might not have the malicious intent you infer, but perhaps just made an honest mistake. Thijs signature.asc Description: This is a digitally signed message part
Re: make -j in Debian packages
Lars Wirzenius [EMAIL PROTECTED] wrote: It has come to my attention that the gem package is currently built using 'make -j 4', to have four compiler processes running at the same time. This is a bit troublesome for the poor m68k buildd, which is now suffering under High Load And Constant Swapping (HLACS). As far as I can see, using make's -j option is only useful if you have multiple processors. Packages should not make such assumptions of the build environment. If package maintainer wants to build it faster on their own machine, I would imagine that checking for an environment variable (DEB_MAKE_OPTS or something, perhaps?) and using that would be the way to go. By default, build with a single processor. kernel-package uses the CONCURRENCY_LEVEL envrionment variable for this. And if I do a CONCURRENCY_LEVEL=4 on my single-CPU system, it does actually go quite a bit faster. :) - Tyler -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Sun, Jun 25, 2006 at 06:51:31PM +0200, Petter Reinholdtsen wrote: [Lars Wirzenius] As far as I can see, using make's -j option is only useful if you have multiple processors. Packages should not make such assumptions of the build environment. Actually, I've seem speedup with -j2 on a single CPU machine. I suspect one process is compiling while the other fetches the source from disk. :) Yes, I've seen similar behaviour as well, not only with make/compiler. Overloading can speed up things for rendering processes as well, depending on the arch. But that's not the point here. Although gem was using 4 parallel makes, the buildd performed quite well, but this was just luck. If the source file would've been bigger in size, it would have been bad for the buildd. However, not all buildds have 128M or more memory. For example the armeb buildds only have 32M and Joey said something that the (or some) arm buildds are just having 64M w/o swap. So, finding a solution for this problem would be nice. When you have more core/CPUs you may want to make use of these, but it should be configurable by the buildd admin, if it's ok for a package to use multiple instances of the compiler. So, some sort of an environmental variable needs to be set, I think. The package could then check for this variable and when it is known to build without any problems using -jX, it can go right away doing so. Otherwise it should use -j1 (or no -j option at all). When a policy change is needed, I think it would be worthwhile to think of allowing crosscompiler builds as well. When there would be a way to use distcc on slower archs to crosscompile large packages, it would make many people happy, I think. ;) But of course the very first question is: is it wanted that there's something like that in the future or not? If yes, then let a group of porters, policy people and such decide how this can be achieved in the best way and let them make a proposal after some time. Porters are needed because they will have to deal with those issues when a build fails because of this. Policy people, because they need to write it down. DAK people, because they need to implement the needed changes, etc... -- Ciao...//Fon: 0381-2744150 Ingo \X/ SIP: [EMAIL PROTECTED] gpg pubkey: http://www.juergensmann.de/ij/public_key.asc -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
su, 2006-06-25 kello 10:41 -0700, Tyler MacDonald kirjoitti: kernel-package uses the CONCURRENCY_LEVEL envrionment variable for this. And if I do a CONCURRENCY_LEVEL=4 on my single-CPU system, it does actually go quite a bit faster. :) Sure, even on a single CPU -jX (X 1) can be faster, but it depends on various factors, such as available memory, and other load on the machine. Using -j is not something that should be on by default, but it would be *really* nice if it were easy to turn on, for any package. Same with other compilation options, as it happens. -- Yet another quit message no-one will ever comment on. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Sun, Jun 25, 2006, Lars Wirzenius wrote: Sure, even on a single CPU -jX (X 1) can be faster, but it depends on various factors, such as available memory, and other load on the machine. Using -j is not something that should be on by default, but it would be *really* nice if it were easy to turn on, for any package. Same with other compilation options, as it happens. Maybe through USE flags? (*ducks*) -- Sam. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Sun, Jun 25, 2006 at 06:11:24PM +0300, Lars Wirzenius wrote: su, 2006-06-25 kello 16:36 +0200, Wouter Verhelst kirjoitti: It has come to my attention that the gem package is currently built using 'make -j 4', to have four compiler processes running at the same time. This is a bit troublesome for the poor m68k buildd, which is now suffering under High Load And Constant Swapping (HLACS). [...] I doubt we need a policy change for this. At some point, we need to stop legislating and start assuming the package maintainers have common sense. It's not a question of legislating; it's more a question of picking a good option and writing the specification in policy. I would actually like to be able to specify 'make -jX' instead of just 'make' in a build. Currently, that's not possible. The absence of common sense in this particular example is what sparked this suggestion, it is not what I'm trying to avoid in the future (though that is, of course, an interesting byproduct -- Fun will now commence -- Seven Of Nine, Ashes to Ashes, stardate 53679.4 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: make -j in Debian packages
On Sun, Jun 25, 2006 at 09:07:40PM +0200, Wouter Verhelst wrote: It's not a question of legislating; it's more a question of picking a good option and writing the specification in policy. I fully agree with Wouter on this. Although the specification doesn't necessarily have to be in policy (it could be in dev-ref or the package tools documentation). But I don't think policy is neecssarily a bad place for it either. Beyond telling us what we can and cannot do, policy helps our users understand what they can and cannot expect. I think having a standardized option here to allow -j X to be used if the packaging supports it is an excellent idea. If someone wanted to write up an actual proposal and post it to the policy list, well, we could at least judge the proposal on its merits, and, if it were a good proposal, I would definitely support it. But I don't actually care enough to write a proposal myself. Unless you guys want to wait until I _happen_ to find enough free time :) -- Chris Waters | Pneumonoultra-osis is too long [EMAIL PROTECTED] | microscopicsilico-to fit into a single or [EMAIL PROTECTED] | volcaniconi- standalone haiku -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
