Re: debian archive disk space requirements.
On Sun, Aug 31, 2003 at 04:27:57PM +0200, Goswin von Brederlow wrote: Marcelo E. Magallon [EMAIL PROTECTED] writes: [resorted by size] architecture |size --+ i386 | 3378532922 ia64 | 3394287226 And people allways say that 64 Bit archs need much bigger executables. And ia64 was the biggest on the list, despite having 600 fewer packages than i386 according to Marcelo's followup post. So what was your point Goswin? Hamish -- Hamish Moffatt VK3SB [EMAIL PROTECTED] [EMAIL PROTECTED]
Re: debian archive disk space requirements.
Marcelo E. Magallon [EMAIL PROTECTED] writes:
On Sun, Aug 31, 2003 at 12:56:19PM -0700, Joshua Kwan wrote:
[alpha]7283
[i386] 7889
I'd not include non-free in the batch because many non-free on i386 are
i386 only, i.e. binary driver installers.
Ah, well spotted, but still there's some significant difference:
[EMAIL PROTECTED]:~/ftp/dists$ zcat
testing{,-proposed-updates}/main/binary-alpha/Packages.gz | grep-dctrl -s
Package -F Architecture -v all | wc -l
7105
[EMAIL PROTECTED]:~/ftp/dists$ zcat
testing{,-proposed-updates}/main/binary-i386/Packages.gz | grep-dctrl -s
Package -F Architecture -v all | wc -l
7628
A lot of that is due to the gazillion kernel flavors and their modules,
some compilers, some sensors and sensor-related stuff, a few acpi
packages, old libc5 stuff, and the good old uh? stuff (firebird for
example).
How much space do those 500 odd packages take up?
Given a 50% sizce increase on binaries alpha should have another 1.8G
of debs. If those 500 packages make up 1.2G (+50%=1.8G) then the 50%
claim would be right.
Slightly less for less % increases, you do the math.
MfG
Goswin
Re: debian archive disk space requirements.
Hamish Moffatt [EMAIL PROTECTED] writes: On Sun, Aug 31, 2003 at 04:27:57PM +0200, Goswin von Brederlow wrote: Marcelo E. Magallon [EMAIL PROTECTED] writes: [resorted by size] architecture |size --+ i386 | 3378532922 ia64 | 3394287226 And people allways say that 64 Bit archs need much bigger executables. And ia64 was the biggest on the list, despite having 600 fewer packages than i386 according to Marcelo's followup post. So what was your point Goswin? Just saying that bit size has not much to do with code size. The opcodes available have more impact, or so it seems. MfG Goswin
Re: debian archive disk space requirements.
On Mon, Sep 01, 2003 at 12:19:36PM +0200, Goswin von Brederlow wrote: How much space do those 500 odd packages take up? Given a 50% sizce increase on binaries alpha should have another 1.8G of debs. If those 500 packages make up 1.2G (+50%=1.8G) then the 50% claim would be right. You're assuming that executables make up the bulk of most packages, and that compression rates for those executables are similar. I highly doubt both of those assumptions. Richard Braakman
Re: debian archive disk space requirements.
On Mon, Sep 01, 2003 at 12:19:36PM +0200, Goswin von Brederlow wrote:
How much space do those 500 odd packages take up?
I really have no intention to peg the database up with a query like
that. Here:
$ zcat testing{,-proposed-updates}/main/binary-alpha/Packages.gz |
grep-dctrl -n -s Package -F Architecture -v all |
sort ~/tmp/alpha
$ zcat testing{,-proposed-updates}/main/binary-i386/Packages.gz |
grep-dctrl -n -s Package -F Architecture -v all |
sort ~/tmp/i386
$ diff -u alpha i386 |
grep '^\+[^+]' |
cut -d + -f 2 |
while read p ; do
grep-dctrl -n -s Installed-Size -PX $p testing/main/binary-i386/Packages;
done |
perl -le '$t+=$_ foreach ; print $t' # golf anyone? it's an easy one
Have fun.
Given a 50% sizce increase on binaries alpha should have another 1.8G
of debs. If those 500 packages make up 1.2G (+50%=1.8G) then the 50%
claim would be right.
Like Richard said, that's hard to believe since not everything in a
package is a executable. Anyway:
$ dpkg --contents libc6_2.3.2-4_i386.deb | grep /lib/libc-2.3.2.so
-rwxr-xr-x root/root 1143024 2003-08-26 13:47:34 ./lib/libc-2.3.2.so
$ dpkg --contents libc6.1_2.3.2-4_alpha.deb | grep /lib/libc-2.3.2.so
-rwxr-xr-x root/root 1586408 2003-08-26 19:47:37 ./lib/libc-2.3.2.so
$ dpkg --contents libc6.1_2.3.2-4_ia64.deb | grep /lib/libc-2.3.2.so
-rwxr-xr-x root/root 2391480 2003-08-26 19:10:55 ./lib/libc-2.3.2.so
--
Marcelo
Re: debian archive disk space requirements.
* Matt Zimmerman ([EMAIL PROTECTED]) [030831 00:05]: On Sat, Aug 30, 2003 at 11:40:34AM +1000, jason andrade wrote: Is there any way to reduce the size of the archive over the next 4-6 weeks ? Drop potato? Or - allowing to act locally - put potato on an nfs-mounted volume (assuming that not so many people are accessing potato now). Cheers, Andi -- http://home.arcor.de/andreas-barth/ PGP 1024/89FB5CE5 DC F1 85 6D A6 45 9C 0F 3B BE F1 D0 C5 D1 D9 0C
Re: debian archive disk space requirements.
Marcelo E. Magallon [EMAIL PROTECTED] writes: On Sat, Aug 30, 2003 at 12:18:50PM +0200, Josip Rodin wrote: Pray to god that testing and unstable stop diverging so much? :) FYI, this is the size of all the binaries belonging to the given architecture, in the specifies suites: testing+testing-proposed-updates: [resorted by size] architecture |size --+ i386 | 3378532922 ia64 | 3394287226 all | 3113135590 alpha| 3050200486 powerpc | 2996108486 hppa | 2850080658 sparc| 2740423638 s390 | 2713124396 arm | 2667813072 m68k | 2571810596 mips | 2569942598 mipsel | 2531507858 And people allways say that 64 Bit archs need much bigger executables. Somehow alpha saves 328MB compared to i386 and a suggested 30-50% size increase on binaries would mean a lot of packages are not available on alpha, which isn't true. Nice statistics. MfG Goswin
Re: debian archive disk space requirements.
On Sun, Aug 31, 2003 at 04:27:57PM +0200, Goswin von Brederlow wrote: Marcelo E. Magallon [EMAIL PROTECTED] writes: On Sat, Aug 30, 2003 at 12:18:50PM +0200, Josip Rodin wrote: Pray to god that testing and unstable stop diverging so much? :) FYI, this is the size of all the binaries belonging to the given architecture, in the specifies suites: testing+testing-proposed-updates: [resorted by size] architecture |size --+ i386 | 3378532922 ia64 | 3394287226 all | 3113135590 alpha| 3050200486 powerpc | 2996108486 hppa | 2850080658 sparc| 2740423638 s390 | 2713124396 arm | 2667813072 m68k | 2571810596 mips | 2569942598 mipsel | 2531507858 And people allways say that 64 Bit archs need much bigger executables. Somehow alpha saves 328MB compared to i386 and a suggested 30-50% size increase on binaries would mean a lot of packages are not available on alpha, which isn't true. It probably achieves this through the cunning, space-saving strategy of rendering poorly written software un-buildable on 64-bit architectures. ;) -- Steve Langasek postmodern programmer pgphhS2my5DS9.pgp Description: PGP signature
Re: debian archive disk space requirements.
On Sun, Aug 31, 2003 at 04:27:57PM +0200, Goswin von Brederlow wrote:
And people allways say that 64 Bit archs need much bigger executables.
Somehow alpha saves 328MB compared to i386 and a suggested 30-50% size
increase on binaries would mean a lot of packages are not available on
alpha, which isn't true.
Interesting theory...
architecture |size| count
--++---
mipsel | 2531507858 | 7022
mips | 2569942598 | 7041
m68k | 2571810596 | 7267
arm | 2667813072 | 7272
s390 | 2713124396 | 7259
sparc| 2740423638 | 7292
hppa | 2850080658 | 7104
powerpc | 2996108486 | 7379
alpha| 3050200486 | 7283
i386 | 3378532922 | 7889
ia64 | 3394287226 | 7204
Funny that the count difference is so large... let me double check:
[EMAIL PROTECTED]:dists$ zcat
testing{,-proposed-updates}/{main,contrib,non-free}/binary-alpha/Packages.gz |
grep-dctrl -s Package -F Architecture -v all | wc -l
7283
[EMAIL PROTECTED]:dists$ zcat
testing{,-proposed-updates}/{main,contrib,non-free}/binary-i386/Packages.gz |
grep-dctrl -s Package -F Architecture -v all | wc -l
7889
Yup.
--
Marcelo
Re: debian archive disk space requirements.
On Sun, Aug 31, 2003 at 09:05:24PM +0200, Marcelo E. Magallon wrote:
Funny that the count difference is so large... let me double check:
[EMAIL PROTECTED]:dists$ zcat
testing{,-proposed-updates}/{main,contrib,non-free}/binary-alpha/Packages.gz
| grep-dctrl -s Package -F Architecture -v all | wc -l
7283
[EMAIL PROTECTED]:dists$ zcat
testing{,-proposed-updates}/{main,contrib,non-free}/binary-i386/Packages.gz |
grep-dctrl -s Package -F Architecture -v all | wc -l
7889
Yup.
I'd not include non-free in the batch because many non-free on i386 are
i386 only, i.e. binary driver installers.
What does it look like then?
--
Joshua Kwan
pgpyJn00Sb8qd.pgp
Description: PGP signature
Re: debian archive disk space requirements.
On Sun, Aug 31, 2003 at 12:56:19PM -0700, Joshua Kwan wrote:
[alpha]7283
[i386] 7889
I'd not include non-free in the batch because many non-free on i386 are
i386 only, i.e. binary driver installers.
Ah, well spotted, but still there's some significant difference:
[EMAIL PROTECTED]:~/ftp/dists$ zcat
testing{,-proposed-updates}/main/binary-alpha/Packages.gz | grep-dctrl -s
Package -F Architecture -v all | wc -l
7105
[EMAIL PROTECTED]:~/ftp/dists$ zcat
testing{,-proposed-updates}/main/binary-i386/Packages.gz | grep-dctrl -s
Package -F Architecture -v all | wc -l
7628
A lot of that is due to the gazillion kernel flavors and their modules,
some compilers, some sensors and sensor-related stuff, a few acpi
packages, old libc5 stuff, and the good old uh? stuff (firebird for
example).
--
Marcelo
Re: debian archive disk space requirements.
Goswin von Brederlow wrote: And people allways say that 64 Bit archs need much bigger executables. But the numbers quoted are for compressed executables. -- see shy jo pgpJrjjbpIoUt.pgp Description: PGP signature
Re: debian archive disk space requirements.
On Sat, Aug 30, 2003 at 11:40:34AM +1000, jason andrade wrote: Hi, I've noticed the growth in the debian archive with some concern over the last few weeks/months. We're now hitting the limit of the partition that debian is on and it will be quite difficult technically for us to deal with this in the short term. [snip] Is there any way to reduce the size of the archive over the next 4-6 weeks ? If you have info on what people dowload, you could delete things that people aren't fetching. apt falls-back to other archives if the first one fails, assuming the same package is listed in one of the pther archives in the sources.list. I'm not sure how difficult this would be. If you also regenerate the Package file then they will simply become invisible to users. This could get confusing. Hope this helps, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ All that is needed for the forces of evil to triumph is for enough good men to do nothing. - Edmond Burke The penalty good people pay for not being interested in politics is to be governed by people worse than themselves. - Plato pgppEwun2nGvL.pgp Description: PGP signature
Re: debian archive disk space requirements.
jason andrade [EMAIL PROTECTED] writes: Hi, I've noticed the growth in the debian archive with some concern over the last few weeks/months. We're now hitting the limit of the partition that debian is on and it will be quite difficult technically for us to deal with this in the short term. We have some new storage which is being comissioned but it won't be in place for about a month and the debian archive has already filled up a 100G partition we've dedicated to it here (that doesn't include the debian-cd or debian-non-US or other debian related archives.. just the main debian one). I'm stuck between a rock and a hard place here as we have a large dependency tree before we can move equipment over and get access to the new disk.. Is there any way to reduce the size of the archive over the next 4-6 weeks ? What architectures are you mirroring? You could drop some of the more uncommon arches (see access logs if any are completly unused) till you have space for them again. Preferably just drop the Packages file for the arch and delete as little as possible of the actual debs. MfG Goswin PS: What are you using to mirror?
Re: debian archive disk space requirements.
On Sat, Aug 30, 2003 at 09:29:16AM +0200, Goswin von Brederlow wrote: What architectures are you mirroring? You could drop some of the more uncommon arches No, he cannot, it's ftp.au.debian.org. All official mirrors have been, are or will be in a similar situation. -- 2. That which causes joy or happiness.
Re: debian archive disk space requirements.
On Sat, Aug 30, 2003 at 11:40:34AM +1000, jason andrade wrote: I've noticed the growth in the debian archive with some concern over the last few weeks/months. We're now hitting the limit of the partition that debian is on and it will be quite difficult technically for us to deal with this in the short term. I hear you. /dev/sda3 97G 97G 470M 100.0 [] /bla Is there any way to reduce the size of the archive over the next 4-6 weeks ? We are still waiting for Joey to officially announce the obsolescence of potato on -announce so that it can be moved to archive.debian.org. That will buy us six, seven gigabytes at most. Other than that, I've no idea. Pray to god that testing and unstable stop diverging so much? :) -- 2. That which causes joy or happiness.
Re: debian archive disk space requirements.
On Sat, 30 Aug 2003, Josip Rodin wrote: /dev/sda3 97G 97G 470M 100.0 [] /bla /dev/sdf1100798036 98652428 2145608 98% /raid/lun1p1 and that is by moving a chunk of debian archives into another disk and symlinking back in.. Is there any way to reduce the size of the archive over the next 4-6 weeks ? We are still waiting for Joey to officially announce the obsolescence of potato on -announce so that it can be moved to archive.debian.org. That will buy us six, seven gigabytes at most. Other than that, I've no idea. Pray to god that testing and unstable stop diverging so much? :) hmm ok. 6-7G would probably help for 4 weeks. once we move to the new disk i don't mind too much about debian growing again :-) /dev/sdb11220623064 32828 1208380596 1% /data1 i just need to get the time to be able to migrate stuff across.. and i really don't want to break ftp.au.debian.org for people.. regards, -jason
Re: debian archive disk space requirements.
On Sat, Aug 30, 2003 at 12:18:50PM +0200, Josip Rodin wrote: Is there any way to reduce the size of the archive over the next 4-6 weeks ? We are still waiting for Joey to officially announce the obsolescence of potato on -announce so that it can be moved to archive.debian.org. That will buy us six, seven gigabytes at most. Other than that, I've no idea. Pray to god that testing and unstable stop diverging so much? :) Hm, another possibility is to just make a new stable release, and then to delete all those woody packages. ;-) Cheers, Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯
Re: debian archive disk space requirements.
[I am not on debian-devel but reading the list through Usenet gateway] jason andrade [EMAIL PROTECTED] writes: /dev/sdf1100798036 98652428 2145608 98% /raid/lun1p1 and that is by moving a chunk of debian archives into another disk and symlinking back in.. You may also want to see mount(8) and look for bind from the manual page. Since there is not very much about it, I will just quote it below. From mount(8): Since Linux 2.4.0 it is possible to remount part of the file hierarchy somewhere else. The call is mount --bind olddir newdir ftp.fi.debian.org has the whole debian/ on the same partition but a couple of directories, such as debian-cd/, that were previously on the same partition with the mirror root are now mounted from a new location with the --bind option. /etc/fstab also works with bind option with something like: /newloc/debian-cd /oldloc/debian-cd none bind,noatime,noexec,nodev Using bind should keep all mirroring programs happy since they do not have to get confused when they find a symlink instead of directory. -- Heikki Vatiainen * [EMAIL PROTECTED] Tampere University of Technology * Tampere, Finland
Re: debian archive disk space requirements.
On Sat, Aug 30, 2003 at 12:18:50PM +0200, Josip Rodin wrote: Pray to god that testing and unstable stop diverging so much? :) FYI, this is the size of all the binaries belonging to the given architecture, in the specifies suites: architecture | any| unstable| u+t| u+t+s | u+t+s+o --++-+++ all | 6907585148 | 3400497424 | 4086646374 | 5709009750 | 686078 alpha| 8181555068 | 3407524450 | 4906783652 | 6562026256 | 7611929650 arm | 6594851918 | 2914482000 | 4192165600 | 5566376046 | 6233666136 hppa | 6447723766 | 3185688176 | 4554324284 | 5977389114 | 5977389114 hurd-i386| 554627760 | 554464016 | 554464016 | 554464016 | 554464016 i386 | 9027667988 | 3693410444 | 5352013018 | 7153730830 | 8206129322 ia64 | 7683523572 | 3714676258 | 5380423374 | 7121518964 | 7121518964 m68k | 6396308550 | 2809338886 | 4069205302 | 5393380388 | 6114386710 mips | 5416612358 | 2851405680 | 3920631098 | 5215452326 | 5215452326 mipsel | 5550145962 | 2819073082 | 3995118416 | 5262682024 | 5262682024 powerpc | 7680418788 | 3336303070 | 4780596742 | 6309202432 | 7112407388 s390 | 6147469798 | 3053291988 | 4427782760 | 5820464436 | 5820464436 sparc| 7331766068 | 3071979288 | 921552 | 5917654266 | 6790963196 any means any of: experimental oldstable old-proposed-updates proposed-updates unstable testing testing-proposed-updates stable stable+proposed-updates: architecture |size --+ all | 2195553250 alpha| 2438601088 arm | 1951980644 hppa | 260648 i386 | 2753622878 ia64 | 2531882184 m68k | 1888659560 mips | 1801959356 mipsel | 1763094946 powerpc | 2196413794 s390 | 1947506370 sparc| 2065059050 testing+testing-proposed-updates: architecture |size --+ all | 3113135590 alpha| 3050200486 arm | 2667813072 hppa | 2850080658 i386 | 3378532922 ia64 | 3394287226 m68k | 2571810596 mips | 2569942598 mipsel | 2531507858 powerpc | 2996108486 s390 | 2713124396 sparc| 2740423638 From the numbers quoted above you can make a reasonable estimation of how much unstable and testing overlap. You can assume testing-proposed-updates is empty (you'll be off only by a few MB). My napkin says 50%, which IMO is hideously bad, but that's me. Drop a factor of 2-3 somewhere if you want to consider source. This data does not come from the filesystem, but from the database used to wrangle the whole mess. Account for 10% wasted space or so for a guess of how much space you actually need for a mirror (adapt to your filesystem of choice -- information which I don't really care about). -- Marcelo
Re: debian archive disk space requirements.
On Sat, Aug 30, 2003 at 08:25:36PM +0300, Heikki Vatiainen wrote: ftp.fi.debian.org has the whole debian/ on the same partition but a couple of directories, such as debian-cd/, that were previously on the same partition with the mirror root are now mounted from a new location with the --bind option. It's harder within debian/ because you have to hand-pick parts of pool/ to bind-mount... messy. -- 2. That which causes joy or happiness.
Re: debian archive disk space requirements.
On Sat, Aug 30, 2003 at 12:18:50PM +0200, Josip Rodin wrote: We are still waiting for Joey to officially announce the obsolescence of potato on -announce so that it can be moved to archive.debian.org. I've found out today that Joey doesn't feel there should be any more announcements. I've sent an e-mail so I guess ftpmasters just need to find the time to pull out the magic wand, move leftover source packages to pool update anything else that is necessary, and move old potato stuff out. -- 2. That which causes joy or happiness.
Re: debian archive disk space requirements.
On Sat, Aug 30, 2003 at 11:40:34AM +1000, jason andrade wrote: Is there any way to reduce the size of the archive over the next 4-6 weeks ? Drop potato? -- - mdz
Re: debian archive disk space requirements.
On Sun, 30 Aug 2003, Heikki Vatiainen wrote: [I am not on debian-devel but reading the list through Usenet gateway] i didn't realise debian-devel is gatewayed to a newsgroup.. interesting. You may also want to see mount(8) and look for bind from the manual page. Since there is not very much about it, I will just quote it below. From mount(8): [...] ftp.fi.debian.org has the whole debian/ on the same partition but a couple of directories, such as debian-cd/, that were previously on the same partition with the mirror root are now mounted from a new location with the --bind option. /etc/fstab also works with bind option with something like: /newloc/debian-cd /oldloc/debian-cd none bind,noatime,noexec,nodev Using bind should keep all mirroring programs happy since they do not have to get confused when they find a symlink instead of directory. i'm not entirely sure how this helps unless you mean we can use the above to start moving parts of the debian tree to different partitions and then remounting them inside the tree.. this would be a nice temporary solution except that we have a more complicated architecture as we have a back end fileserver where everything is mirrored and front end servers that actually provide the ftp/http/rsync services which read only NFS mount from the back end.. once you start doing loops and binds on the back end it tends to fall over with NFS structures :-/ thanks for the tip though. regards, -jason
Re: debian archive disk space requirements.
On Aug 30, Josip Rodin [EMAIL PROTECTED] wrote: It's harder within debian/ because you have to hand-pick parts of pool/ to bind-mount... messy. BTDT. It's even harder because the archive contain hard links between random directories. -- ciao, | Marco | [1565 id6qO1u0SBERA]
Re: debian archive disk space requirements.
El 30 Aug 2003 20:25:36 +0300 Heikki Vatiainen [EMAIL PROTECTED] escribió: Since Linux 2.4.0 it is possible to remount part of the file hierarchy somewhere else. The call is mount --bind olddir newdir with 2.6 we also got mount --move olddir newdir 8)
debian archive disk space requirements.
Hi, I've noticed the growth in the debian archive with some concern over the last few weeks/months. We're now hitting the limit of the partition that debian is on and it will be quite difficult technically for us to deal with this in the short term. We have some new storage which is being comissioned but it won't be in place for about a month and the debian archive has already filled up a 100G partition we've dedicated to it here (that doesn't include the debian-cd or debian-non-US or other debian related archives.. just the main debian one). I'm stuck between a rock and a hard place here as we have a large dependency tree before we can move equipment over and get access to the new disk.. Is there any way to reduce the size of the archive over the next 4-6 weeks ? regards, -jason
