I have changed the IGLU mirroring to mirror Mandrake itself, as well as the updates it was mirroring before. I will be erasing Mandrake from beak as soon as these things are over.
Also, there has been talk about using hard links to save space (for merging files identical between distributions). I implemented it on beak (preliminary). It doesn't save that much. Most of the links are either between a distro and itself (for example - same file in two different locations), and a little more frequent - between the upstream sources of debian and gentoo. There were even a few cases where the patches both debian and gentoo added were the same.
All told, the links saved about 4GB, out of 125GB. I'm not sure it was worth the effort (though we were desperate, so I guess it was :-).
I'm also considering changing the mirror direction mechanism to be DNS based. The idea is that we all agree on a consistant relative path a mirror will have. For example, at the moment, both beak and iglu use /pub/mirrors/name. We then have:
"mandrake.mirror.hamakor.org.il", "debian.mirror.hamakor.org.il" etc. Moving a mirror between two servers then becomes a trivial matter. We can even implement the gentoo approach for multiple mirrors. For example, we can have redhat.actcom.mirror.hamakor.org.il, and redhat.netvision.hamakor.org.il, and have redhat.mirror.hamakor.org.il be a round robin for both.
Shachar
Shachar Shemesh wrote:
Ok, I have managed to free up ~4GB of space by hard linking the identical files from the gentoo and debian archives. This is NOT a solution - we freed up less than 4% of the disk that way. The process itself is painfully long (need to perform MD5 on all files of the mirror). We can probably free up an additional 3G by loopback mounting the Mandrake ISOs, but I'm currently thinking that we would be better off just moving them over to IGLU (~20G free there).
I have added Oleg and iglu-web to the CC list, and removed the irrelevant parties.
Shachar
Shachar Shemesh wrote:
Gilad Ben-Yossef wrote:
Cron Daemon wrote:beak:/var/log/apache# zcat mirror-access.log*.gz | grep -c 'debian/.*i386'
rsync: recv_generator: mkdir "app-games/mirrormagic": No space left on device (2)
stat app-games/mirrormagic : No such file or directory
rsync: recv_generator: mkdir "app-games/mirrormagic/files": No such file or directory (2)
stat app-games/mirrormagic/files : No such file or directory
write failed on app-games/excido/Manifest : No space left on device
rsync error: error in file IO (code 11) at receiver.c(243)
rsync: connection unexpectedly closed (1331019 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(150)
We just ran out of storage space on beak:
[EMAIL PROTECTED]:~$ df
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/md0 3421584 907416 2514168 27% /
/dev/sda2 995992 32856 963136 4% /var/chroot/noexec
/dev/md1 133527952 133525956 1996 100% /var/chroot/exec/pubfiles/var/storage
What can/should be deleted?
Gilad
125899
beak:/var/log/apache# zcat mirror-access.log*.gz | grep -c 'debian/.*alpha'
14871
beak:/var/log/apache# zcat mirror-access.log*.gz | grep -c 'debian/.*powerpc'
11124
beak:/var/log/apache# zcat mirror-access.log*.gz | grep -c 'debian/.*sparc'
12681
No clear cut winner in terms of preferred debian mirror to take down. I was hoping for a knockout in that respect. Damn!
beak:/var/log/apache# zcat mirror-access.log*.gz | grep -ci mandrake 22604
beak:/var/log/apache# zcat mirror-access.log*.gz | grep -ci gentoo 63842
If it matters:
beak:/var/log/apache# zcat mirror-access.log*.gz | grep -c /pub/mirrors/Mandrake-iso/
900
beak:/var/log/apache# zcat mirror-access.log*.gz | grep -c /pub/mirrors/Mandrake/
21590
(this last one is probably 100% meaningless, as ISOs require less downloads, of bigger volume).
There is probably some saving to be had by loopback mounting the Mandrake ISOs loopback. Suggestions?
Shachar
-- Shachar Shemesh Open Source integration consultant Home page & resume - http://www.shemesh.biz/
