Re: Accessibility of official jigdo files for Sarge
On Thu, Jul 31, 2003 at 09:26:52PM +0200, Richard Atterer wrote: Hm, is this feature also somehow available in the Debian zlib packages? jigdo-file uses zlib to compress the .template files... Hm, it just occurred to me that it might be possible for me to allow rsyncability for .template files even without a patched zlib! The gzip patch works by calculating a rolling checksum for the last n bytes of the _un_compressed data. Whenever the checksum reaches a magic value (0), the compress stream is flushed. This ensures that the flushing always takes place at the same positions, just after the same data patterns. With my knowledge about the .template files, I can do a similar (possibly even better) thing: The compressed part of the .template consists of chunks of image data which were not matched by any .deb in the Debian mirror. Flushing between writing those chunks will do the trick. To prevent this from affecting the compression ratio too much, it mustn't happen too often, but current versions of jigdo-file already flush the stream from time to time for other reasons, so it should be possible to just move the boundary of the flush in an rsync-friendly way. Cheers, Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Thu, Jul 31, 2003 at 09:26:52PM +0200, Richard Atterer wrote: Hm, is this feature also somehow available in the Debian zlib packages? jigdo-file uses zlib to compress the .template files... Hm, it just occurred to me that it might be possible for me to allow rsyncability for .template files even without a patched zlib! The gzip patch works by calculating a rolling checksum for the last n bytes of the _un_compressed data. Whenever the checksum reaches a magic value (0), the compress stream is flushed. This ensures that the flushing always takes place at the same positions, just after the same data patterns. With my knowledge about the .template files, I can do a similar (possibly even better) thing: The compressed part of the .template consists of chunks of image data which were not matched by any .deb in the Debian mirror. Flushing between writing those chunks will do the trick. To prevent this from affecting the compression ratio too much, it mustn't happen too often, but current versions of jigdo-file already flush the stream from time to time for other reasons, so it should be possible to just move the boundary of the flush in an rsync-friendly way. Cheers, Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯
Re: Accessibility of official jigdo files for Sarge
John Winters wrote: IMHO, rsync is not more efficient for mirroring .jigdo/.template files than http/ftp, because all the data is gzipped. Some more mirrors for the images on gluck would still be nice, though. So they are! Funny, I could have sworn the .template files used to be text. Presumably my fractured imagination. Further suggestion then - could an un-gzipped version be made available by rsync *only*. Output of gzip --help on sid: gzip 1.3.5 (2002-09-30) [...] --rsyncable Make rsync-friendly archive Maybe this feature could be of interest here? I haven't experimented with the effects myself yet though. //conny -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Thu, Jul 31, 2003 at 08:35:11PM +0200, Conny Brunnkvist wrote: --rsyncable Make rsync-friendly archive Maybe this feature could be of interest here? I haven't experimented with the effects myself yet though. Hm, is this feature also somehow available in the Debian zlib packages? jigdo-file uses zlib to compress the .template files... Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
John Winters wrote: IMHO, rsync is not more efficient for mirroring .jigdo/.template files than http/ftp, because all the data is gzipped. Some more mirrors for the images on gluck would still be nice, though. So they are! Funny, I could have sworn the .template files used to be text. Presumably my fractured imagination. Further suggestion then - could an un-gzipped version be made available by rsync *only*. Output of gzip --help on sid: gzip 1.3.5 (2002-09-30) [...] --rsyncable Make rsync-friendly archive Maybe this feature could be of interest here? I haven't experimented with the effects myself yet though. //conny
Re: Accessibility of official jigdo files for Sarge
On Thu, Jul 31, 2003 at 08:35:11PM +0200, Conny Brunnkvist wrote: --rsyncable Make rsync-friendly archive Maybe this feature could be of interest here? I haven't experimented with the effects myself yet though. Hm, is this feature also somehow available in the Debian zlib packages? jigdo-file uses zlib to compress the .template files... Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯
Re: Accessibility of official jigdo files for Sarge
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Friday 25 July 2003 11:09 am, Attila Nagy wrote: BTW, I try to offer more than only the i386 stuff, so now there are CDs and DVDs for all architectures in sarge and sid in jigdo format and ISOs for i386 CDs and DVDs. I hope somebody can benefit from these. I have been using your i386 sid dvd jigdo files template. So, yes there are useful, and thanks for all the work :) - -- Greg Madden Debian GNU/Linux -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/J1x0k7rtxKWZzGsRAu2FAKCwyWWOS6/SQIuJE4uvtn9YIWHiEgCgm66e 9/cEzmffGoHmAHScHafeltU= =mRPk -END PGP SIGNATURE-
Re: Accessibility of official jigdo files for Sarge
On Tue, Jul 29, 2003 at 10:00:16AM +0200, Attila Nagy wrote: For Debian mirrors, we have practically no use in features like compression, RCS, deltas and whatever else. To eliminate the initial delay in rsyncing, now that would be a useful thing. Did you try that, is that improved with cvsup? No, I didn't try it with Debian mirrors, but I did with FreeBSD. It contains much more files than Debian, because of the CVS repositories checked out. It starts the synchronization much earlier than rsync debian/ mirrors don't contain CVS stuff, and most probably never will. When you look at it, they are quite trivial, because just a handful of files are ever updated -- only new files are added. Hence, results with updating, diffing files, they probably don't matter to us. We need an experiment with our stuff. What needs to be done to test this? and which is even better, the memory requirement is very low (maximum 4-6 MBs). I guess only the latter worth the change. We had many rsync mirrors and our machine ran out of memory very soon. Just imagine a mirror of four-five bigger projects (like NetBSD, FreeBSD, OpenBSD and Debian) and voila, you need minimum 1 GBs of RAM just for the rsync in memory file list. I didn't notice any noteworthy problems with a machine with 96 MB RAM, although I mass-mirror stuff less than half the size of debian/ (linux, gnu, php etc). -- 2. That which causes joy or happiness. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Tue, 29 Jul 2003, Josip Rodin wrote: On Tue, Jul 29, 2003 at 10:00:16AM +0200, Attila Nagy wrote: For Debian mirrors, we have practically no use in features like compression, RCS, deltas and whatever else. To eliminate the initial delay in rsyncing, now that would be a useful thing. Did you try that, is that improved with cvsup? No, I didn't try it with Debian mirrors, but I did with FreeBSD. It contains much more files than Debian, because of the CVS repositories checked out. It starts the synchronization much earlier than rsync debian/ mirrors don't contain CVS stuff, and most probably never will. When you look at it, they are quite trivial, because just a handful of files are ever updated -- only new files are added. Hence, results with updating, diffing files, they probably don't matter to us. We need an experiment with our stuff. What needs to be done to test this? 1. Run cvsupd on a machine. 2. Run cvsup to update a mirror from that first machine. Just look at top and so on during this time. :) The big expected difference is due to a local cache of filenames that is kept on the client side and threaded approach to start sending needed files as soon as they are discovered, before doing a full directory tree traversal and so on. and which is even better, the memory requirement is very low (maximum 4-6 MBs). I guess only the latter worth the change. We had many rsync mirrors and our machine ran out of memory very soon. Just imagine a mirror of four-five bigger projects (like NetBSD, FreeBSD, OpenBSD and Debian) and voila, you need minimum 1 GBs of RAM just for the rsync in memory file list. I didn't notice any noteworthy problems with a machine with 96 MB RAM, although I mass-mirror stuff less than half the size of debian/ (linux, gnu, php etc). The memory usage for the rsync process is about 40 megs for debian/ and increases with every file/dir/link/whatever added. /Mattias Wadenstein -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Friday 25 July 2003 11:09 am, Attila Nagy wrote: BTW, I try to offer more than only the i386 stuff, so now there are CDs and DVDs for all architectures in sarge and sid in jigdo format and ISOs for i386 CDs and DVDs. I hope somebody can benefit from these. I have been using your i386 sid dvd jigdo files template. So, yes there are useful, and thanks for all the work :) - -- Greg Madden Debian GNU/Linux -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/J1x0k7rtxKWZzGsRAu2FAKCwyWWOS6/SQIuJE4uvtn9YIWHiEgCgm66e 9/cEzmffGoHmAHScHafeltU= =mRPk -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
Josip Rodin wrote: For Debian mirrors, we have practically no use in features like compression, RCS, deltas and whatever else. To eliminate the initial delay in rsyncing, now that would be a useful thing. Did you try that, is that improved with cvsup? No, I didn't try it with Debian mirrors, but I did with FreeBSD. It contains much more files than Debian, because of the CVS repositories checked out. It starts the synchronization much earlier than rsync and which is even better, the memory requirement is very low (maximum 4-6 MBs). I guess only the latter worth the change. We had many rsync mirrors and our machine ran out of memory very soon. Just imagine a mirror of four-five bigger projects (like NetBSD, FreeBSD, OpenBSD and Debian) and voila, you need minimum 1 GBs of RAM just for the rsync in memory file list. -- Attila Nagy e-mail: [EMAIL PROTECTED] Free Software Network (FSN.HU) phone @work: +361 210 1415/127 ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758
Re: Accessibility of official jigdo files for Sarge
On Tue, Jul 29, 2003 at 10:00:16AM +0200, Attila Nagy wrote: For Debian mirrors, we have practically no use in features like compression, RCS, deltas and whatever else. To eliminate the initial delay in rsyncing, now that would be a useful thing. Did you try that, is that improved with cvsup? No, I didn't try it with Debian mirrors, but I did with FreeBSD. It contains much more files than Debian, because of the CVS repositories checked out. It starts the synchronization much earlier than rsync debian/ mirrors don't contain CVS stuff, and most probably never will. When you look at it, they are quite trivial, because just a handful of files are ever updated -- only new files are added. Hence, results with updating, diffing files, they probably don't matter to us. We need an experiment with our stuff. What needs to be done to test this? and which is even better, the memory requirement is very low (maximum 4-6 MBs). I guess only the latter worth the change. We had many rsync mirrors and our machine ran out of memory very soon. Just imagine a mirror of four-five bigger projects (like NetBSD, FreeBSD, OpenBSD and Debian) and voila, you need minimum 1 GBs of RAM just for the rsync in memory file list. I didn't notice any noteworthy problems with a machine with 96 MB RAM, although I mass-mirror stuff less than half the size of debian/ (linux, gnu, php etc). -- 2. That which causes joy or happiness.
Re: Accessibility of official jigdo files for Sarge
On Tue, 29 Jul 2003, Josip Rodin wrote: On Tue, Jul 29, 2003 at 10:00:16AM +0200, Attila Nagy wrote: For Debian mirrors, we have practically no use in features like compression, RCS, deltas and whatever else. To eliminate the initial delay in rsyncing, now that would be a useful thing. Did you try that, is that improved with cvsup? No, I didn't try it with Debian mirrors, but I did with FreeBSD. It contains much more files than Debian, because of the CVS repositories checked out. It starts the synchronization much earlier than rsync debian/ mirrors don't contain CVS stuff, and most probably never will. When you look at it, they are quite trivial, because just a handful of files are ever updated -- only new files are added. Hence, results with updating, diffing files, they probably don't matter to us. We need an experiment with our stuff. What needs to be done to test this? 1. Run cvsupd on a machine. 2. Run cvsup to update a mirror from that first machine. Just look at top and so on during this time. :) The big expected difference is due to a local cache of filenames that is kept on the client side and threaded approach to start sending needed files as soon as they are discovered, before doing a full directory tree traversal and so on. and which is even better, the memory requirement is very low (maximum 4-6 MBs). I guess only the latter worth the change. We had many rsync mirrors and our machine ran out of memory very soon. Just imagine a mirror of four-five bigger projects (like NetBSD, FreeBSD, OpenBSD and Debian) and voila, you need minimum 1 GBs of RAM just for the rsync in memory file list. I didn't notice any noteworthy problems with a machine with 96 MB RAM, although I mass-mirror stuff less than half the size of debian/ (linux, gnu, php etc). The memory usage for the rsync process is about 40 megs for debian/ and increases with every file/dir/link/whatever added. /Mattias Wadenstein
Re: Accessibility of official jigdo files for Sarge
Mattias Wadenstein wrote: Oh, well. There exists a debian package. So for some of our mirrors it shouldn't be much of a problem. But I guess that rsync will be the standard way for some more years. I didn't even know that cvsup exists in Debian. That's great. BTW, cvsup uses rsync for transferring binary files. The advantage is that it uses the rsync method only for standalone files, not for building the file list, so it has all advantages of rsync without its disadvantages. Thanks for the information! -- Attila Nagy e-mail: [EMAIL PROTECTED] Free Software Network (FSN.HU) phone @work: +361 210 1415/127 ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Fri, 2003-07-25 at 01:35, Richard Atterer wrote: On Thu, Jul 24, 2003 at 04:58:09PM +0100, John Winters wrote: I sell (amongst lots of other things) CDs of Debian Woody and testing snapshots of Sarge. Currently it appears that the only way one can download the official .jigdo files for unofficial Sarge CDs (!) is by means of http from gluck.debian.org. This is inefficient, particularly if one wants to keep doing it. Is there any way they could be made available by rsync? IMHO, rsync is not more efficient for mirroring .jigdo/.template files than http/ftp, because all the data is gzipped. Some more mirrors for the images on gluck would still be nice, though. So they are! Funny, I could have sworn the .template files used to be text. Presumably my fractured imagination. Further suggestion then - could an un-gzipped version be made available by rsync *only*. The changes from week to week can only be small and it would make sense to rsync the differences and re-build the files rather than re-download what can be a very big file. (Some of the later ones from fsn.hu have been 90M for just one (gzipped) .template file.) John -- The Linux Emporium - the source for Linux in the UK See http://www.linuxemporium.co.uk/ We had a woodhenge here once but it rotted. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Fri, Jul 25, 2003 at 07:49:48AM +0100, John Winters wrote: Further suggestion then - could an un-gzipped version be made available by rsync *only*. It should be possible to use jigdo-file's -0 switch to switch off compression. I /guess/ that would make the files rsyncable. The changes from week to week can only be small and it would make sense to rsync the differences and re-build the files rather than re-download what can be a very big file. (Some of the later ones from fsn.hu have been 90M for just one (gzipped) .template file.) I wonder why they're getting so big - if all files inside an .iso are found by jigdo-file, the resulting .template file should be 5MB. :-/ Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Fri, 2003-07-25 at 12:11, Attila Nagy wrote: Richard Atterer wrote: The changes from week to week can only be small and it would make sense to rsync the differences and re-build the files rather than re-download what can be a very big file. (Some of the later ones from fsn.hu have been 90M for just one (gzipped) .template file.) I wonder why they're getting so big - if all files inside an .iso are found by jigdo-file, the resulting .template file should be 5MB. :-/ The largest ones at ftp.fsn.hu are: 35M./sarge-dvd/jigdo/sarge-ia64-1.template 35M./sarge-dvd/jigdo/sarge-mips-1.template 35M./sarge-dvd/jigdo/sarge-mipsel-1.template 35M./sarge-dvd/jigdo/sarge-s390-1.template 35M./sid-dvd/jigdo/sid-i386-1.template 38M./sarge-dvd/jigdo/sarge-arm-1.template 38M./sarge-dvd/jigdo/sarge-m68k-1.template 38M./sarge-dvd/jigdo/sarge-powerpc-1.template 38M./sarge-dvd/jigdo/sarge-sparc-1.template 39M./sarge-dvd/jigdo/sarge-alpha-1.template 47M./sarge-dvd/jigdo/sarge-i386-1.template Which I guess makes sense, because the following files are left out from the .jigdos: Ah good. I'm glad they've come down again. I moved away from using the files from fsn.hu because a) they got so big and b) they couldn't be used to create .isos - they referenced files which weren't available on any mirror, not even fsn.hu. Perhaps something had gone wrong with the creation process there. (This was back in mid-April). John -- The Linux Emporium - the source for Linux in the UK See http://www.linuxemporium.co.uk/ We had a woodhenge here once but it rotted. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Fri, 25 Jul 2003, Attila Nagy wrote: Richard Atterer wrote: The changes from week to week can only be small and it would make sense to rsync the differences and re-build the files rather than re-download what can be a very big file. (Some of the later ones from fsn.hu have been 90M for just one (gzipped) .template file.) I wonder why they're getting so big - if all files inside an .iso are found by jigdo-file, the resulting .template file should be 5MB. :-/ The largest ones at ftp.fsn.hu are: 35M./sarge-dvd/jigdo/sarge-ia64-1.template 35M./sarge-dvd/jigdo/sarge-mips-1.template 35M./sarge-dvd/jigdo/sarge-mipsel-1.template 35M./sarge-dvd/jigdo/sarge-s390-1.template 35M./sid-dvd/jigdo/sid-i386-1.template 38M./sarge-dvd/jigdo/sarge-arm-1.template 38M./sarge-dvd/jigdo/sarge-m68k-1.template 38M./sarge-dvd/jigdo/sarge-powerpc-1.template 38M./sarge-dvd/jigdo/sarge-sparc-1.template 39M./sarge-dvd/jigdo/sarge-alpha-1.template 47M./sarge-dvd/jigdo/sarge-i386-1.template The jigdo directories went up to 10+ gigs and we had to stop mirroring for a while (the debian main archive is kind of more important to us). Is this problem fixed now? Which I guess makes sense, because the following files are left out from the .jigdos: egrep -v '/Contents|/Packages|/README|INDEX$|/Maintainers|/Release$|/debian-ke yring\.tar\.gz$|/ls-lR|//doc/[^/]+/?[^/]*\.(txt|html)$' \ Can the d-i images be found by jigdo or are they still downloaded from ~tfheen somewhere? /Mattias Wadenstein -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
John Winters wrote: Ah good. I'm glad they've come down again. I moved away from using the files from fsn.hu because a) they got so big and b) they couldn't be used to create .isos - they referenced files which weren't available on any mirror, not even fsn.hu. Perhaps something had gone wrong with the creation process there. (This was back in mid-April). I don't really know what happened, because I did not even touch the scripts. BTW, I try to offer more than only the i386 stuff, so now there are CDs and DVDs for all architectures in sarge and sid in jigdo format and ISOs for i386 CDs and DVDs. I hope somebody can benefit from these. -- Attila Nagy e-mail: [EMAIL PROTECTED] Free Software Network (FSN.HU) phone @work: +361 210 1415/127 ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
Mattias Wadenstein wrote:
The jigdo directories went up to 10+ gigs and we had to stop mirroring for
a while (the debian main archive is kind of more important to us). Is this
problem fixed now?
find . -name jigdo -exec du -hs {} \;
874M./sarge/jigdo
484M./sarge-dvd/jigdo
872M./sid/jigdo
375M./sid-dvd/jigdo
I guess, yes.
Can the d-i images be found by jigdo or are they still downloaded from
~tfheen somewhere?
I use a small script which tries to download the debian-installer images
from people.debian.org/~tfheen and if they cannot be found there, the
debian-cd script uses the previously downloaded ones.
Last time I checked, they were dated at 13 july, with the netboot image
being even older.
tfheen@ wrote that he is working on the issue, so I assume it will be
corrected when his time allows. (hopefully in the near future)
--
Attila Nagy e-mail: [EMAIL PROTECTED]
Free Software Network (FSN.HU) phone @work: +361 210 1415/127
ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
Mattias Wadenstein wrote: Thanks for the tip, I'll take a look at it. It still doesn't change the fact that rsync is the well-known tool by mirror admins though. The only problem is that cvsup is written in Modula-3, so it's not quite in perl or C :) http://www.cvsup.org/ -- Attila Nagy e-mail: [EMAIL PROTECTED] Free Software Network (FSN.HU) phone @work: +361 210 1415/127 ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758 -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Fri, 25 Jul 2003, Attila Nagy wrote: Mattias Wadenstein wrote: And mirrors would like to use rsync even if the rsync algorithm doesn't matter. It is a convenient and well-known tool for mirroring a directory structure, including all the metadata and stuff. If we wouldn't talk about Linux I would suggest forget rsync. It's very bad in handling many thousands of files, both on the client and the server side. We switched to cvsup where we could (for example FreeBSD mirror). It's a lot more efficient. A full FreeBSD mirror can be done in about 4 MBs of memory (about 600-800,000 files as far as I can remember). And also it is much faster in the start of the process. Ah, this seems like a good tool in that regard then. Not that it matters for cdimages and jigdos but it could be relevant for the man debian archive where the rsync file list building dominates sync time in some cases. Thanks for the tip, I'll take a look at it. It still doesn't change the fact that rsync is the well-known tool by mirror admins though. /Mattias Wadenstein -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Fri, 25 Jul 2003, Attila Nagy wrote: Mattias Wadenstein wrote: Thanks for the tip, I'll take a look at it. It still doesn't change the fact that rsync is the well-known tool by mirror admins though. The only problem is that cvsup is written in Modula-3, so it's not quite in perl or C :) http://www.cvsup.org/ Ah, seems like quite an unfriendly language from a sysadmins PoV with a very limited set of ports from what I could find. Oh, well. There exists a debian package. So for some of our mirrors it shouldn't be much of a problem. But I guess that rsync will be the standard way for some more years. /Mattias Wadenstein -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Fri, 2003-07-25 at 01:35, Richard Atterer wrote: On Thu, Jul 24, 2003 at 04:58:09PM +0100, John Winters wrote: I sell (amongst lots of other things) CDs of Debian Woody and testing snapshots of Sarge. Currently it appears that the only way one can download the official .jigdo files for unofficial Sarge CDs (!) is by means of http from gluck.debian.org. This is inefficient, particularly if one wants to keep doing it. Is there any way they could be made available by rsync? IMHO, rsync is not more efficient for mirroring .jigdo/.template files than http/ftp, because all the data is gzipped. Some more mirrors for the images on gluck would still be nice, though. So they are! Funny, I could have sworn the .template files used to be text. Presumably my fractured imagination. Further suggestion then - could an un-gzipped version be made available by rsync *only*. The changes from week to week can only be small and it would make sense to rsync the differences and re-build the files rather than re-download what can be a very big file. (Some of the later ones from fsn.hu have been 90M for just one (gzipped) .template file.) John -- The Linux Emporium - the source for Linux in the UK See http://www.linuxemporium.co.uk/ We had a woodhenge here once but it rotted.
Re: Accessibility of official jigdo files for Sarge
On Fri, Jul 25, 2003 at 07:49:48AM +0100, John Winters wrote: Further suggestion then - could an un-gzipped version be made available by rsync *only*. It should be possible to use jigdo-file's -0 switch to switch off compression. I /guess/ that would make the files rsyncable. The changes from week to week can only be small and it would make sense to rsync the differences and re-build the files rather than re-download what can be a very big file. (Some of the later ones from fsn.hu have been 90M for just one (gzipped) .template file.) I wonder why they're getting so big - if all files inside an .iso are found by jigdo-file, the resulting .template file should be 5MB. :-/ Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯
Re: Accessibility of official jigdo files for Sarge
Richard Atterer wrote: The changes from week to week can only be small and it would make sense to rsync the differences and re-build the files rather than re-download what can be a very big file. (Some of the later ones from fsn.hu have been 90M for just one (gzipped) .template file.) I wonder why they're getting so big - if all files inside an .iso are found by jigdo-file, the resulting .template file should be 5MB. :-/ The largest ones at ftp.fsn.hu are: 35M./sarge-dvd/jigdo/sarge-ia64-1.template 35M./sarge-dvd/jigdo/sarge-mips-1.template 35M./sarge-dvd/jigdo/sarge-mipsel-1.template 35M./sarge-dvd/jigdo/sarge-s390-1.template 35M./sid-dvd/jigdo/sid-i386-1.template 38M./sarge-dvd/jigdo/sarge-arm-1.template 38M./sarge-dvd/jigdo/sarge-m68k-1.template 38M./sarge-dvd/jigdo/sarge-powerpc-1.template 38M./sarge-dvd/jigdo/sarge-sparc-1.template 39M./sarge-dvd/jigdo/sarge-alpha-1.template 47M./sarge-dvd/jigdo/sarge-i386-1.template Which I guess makes sense, because the following files are left out from the .jigdos: egrep -v '/Contents|/Packages|/README|INDEX$|/Maintainers|/Release$|/debian-ke yring\.tar\.gz$|/ls-lR|//doc/[^/]+/?[^/]*\.(txt|html)$' \ -- Attila Nagy e-mail: [EMAIL PROTECTED] Free Software Network (FSN.HU) phone @work: +361 210 1415/127 ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758
Re: Accessibility of official jigdo files for Sarge
On Fri, 2003-07-25 at 12:11, Attila Nagy wrote: Richard Atterer wrote: The changes from week to week can only be small and it would make sense to rsync the differences and re-build the files rather than re-download what can be a very big file. (Some of the later ones from fsn.hu have been 90M for just one (gzipped) .template file.) I wonder why they're getting so big - if all files inside an .iso are found by jigdo-file, the resulting .template file should be 5MB. :-/ The largest ones at ftp.fsn.hu are: 35M./sarge-dvd/jigdo/sarge-ia64-1.template 35M./sarge-dvd/jigdo/sarge-mips-1.template 35M./sarge-dvd/jigdo/sarge-mipsel-1.template 35M./sarge-dvd/jigdo/sarge-s390-1.template 35M./sid-dvd/jigdo/sid-i386-1.template 38M./sarge-dvd/jigdo/sarge-arm-1.template 38M./sarge-dvd/jigdo/sarge-m68k-1.template 38M./sarge-dvd/jigdo/sarge-powerpc-1.template 38M./sarge-dvd/jigdo/sarge-sparc-1.template 39M./sarge-dvd/jigdo/sarge-alpha-1.template 47M./sarge-dvd/jigdo/sarge-i386-1.template Which I guess makes sense, because the following files are left out from the .jigdos: Ah good. I'm glad they've come down again. I moved away from using the files from fsn.hu because a) they got so big and b) they couldn't be used to create .isos - they referenced files which weren't available on any mirror, not even fsn.hu. Perhaps something had gone wrong with the creation process there. (This was back in mid-April). John -- The Linux Emporium - the source for Linux in the UK See http://www.linuxemporium.co.uk/ We had a woodhenge here once but it rotted.
Re: Accessibility of official jigdo files for Sarge
On Fri, 25 Jul 2003, Richard Atterer wrote: On Thu, Jul 24, 2003 at 04:58:09PM +0100, John Winters wrote: I sell (amongst lots of other things) CDs of Debian Woody and testing snapshots of Sarge. Currently it appears that the only way one can download the official .jigdo files for unofficial Sarge CDs (!) is by means of http from gluck.debian.org. This is inefficient, particularly if one wants to keep doing it. Is there any way they could be made available by rsync? IMHO, rsync is not more efficient for mirroring .jigdo/.template files than http/ftp, because all the data is gzipped. Some more mirrors for the images on gluck would still be nice, though. And mirrors would like to use rsync even if the rsync algorithm doesn't matter. It is a convenient and well-known tool for mirroring a directory structure, including all the metadata and stuff. /Mattias Wadenstein
Re: Accessibility of official jigdo files for Sarge
John Winters wrote: Ah good. I'm glad they've come down again. I moved away from using the files from fsn.hu because a) they got so big and b) they couldn't be used to create .isos - they referenced files which weren't available on any mirror, not even fsn.hu. Perhaps something had gone wrong with the creation process there. (This was back in mid-April). I don't really know what happened, because I did not even touch the scripts. BTW, I try to offer more than only the i386 stuff, so now there are CDs and DVDs for all architectures in sarge and sid in jigdo format and ISOs for i386 CDs and DVDs. I hope somebody can benefit from these. -- Attila Nagy e-mail: [EMAIL PROTECTED] Free Software Network (FSN.HU) phone @work: +361 210 1415/127 ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758
Re: Accessibility of official jigdo files for Sarge
Mattias Wadenstein wrote:
The jigdo directories went up to 10+ gigs and we had to stop mirroring for
a while (the debian main archive is kind of more important to us). Is this
problem fixed now?
find . -name jigdo -exec du -hs {} \;
874M./sarge/jigdo
484M./sarge-dvd/jigdo
872M./sid/jigdo
375M./sid-dvd/jigdo
I guess, yes.
Can the d-i images be found by jigdo or are they still downloaded from
~tfheen somewhere?
I use a small script which tries to download the debian-installer images
from people.debian.org/~tfheen and if they cannot be found there, the
debian-cd script uses the previously downloaded ones.
Last time I checked, they were dated at 13 july, with the netboot image
being even older.
tfheen@ wrote that he is working on the issue, so I assume it will be
corrected when his time allows. (hopefully in the near future)
--
Attila Nagy e-mail: [EMAIL PROTECTED]
Free Software Network (FSN.HU) phone @work: +361 210 1415/127
ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758
Re: Accessibility of official jigdo files for Sarge
Mattias Wadenstein wrote: And mirrors would like to use rsync even if the rsync algorithm doesn't matter. It is a convenient and well-known tool for mirroring a directory structure, including all the metadata and stuff. If we wouldn't talk about Linux I would suggest forget rsync. It's very bad in handling many thousands of files, both on the client and the server side. We switched to cvsup where we could (for example FreeBSD mirror). It's a lot more efficient. A full FreeBSD mirror can be done in about 4 MBs of memory (about 600-800,000 files as far as I can remember). And also it is much faster in the start of the process. -- Attila Nagy e-mail: [EMAIL PROTECTED] Free Software Network (FSN.HU) phone @work: +361 210 1415/127 ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758
Re: Accessibility of official jigdo files for Sarge
Mattias Wadenstein wrote: Thanks for the tip, I'll take a look at it. It still doesn't change the fact that rsync is the well-known tool by mirror admins though. The only problem is that cvsup is written in Modula-3, so it's not quite in perl or C :) http://www.cvsup.org/ -- Attila Nagy e-mail: [EMAIL PROTECTED] Free Software Network (FSN.HU) phone @work: +361 210 1415/127 ISOs: http://www.fsn.hu/?f=downloadcell.: +3630 306 6758
Re: Accessibility of official jigdo files for Sarge
On Fri, 25 Jul 2003, Attila Nagy wrote: Mattias Wadenstein wrote: And mirrors would like to use rsync even if the rsync algorithm doesn't matter. It is a convenient and well-known tool for mirroring a directory structure, including all the metadata and stuff. If we wouldn't talk about Linux I would suggest forget rsync. It's very bad in handling many thousands of files, both on the client and the server side. We switched to cvsup where we could (for example FreeBSD mirror). It's a lot more efficient. A full FreeBSD mirror can be done in about 4 MBs of memory (about 600-800,000 files as far as I can remember). And also it is much faster in the start of the process. Ah, this seems like a good tool in that regard then. Not that it matters for cdimages and jigdos but it could be relevant for the man debian archive where the rsync file list building dominates sync time in some cases. Thanks for the tip, I'll take a look at it. It still doesn't change the fact that rsync is the well-known tool by mirror admins though. /Mattias Wadenstein
Re: Accessibility of official jigdo files for Sarge
On Fri, 25 Jul 2003, Attila Nagy wrote: Mattias Wadenstein wrote: Thanks for the tip, I'll take a look at it. It still doesn't change the fact that rsync is the well-known tool by mirror admins though. The only problem is that cvsup is written in Modula-3, so it's not quite in perl or C :) http://www.cvsup.org/ Ah, seems like quite an unfriendly language from a sysadmins PoV with a very limited set of ports from what I could find. Oh, well. There exists a debian package. So for some of our mirrors it shouldn't be much of a problem. But I guess that rsync will be the standard way for some more years. /Mattias Wadenstein
Re: Accessibility of official jigdo files for Sarge
On Thu, 24 Jul 2003, John Winters wrote: Hi all, I sell (amongst lots of other things) CDs of Debian Woody and testing snapshots of Sarge. Currently it appears that the only way one can download the official .jigdo files for unofficial Sarge CDs (!) is by means of http from gluck.debian.org. This is inefficient, particularly if one wants to keep doing it. Is there any way they could be made available by rsync? This is in the plans for the new cdimage.debian.org so you could mirror from there. But then I too need an efficient method of getting them to that host. /Mattias Wadenstein -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Thu, Jul 24, 2003 at 04:58:09PM +0100, John Winters wrote: I sell (amongst lots of other things) CDs of Debian Woody and testing snapshots of Sarge. Currently it appears that the only way one can download the official .jigdo files for unofficial Sarge CDs (!) is by means of http from gluck.debian.org. This is inefficient, particularly if one wants to keep doing it. Is there any way they could be made available by rsync? IMHO, rsync is not more efficient for mirroring .jigdo/.template files than http/ftp, because all the data is gzipped. Some more mirrors for the images on gluck would still be nice, though. Cheers, Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Accessibility of official jigdo files for Sarge
On Thu, 24 Jul 2003, John Winters wrote: Hi all, I sell (amongst lots of other things) CDs of Debian Woody and testing snapshots of Sarge. Currently it appears that the only way one can download the official .jigdo files for unofficial Sarge CDs (!) is by means of http from gluck.debian.org. This is inefficient, particularly if one wants to keep doing it. Is there any way they could be made available by rsync? This is in the plans for the new cdimage.debian.org so you could mirror from there. But then I too need an efficient method of getting them to that host. /Mattias Wadenstein
Re: Accessibility of official jigdo files for Sarge
On Thu, Jul 24, 2003 at 04:58:09PM +0100, John Winters wrote: I sell (amongst lots of other things) CDs of Debian Woody and testing snapshots of Sarge. Currently it appears that the only way one can download the official .jigdo files for unofficial Sarge CDs (!) is by means of http from gluck.debian.org. This is inefficient, particularly if one wants to keep doing it. Is there any way they could be made available by rsync? IMHO, rsync is not more efficient for mirroring .jigdo/.template files than http/ftp, because all the data is gzipped. Some more mirrors for the images on gluck would still be nice, though. Cheers, Richard -- __ _ |_) /| Richard Atterer | GnuPG key: | \/¯| http://atterer.net | 0x888354F7 ¯ '` ¯
