Re: CD release announcement, take 2
[I sent this before, but it doesn't seem to be getting through.] Here's the next version of the CD release announcement to send to CD mirror admins. If I don't get feedback, I'll assume it is OK like that. I've just added include/exclude support to jigdo-mirror. PLEASE TEST the attached version; if it works, I'll make a new jigdo release which contains it. Changes since the first version: - Complete set is 96 CD images (11*8 + 8 source), not 88 - States that all images except 1_NONUS (84 images, 50 GB) will be available via rsync - Discusses excluding of certain files with jigdo-mirror/rsync/debcdmirror - Mentions the DNS server change from open to raff -- From: Richard Atterer [EMAIL PROTECTED] Reply-To: [EMAIL PROTECTED] Subject: Forthcoming release of CD images for Debian 3.0 rev0 (woody) This mail was sent to you because your address is stored in our database of Debian CD mirror servers and your server is listed on http://www.debian.org/CD/http-ftp/. Note that if you reply to this mail, your answer will by default be sent to [EMAIL PROTECTED], a public mailing list. Within the next weeks, Debian 3.0 woody will be released and new CD images will be made available. The exact release date is not known yet - subscribe to the debian-cd mailing list to be informed about it. Here is some information about the release of the new CD images. Required disc space grows to 50 or 57 GB The images of the current stable distribution (potato) need 17 GB of disc space for 28 CD images. In contrast to this, the full set of woody CDs needs about 57 GB for 96 CD images! However, by default the 1_NONUS CDs will not be available for rsync-based mirroring (and thus also debcdmirror), which reduces the amount of space to 50 GB for 84 images. The increase is due to the larger number of CDs (8, used to be 4) per architecture, and the larger number of architectures (11, used to be 6). If you do not want to mirror the complete set of images, it is possible to restrict which images to mirror and which to omit - see below for instructions. Setup changes on your mirror You do not need to change anything about your current mirror setup if you want to distribute the set of 84 CD images which needs about 50 GB - the old debcdmirror scheme as well as rsync or FTP/HTTP mirroring will continue to work. However, consider changing the mirror setup as described below if one of the following applies: - You want to update your mirror quickly. In our experience, the master site will be under heavy load around release time, possibly even to the point of not being reachable. - You already have a local regular Debian FTP mirror. In this case, the mirroring can be made much more efficient now. - You do not want to offer the full set of 96 CD images. New way of mirroring: jigdo-mirror ~~ jigdo is a new way of generating Debian CD images. A local (=same machine) Debian FTP mirror is required for this. Additionally, if the mirror does not run Linux on Intel, you'll have to compile jigdo yourself - you need a recent C++ compiler (e.g. GCC 2.95) for this. The jigdo-mirror script to automate mirroring of Debian's CD images is new and needs more testing - if you can, please try it out now on the 3.0 pre-release images and report any success/failure to us! jigdo-mirror takes packages from the mirror as well as special files with .jigdo and .template extensions, and assembles the CD images from all this information. This makes it similar to how debcdmirror works, with the important difference that jigdo does not rely on rsync to produce the final image. A jigdo-based mirror requires - setting up a normal Debian FTP mirror http://www.debian.org/mirror/ - setting up HTTP mirroring of the .jigdo/.template files at http://cdimage.debian.org/jigdo-area/current/jigdo/ - setting up a cronjob which runs jigdo-mirror about once a day. Alternatively, you can also run jigdo-mirror manually whenever a new version of the CDs is released. - configuring jigdo-mirror. This should be easy, it hardly needs more information than the paths to the .jigdo/.template files and your Debian FTP mirror. Excluding certain CD images from being stored on your mirror The 96 images can be broken up as follows: For each of the 11 architectures (alpha, arm, hppa, i386, ia64, m68k, mips, mipsel, powerpc, s390, sparc), there are 8 CDs, CD 1-7 and CD 1_NONUS. Furthermore, there are 8 source CDs, again 7 CDs and one NONUS version of the first CD. Mirror sites located in the United States should not mirror the 1_NONUS CD, because it contains software which is not distributable in the US, e.g. because of patent issues. In contrast, mirrors outside the US may *only* want to mirror CD 1_NONUS, not CD 1
Re: CD release announcement
On Thu, May 30, 2002 at 02:35:58PM +0100, Philip Hands wrote: http://cdimage.debian.org/jigdo-area/current/ I just noticed: The way this is set up means that mirror admins who want to use jigdo-mirror should be instructed to HTTP-mirror http://cdimage.debian.org/jigdo-area/current/jigdo/, not http://cdimage.debian.org/jigdo-area/current/. I'll fix my announcement. BTW, this means that rsync-based mirrors will contain files like 3.0r0/i386/debian-3.0r0-i386-binary-1.iso whereas jigdo-mirrored ones just contain i386/debian-3.0r0-i386-binary-1.iso with no 3.0r0 directory. I can see no immediate problems with that, but it's probably something we should be aware of. Cheers, Richard -- __ _ |_) /| Richard Atterer | CS student at the Technische | GnuPG key: | \/¯| http://atterer.net | Universität München, Germany | 0x888354F7 ¯ '` ¯ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: CD release announcement
On Fri, 2002-05-31 at 04:00, Mattias Wadenstein wrote: Where will the jigdo files turn up? open is a bit closer, but I get good rates to raff too. Just wondering if I should have that cron job that mirrors jigdo-area point at cdimage or open. I'll be building the images on open, but the plan is to point the public at raff, and make sure that's mirrored ASAP after the images are on open. Cheers, Phil. -- Say no to software patents! http://petition.eurolinux.org/ |)| Philip Hands [+44 (0)20 8530 9560]http://www.hands.com/ |-| HANDS.COM Ltd.http://www.uk.debian.org/ |(| 10 Onslow Gardens, South Woodford, London E18 1NE ENGLAND signature.asc Description: This is a digitally signed message part
Re: CD release announcement
Hello, this is my first attempt at a note to mail to all CD mirror admins. Doubtlessly it is inaccurate in some places - please correct any mistakes. Is there any important issue which I missed, and which the mirror maintainers should know about? Where will the official 3.0r0 jigdo files be placed? I think it would be wise *not* to use cdimage/jigdo-area for that, because - ATM jigdo-mirror will attempt to re-generate any non-existent images every time it is run. IOW, if mirror admins mirror jigdo-area/ and then let jigdo-mirror loose on it, the script will work several minutes on each pre-release image, only to realize that it is unable to re-create it. - When 3.0r0 is out and Phil starts making 3.0r1 pre-releases, all mirrors will pick those up, attempt to create them, maybe overflow their disks etc... not desirable! Another related question: At the moment, jigdo-mirror does not allow you to filter which images it should create and which not. People using it will only be able to mirror all 88 images, there's no way to prevent it from creating e.g. binary-3 to binary-7, or the non-us CDs. Should I modify it to allow such filtering? -- Release of CD images for the Debian Linux 3.0r0 release (woody) ~ This mail was sent to you because your address is stored in our database of Debian CD mirror servers and your server is listed on http://www.debian.org/CD/http-ftp/. Note that if you reply to this mail, your answer will by default be sent to [EMAIL PROTECTED], a public mailing list. [Reply-To will be set up accordingly.] Within the next weeks, Debian 3.0 woody will be released and new CD images will be made available. Here is some information about the release of the new CD images. Size requirements ~ [Please correct numbers - they may be completely wrong!] The images of the current stable distribution (potato) need 13 GB of disc space for 22 CD images. In contrast to this, the full set of woody CDs needs about 53 GB for 88 CD images! [The 88 is 8*10 + 8 source.] This increase is due to the larger number of CDs (8, used to be 3) per architecture, and the larger number of architectures (10, used to be 6). Because we expect that most mirror admins do not want to dedicate so much space to CD images, by default not all CD images will be made available, only a subset which will take about 21 GB for 34 CD images. (We omit CDs 3-8 for the 9 non-Intel architectures.) Required setup changes on your mirror ~ You do not need to change anything about your current mirror setup if you want to distribute the default set of 34 CD images - the old debcdmirror scheme as well as rsync or FTP/HTTP mirroring will continue to work. [Is this correct? Will PIK-style .list files be generated? Will the rsync/http/ftp paths stay the same?] However, consider changing the mirror setup as described below if one of the following applies: - You want to update your mirror quickly after the release. In our experience, the master site will be under extremely heavy load immediately after the release, possibly even to the point of not being reachable. - You already have a local regular Debian FTP mirror. In this case, the mirroring can be made much more efficient now. - You want to offer the full set of 88 CD images. New way of mirroring: jigdo-mirror ~~ jigdo is a new way of generating Debian CD images. A local (=same machine) Debian FTP mirror is required for this. Additionally, if the mirror does not run Linux on Intel, you'll have to compile jigdo yourself - you need a recent C++ compiler (e.g. GCC 2.95) for this. The jigdo-mirror script to automate mirroring of Debian's CD images is new and needs more testing - if you can, please try it out now on the 3.0 pre-release images and report any success/failure to us! jigdo-mirror takes packages from the mirror as well as special files with .jigdo and .template extensions, and assembles the CD images from all this information. This makes it similar to how debcdmirror works, with the important difference that jigdo does not rely on rsync to produce the final image. A jigdo-based mirror requires - setting up a normal Debian FTP mirror http://www.debian.org/mirror/ - setting up HTTP mirroring of the .jigdo/.template files - setting up a cronjob which runs jigdo-mirror at regular intervals - configuring jigdo-mirror. This should be easy, it hardly needs more information than the paths to the .jigdo/.template files and your Debian FTP mirror. Links ~ Debian on CD: http://www.debian.org/CD/ Retrieving Debian CDs with jigdo: http://www.debian.org/CD/jigdo-cd/ rsync path for stable CD images: rsync://cdimage.debian.org/debian-cd/ (Try not to mirror directly from the master site if possible.) HTTP access is
Re: CD release announcement
On Thu, 30 May 2002, Richard Atterer wrote: Another related question: At the moment, jigdo-mirror does not allow you to filter which images it should create and which not. People using it will only be able to mirror all 88 images, there's no way to prevent it from creating e.g. binary-3 to binary-7, or the non-us CDs. Should I modify it to allow such filtering? Most definitely yes. Many site only mirror i386. The images of the current stable distribution (potato) need 13 GB of disc space for 22 CD images. In contrast to this, the full set of woody CDs needs about 53 GB for 88 CD images! [The 88 is 8*10 + 8 source.] Because we expect that most mirror admins do not want to dedicate so much space to CD images, by default not all CD images will be made available, only a subset which will take about 21 GB for 34 CD images. (We omit CDs 3-8 for the 9 non-Intel architectures.) Let people choose. Filtering. Unless Phil Hands wants to limit the number. [Is this correct? Will PIK-style .list files be generated? Will the rsync/http/ftp paths stay the same?] I hope so, particulaly rsync. There will be dud jigdo images. PIK and debian-cd can be used to produce starting images for use with rsync. Phil. -- Philip Charles; 39a Paterson Street, Abbotsford, Dunedin, New Zealand +64 3 488 2818Fax +64 3 488 2875Mobile 025 267 9420 [EMAIL PROTECTED] - preferred. [EMAIL PROTECTED] I sell GNU/Linux GNU/Hurd CDs. See http://www.copyleft.co.nz -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: CD release announcement
On Thu, 2002-05-30 at 13:16, Richard Atterer wrote: Where will the official 3.0r0 jigdo files be placed? I think it would be wise *not* to use cdimage/jigdo-area for that, because does this work if you point it at: http://cdimage.debian.org/jigdo-area/current/ ? If so, there's no problem (as long as I remember to update the symlink :-) Another related question: At the moment, jigdo-mirror does not allow you to filter which images it should create and which not. People using it will only be able to mirror all 88 images, there's no way to prevent it from creating e.g. binary-3 to binary-7, or the non-us CDs. Should I modify it to allow such filtering? Well, US sites need to be able to exclude the non-US images at the very least, but I don't suppose there are many of the potential users of this that actually want to carry all the images, so it would be good to be able to exclude various combinations of image. The other thing that is likely to happen (I should be able to sort it out soon) is that cdimage will actually be a CNAME for raff.d.o and non-us.cdimage.debian.org will point at open. raff is in the USA, so cannot carry the non-US CDs. It can however carry the non-US jigdo images, since they have no non-US data other than the package names and checksums in them. This may make the explanation a bit more complicated. Cheers, Phil. -- Say no to software patents! http://petition.eurolinux.org/ |)| Philip Hands [+44 (0)20 8530 9560]http://www.hands.com/ |-| HANDS.COM Ltd.http://www.uk.debian.org/ |(| 10 Onslow Gardens, South Woodford, London E18 1NE ENGLAND signature.asc Description: This is a digitally signed message part
Re: CD release announcement
On Thu 30 May, Richard Atterer wrote: Another related question: At the moment, jigdo-mirror does not allow you to filter which images it should create and which not. Should I modify it to allow such filtering? yes. It's pretty close to being essential I think. Because we expect that most mirror admins do not want to dedicate so much space to CD images, by default not all CD images will be made available, only a subset which will take about 21 GB for 34 CD images. (We omit CDs 3-8 for the 9 non-Intel architectures.) I'm not sure this is the right approach. Most people are interested in a particular arch either entirely or not at all, so I'd expect them to want all 8 arch-whatever CDs, rather than just the first two of a load of arches. (e.g. I want all 8 arm CDs and all 8 source but nothing else (well probably all i386 actually). I suppose the 'get the first 2 CDs then use apt' makes a lot of sense, but it's just as true for i386 as the other arches. All in all I think we need user-level filtering, on arch, non-US and CD number (with an easy '1st 2' and 'others' split making sense for CD numbers). Wookey -- Aleph One Ltd, Bottisham, CAMBRIDGE, CB5 9BA, UK Tel +44 (0) 1223 811679 work: http://www.aleph1.co.uk/ play: http://www.chaos.org.uk/~wookey/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: CD release announcement
OK, I'll implement the filtering. The part about not providing rsyncable CDs 3-7 for non-i386 was the last word from Phil H. about this, but that was before the decision to move almost everything to raff. So... will we be offering rsync access to all 88 images? In that case, a warning to the mirror admins about the size is definitely necessary. :) If this means that already existent mirror setups are going to fetch 53GB of data by default, there should also be instructions on how to limit the CDs by filename. With rsync, e.g. --include '*i386*' --exclude '*-[3-9].*' might be handy. BTW, are my size estimates correct? On Thu, May 30, 2002 at 02:35:58PM +0100, Philip Hands wrote: does this work if you point it at: http://cdimage.debian.org/jigdo-area/current/ ? If so, there's no problem (as long as I remember to update the symlink :-) Ah, sure, that should work just fine! Cheers, Richard -- __ _ |_) /| Richard Atterer | CS student at the Technische | GnuPG key: | \/¯| http://atterer.net | Universität München, Germany | 0x888354F7 ¯ '` ¯ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: CD release announcement
On 30 May 2002, Philip Hands wrote: On Thu, 2002-05-30 at 13:16, Richard Atterer wrote: Where will the official 3.0r0 jigdo files be placed? I think it would be wise *not* to use cdimage/jigdo-area for that, because does this work if you point it at: http://cdimage.debian.org/jigdo-area/current/ ? If so, there's no problem (as long as I remember to update the symlink :-) Just a small mention, I mirror jigdo-area daily now (excluding snapshot) on http://ftp.se.debian.org/mirror/debian-cd/jigdo-area/ - after the release I'll make the symlink (taking out mirror/ in that url) and stuff work so there isn't both a debian-iso and debian-cd with slightly different things (debian-iso is a pure mirror of the potato release images) Right now I made the decision that it is better to leave debian-iso as the mirror it is. The other thing that is likely to happen (I should be able to sort it out soon) is that cdimage will actually be a CNAME for raff.d.o and non-us.cdimage.debian.org will point at open. Where will the jigdo files turn up? open is a bit closer, but I get good rates to raff too. Just wondering if I should have that cron job that mirrors jigdo-area point at cdimage or open. I'll probably do alot of manual work at release time anyway, but it would be good to know where stuff will turn up. /Mattias Wadenstein -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: CD release announcement
On Mon, May 27, 2002 at 07:31:43PM +0200, Othmar Pasteka wrote: well, if you want mirror maintainers to be actually aware of jigdo, joy or whoever should mail the mirror maintainer list. Yes, that'd probably be a good idea! i think it's somewhat closed, or whatever, you probably have to speak to joy (josip rodin, i am so much used to irc nicknames ;) to post something there. That list isn't closed at all, I even have write access to it since it's in CVS! :-) Maybe you were thinking of the CD _vendors_ data, which is held in a database on some debian.org machine and which is not writable by mere mortals like me... I don't think a mailing list has been set up for this, we'll have to extract the email addresses and send the mail ourselves. Hm, ISTR Anne even wrote a script to do that. I'll write a short text tomorrow and post it here for everybody to comment. Cheers, Richard -- __ _ |_) /| Richard Atterer | CS student at the Technische | GnuPG key: | \/¯| http://atterer.net | Universität München, Germany | 0x888354F7 ¯ '` ¯ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
