Re: A sane guess at default Debian mirror for pbuilder
On Sun, May 27, 2007 at 09:50:23AM -0600, Wesley J. Landaker wrote: debootstrap: uses ftp.debian.org as default mirror. debconf questions aside, I think ftp.debian.org is a much saner *default* than ftp.jp.debian.org. I've always wondered where the later silly default came from. =) Well, at least it was different... ftp.debian.org is an even more horrible default, because that's burdering one single machine maxing out its FE card, where we have a network of 300 mirrors out there that are mostly happy to share the load. -- 2. That which causes joy or happiness. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: A sane guess at default Debian mirror for pbuilder
On Tue, May 29, 2007 at 07:48:59PM +0200, Josip Rodin wrote: Well, at least it was different... ftp.debian.org is an even more horrible default, because that's burdering one single machine maxing out its FE card, where we have a network of 300 mirrors out there that are mostly happy to share the load. I have to ask: why isn't ftp.debian.org a RR DNS entry? Or even better, wouldn't it be possible to setup a load balancer in front of ftp.debian.org (with a GE card) to share the load amongst more than one system? Regards Javier signature.asc Description: Digital signature
Re: A sane guess at default Debian mirror for pbuilder
On Wed, May 30, 2007 at 12:25:21AM +0200, Javier Fernández-Sanguino Pe?a wrote: Well, at least it was different... ftp.debian.org is an even more horrible default, because that's burdering one single machine maxing out its FE card, where we have a network of 300 mirrors out there that are mostly happy to share the load. I have to ask: why isn't ftp.debian.org a RR DNS entry? Or even better, wouldn't it be possible to setup a load balancer in front of ftp.debian.org (with a GE card) to share the load amongst more than one system? Last I heard from one of the DSAs in charge of that (I can't remember now which one of them), and that was just barely and years ago, it was because we couldn't get 1 machine to be sufficiently synchronized. I was never allowed to do major changes with the existing US mirrors in order to get something done about that. None of those things changed since then, it seems. -- 2. That which causes joy or happiness. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: A sane guess at default Debian mirror for pbuilder
On Sun, 27 May 2007 20:53:18 +0100 Matt Zimmerman [EMAIL PROTECTED] wrote: On Mon, May 28, 2007 at 12:25:50AM +0900, Junichi Uekawa wrote: After 6 years or so of setting ftp.jp.debian.org as default for pbuilder, I'm finally determined that it shouldn't stay like this. So I'd like to have some default guessing to happen. Preferably I don't want to ask via debconf, since users should have already answered the question at installation-time. I think the most accurate method would be to scan the list of configured apt sources, and choose the first one which matches the Debian release the user is currently running (via the information from Releases). If the necessary API isn't available yet, this could probably be added to python-apt without too much trouble, and might be useful elsewhere as well. It probably needs to be part of devscripts or build-essential (or a dependency of one of those so that it is always available) and personally, I would much prefer that this didn't rely on python. I don't want to have to add python to a debootstrap chroot just to get this functionality. Emdebian is busy removing perl from 'Essential' and is unlikely to support any python on all except the most powerful embedded devices. (Most devices will be C/C++ with a little dash.) -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ pgpIyTqLIFXCy.pgp Description: PGP signature
Re: A sane guess at default Debian mirror for pbuilder
On Sun, 2007-05-27 at 19:49 +0200, Frans Pop wrote: On Sunday 27 May 2007 19:43, Joey Hess wrote: d-i uses the following hack to figure out where to download udebs from when building installation media: Note that this can result in multiple sources. If you want only one, this hack would need to be refined. I can't help thinking that the code should never be reused. Even if its semantics are correct, the repeated re-parsing and pseudo-parsing with different tools is quite opaque. Ben. -- Ben Hutchings Larkinson's Law: All laws are basically false. signature.asc Description: This is a digitally signed message part
Re: A sane guess at default Debian mirror for pbuilder
Hi, After 6 years or so of setting ftp.jp.debian.org as default for pbuilder, I'm finally determined that it shouldn't stay like this. So I'd like to have some default guessing to happen. Preferably I don't want to ask via debconf, since users should have already answered the question at installation-time. I think the most accurate method would be to scan the list of configured apt sources, and choose the first one which matches the Debian release the user is currently running (via the information from Releases). If the necessary API isn't available yet, this could probably be added to python-apt without too much trouble, and might be useful elsewhere as well. I think this is very much useful as a starter. Considering that current apt output is not sufficient (apt-config does not seem to dump sources.list information, and apt-cache policy does not output enough information), it would be a good idea to have APIs for doing it. I am not sure if it should be done at python-apt layer or libapt-pkg, or adding an interface to apt-cache. regards, junichi -- [EMAIL PROTECTED],netfort.gr.jp} Debian Project -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: [Pbuilder-maint] A sane guess at default Debian mirror for pbuilder
Hi,
d-i uses the following hack to figure out where to download udebs from
when building installation media:
grep '^deb[ \t]' $(SYSTEM_SOURCES_LIST) \
|grep -v '^deb[ \t]cdrom:' \
|grep -v
'\(security.debian.org\|volatile.debian.\(net\|org\)\)' \
|grep '[ \t]main' \
|awk '{print $$1 $$2}' \
|sed s,/* *$$, $(SUITE) $(UDEB_COMPONENTS), \
|sed s,^deb file,deb copy, \
|perl -ne 'print unless $$seen{$$_}; $$seen{$$_}=1' ; \
This chunk of code looks volatile, and vulnerable to changes in mirror
structures, if I duplicated it to pbuilder. I'm now more inclined to
have some common code that can be shared from several applications,
and let apt parse sources.list, rather than parsing sources.list
individually.
regards,
junichi
--
[EMAIL PROTECTED],netfort.gr.jp} Debian Project
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
A sane guess at default Debian mirror for pbuilder
Hi, After 6 years or so of setting ftp.jp.debian.org as default for pbuilder, I'm finally determined that it shouldn't stay like this. So I'd like to have some default guessing to happen. Preferably I don't want to ask via debconf, since users should have already answered the question at installation-time. Looking for prior art I found the following: piuparts: Looks at first 'deb' line from /etc/apt/sources.list - Can't handle /etc/apt/sources.list.d - Assumes that the top entry is the best emdebian-tools: Using 'apt-cache policy' to obtain information - told on IRC that it loses port number info. - probably picks up security mirrors too, which can't really be used for running debootstrap. debootstrap: uses ftp.debian.org as default mirror. Probably difficult parts: handling CD-ROM installation mixed stable-backports / security / s-p-u / ubuntu / whatever installation. I'm inclined to add a default configuration which uses the first usable line starting with 'deb http://' from /etc/apt/sources.list, and maybe check with a debconf question. Any good ideas? regards, junichi -- [EMAIL PROTECTED],netfort.gr.jp} Debian Project -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: A sane guess at default Debian mirror for pbuilder
On Sunday 27 May 2007 09:25:50 Junichi Uekawa wrote: After 6 years or so of setting ftp.jp.debian.org as default for pbuilder, I'm finally determined that it shouldn't stay like this. So I'd like to have some default guessing to happen. Preferably I don't want to ask via debconf, since users should have already answered the question at installation-time. [...] debootstrap: uses ftp.debian.org as default mirror. debconf questions aside, I think ftp.debian.org is a much saner *default* than ftp.jp.debian.org. I've always wondered where the later silly default came from. =) -- Wesley J. Landaker [EMAIL PROTECTED] xmpp:[EMAIL PROTECTED] OpenPGP FP: 4135 2A3B 4726 ACC5 9094 0097 F0A9 8A4C 4CD6 E3D2 signature.asc Description: This is a digitally signed message part.
Re: A sane guess at default Debian mirror for pbuilder
On Sun, 27 May 2007 09:50:23 -0600
Wesley J. Landaker [EMAIL PROTECTED] wrote:
On Sunday 27 May 2007 09:25:50 Junichi Uekawa wrote:
After 6 years or so of setting ftp.jp.debian.org as default for
pbuilder, I'm finally determined that it shouldn't stay like this. So
I'd like to have some default guessing to happen. Preferably I don't
want to ask via debconf, since users should have already answered the
question at installation-time.
[...]
debootstrap:
uses ftp.debian.org as default mirror.
debconf questions aside, I think ftp.debian.org is a much saner *default*
than ftp.jp.debian.org. I've always wondered where the later silly default
came from. =)
From my point of view, I would much rather that the default was a
Primary Mirror - ftp.debian.org only supports the most popular
architectures. (I've worked around this in emdebian-tools and
empdebuild {only in SVN so far}) but a default should at least be
capable of supporting more than just a couple of the supported
architectures (IMHO).
BTW: that is why emdebian-tools doesn't really take much care over
things like port numbers, the principle aim of the mirror detection
code in emdebian-tools is just to identify if a primary mirror is
already in use - if not, one is added via a conffile that the user can
tweak later.
I think that would be a useful way for pbuilder too - check all
available sources via apt-cache policy and pick a primary (if it
exists) and let the user modify the conffile to specify a different one
if necessary.
--
Neil Williams
=
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/
pgp7W6C2tZIA9.pgp
Description: PGP signature
Re: A sane guess at default Debian mirror for pbuilder
On ma, 2007-05-28 at 00:25 +0900, Junichi Uekawa wrote: piuparts: Looks at first 'deb' line from /etc/apt/sources.list - Can't handle /etc/apt/sources.list.d - Assumes that the top entry is the best I didn't want to have piuparts use all sources.list entires, since they often have things like debian-multimedia in them. Since apt favors the first one if it can, picking just the first one seemed like a reasonable thing to do, as a guess. If the guess turns out to be wrong, then it's easy enough to override, but it seems to work often enough that it is helpful to guess. emdebian-tools: Using 'apt-cache policy' to obtain information - told on IRC that it loses port number info. - probably picks up security mirrors too, which can't really be used for running debootstrap. I should have to dig up documentation, but I can't see that apt-cache policy outputs enough info to re-create the sources.list lines. For example: 500 http://agnes lenny/main Packages release o=Unofficial Multimedia Packages,a=testing,l=Unofficial Multimedia Packages,c=main origin agnes 500 http://agnes lenny/main Packages release o=Debian,a=testing,l=Debian,c=main origin agnes Those two stanzas correspond to two different sources.list lines. I don't see enough information there to re-create them: deb http://agnes/debian lenny main contrib non-free deb http://agnes/debian-multimedia/ lenny main ) could be wrong, of course: there are many details of apt I do not know. debootstrap: uses ftp.debian.org as default mirror. If ftp.debian.org doesn't contain all architectures, then that's a bad default, imho. Probably difficult parts: handling CD-ROM installation I don't think running pbuilder against CD-ROMs is a workable solution in the modern world. There's way too many CDs involved. Anyone who really wants to do that can be expected to go through the small amount of pain of configuring pbuilder manually. mixed stable-backports / security / s-p-u / ubuntu / whatever installation. The host system should, of course, be able to run such a mix freely. The environment built by pbuilder, piuparts, or other tools, should not, unless explicitly configured to do so, because the risk of contaminating uploads to Debian is too great. I'm inclined to add a default configuration which uses the first usable line starting with 'deb http://' from /etc/apt/sources.list, and maybe check with a debconf question. I don't think using debconf is a good idea. Any answer you get is valid when the package is installed, but won't be valid later. The answer should be generated dynamically, each time pbuilder (or piuparts or whatever) needs it. I propose that it be encoded into a new command, apt-default-sources.list or something like that, which, when run, outputs a sources.list that tools like pbuilder and piuparts can use. Put the heuristics into that command, and then it doesn't need to be duplicated in every package that needs it. -- The most difficult thing in programming is to be simple and straightforward. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: A sane guess at default Debian mirror for pbuilder
Hi, debootstrap: uses ftp.debian.org as default mirror. debconf questions aside, I think ftp.debian.org is a much saner *default* than ftp.jp.debian.org. I've always wondered where the later silly default came from. =) Good Trivia question. The reasons for using ftp.jp.debian.org are, in random order: 1. ftp.debian.org is not a full mirror anymore, it doesn't have the lesser architectures. 2. ftp.jp.debian.org is slow for most people, but it does have the network bandwidth; which is not too bad as a default mirror. 3. It's just a default, people are expected to edit the configuration. 4. Historic reason: at one time in history, ftp.debian.org used to be unstable compared to ftp.jp.debian.org. Changing default mirror to something more reliable probably saved a few bugreports. (However, looking back at the repository log, it was ftp.jp.debian.org from day 2, so I don't really know if that's true) regards, junichi -- [EMAIL PROTECTED],netfort.gr.jp} Debian Project -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: A sane guess at default Debian mirror for pbuilder
On Sun, 27 May 2007 19:22:26 +0300 Lars Wirzenius [EMAIL PROTECTED] wrote: emdebian-tools: Using 'apt-cache policy' to obtain information - told on IRC that it loses port number info. - probably picks up security mirrors too, which can't really be used for running debootstrap. I should have to dig up documentation, but I can't see that apt-cache policy outputs enough info to re-create the sources.list lines. emdebian-tools doesn't try to recreate the sources.list lines - all that check is required to do is identify whether the apt-cache policy already includes a Primary Mirror. emdebian-tools then simply adds the Emdebian repository (/etc/apt/sources.list.d/emdebian.sources.list) and a Primary if a primary is not already in use. Primaries are essential for cross-building so that the apt-cache can get accurate data for whatever architecture is required. It's quite specific to emdebian, it doesn't help that much in a general script like piuparts of pbuilder - except that I would prefer that a primary is always available because it makes cross-building so much easier. debootstrap: uses ftp.debian.org as default mirror. If ftp.debian.org doesn't contain all architectures, then that's a bad default, imho. Agreed. Probably difficult parts: handling CD-ROM installation I don't think running pbuilder against CD-ROMs is a workable solution in the modern world. There's way too many CDs involved. Anyone who really wants to do that can be expected to go through the small amount of pain of configuring pbuilder manually. (Same discussion happened in Emdebian - CDROM support may be possible in the future but it isn't supported in emdebian-tools yet.) I don't think using debconf is a good idea. Any answer you get is valid when the package is installed, but won't be valid later. The answer should be generated dynamically, each time pbuilder (or piuparts or whatever) needs it. I propose that it be encoded into a new command, apt-default-sources.list or something like that, which, when run, outputs a sources.list that tools like pbuilder and piuparts can use. Put the heuristics into that command, and then it doesn't need to be duplicated in every package that needs it. That would be very handy! The default could then be the closest/quickest primary mirror. It would be important (from my perspective) that this default is required to be a primary mirror - maybe offer the user only the list of primaries and don't allow manual editing. (The current list of primaries is in the emdebian-tools postinst or here: http://www.debian.org/mirrors/list ) -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ pgpzVbWUjpsKC.pgp Description: PGP signature
Re: A sane guess at default Debian mirror for pbuilder
Junichi Uekawa wrote:
Looking for prior art I found the following:
d-i uses the following hack to figure out where to download udebs from
when building installation media:
grep '^deb[ \t]' $(SYSTEM_SOURCES_LIST) \
|grep -v '^deb[ \t]cdrom:' \
|grep -v
'\(security.debian.org\|volatile.debian.\(net\|org\)\)' \
|grep '[ \t]main' \
|awk '{print $$1 $$2}' \
|sed s,/* *$$, $(SUITE) $(UDEB_COMPONENTS), \
|sed s,^deb file,deb copy, \
|perl -ne 'print unless $$seen{$$_}; $$seen{$$_}=1' ; \
--
see shy jo
signature.asc
Description: Digital signature
Re: A sane guess at default Debian mirror for pbuilder
On Sunday 27 May 2007 19:43, Joey Hess wrote: d-i uses the following hack to figure out where to download udebs from when building installation media: Note that this can result in multiple sources. If you want only one, this hack would need to be refined. Cheers, FJP pgpKGmesG3WSG.pgp Description: PGP signature
Re: A sane guess at default Debian mirror for pbuilder
On su, 2007-05-27 at 18:05 +0100, Neil Williams wrote: That would be very handy! The default could then be the closest/quickest primary mirror. It would be important (from my perspective) that this default is required to be a primary mirror - maybe offer the user only the list of primaries and don't allow manual editing. That would be a very unfortunate requirement from my point of view. I have a personal mirror and would prefer not to have to continue to configure ever instance of every tool to use it. -- Fun is productive -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: A sane guess at default Debian mirror for pbuilder
On Sun, 27 May 2007 21:01:19 +0300 Lars Wirzenius [EMAIL PROTECTED] wrote: On su, 2007-05-27 at 18:05 +0100, Neil Williams wrote: That would be very handy! The default could then be the closest/quickest primary mirror. It would be important (from my perspective) that this default is required to be a primary mirror - maybe offer the user only the list of primaries and don't allow manual editing. That would be a very unfortunate requirement from my point of view. I have a personal mirror and would prefer not to have to continue to configure ever instance of every tool to use it. Unless your own mirror supports all Debian architectures, you will still need a primary for emdebian-tools. Do you test build your own Debian packages against your own mirror? Is that wise? -- Neil Williams = http://www.data-freedom.org/ http://www.nosoftwarepatents.com/ http://www.linux.codehelp.co.uk/ pgpsmM2I8xkEf.pgp Description: PGP signature
Re: A sane guess at default Debian mirror for pbuilder
On su, 2007-05-27 at 20:06 +0100, Neil Williams wrote: Unless your own mirror supports all Debian architectures, you will still need a primary for emdebian-tools. Do you test build your own Debian packages against your own mirror? Is that wise? I don't use emdebian in any way, so any requirements it has are irrelevant for which mirror piuparts and pbuilder should use when I use them. However, if I did use emdebian, I would be mirroring anything I need, and would be rather upset if emdebian tools would insist on clogging my network connection (which I might not have while running the tools -- think Debcamp). -- On IRC, we sometimes like to watch silence. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: A sane guess at default Debian mirror for pbuilder
On Mon, May 28, 2007 at 12:25:50AM +0900, Junichi Uekawa wrote: After 6 years or so of setting ftp.jp.debian.org as default for pbuilder, I'm finally determined that it shouldn't stay like this. So I'd like to have some default guessing to happen. Preferably I don't want to ask via debconf, since users should have already answered the question at installation-time. I think the most accurate method would be to scan the list of configured apt sources, and choose the first one which matches the Debian release the user is currently running (via the information from Releases). If the necessary API isn't available yet, this could probably be added to python-apt without too much trouble, and might be useful elsewhere as well. -- - mdz -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
