Re: Alternative: Source-Centric Approach [w/code]
Freddie Unpenstein [EMAIL PROTECTED] writes: I'm wondering, what happens if you want to install MOST of the deps from source? Wouldn't it be better to have apt-build (using the official apt algorithms) ask on a dep-by-dep basis whether you want it compiled from source or installed from a binary? Which is basically what sourcerer acomplishes in a nice, transparent, round up fashion (upload pending some spare time). What if I use dselect, aptitude, or any number of other similar packages as my package manager? Can I select packages to be installed or upgraded? Aptitude uses apt (through libapt) and dselect can (and should be) configured to use apt. I think all frontends (can) use apt with the exception of using dpkg directly. But the use of apt is only important to automatically track what is installed and what not. If you don't want that you can manualy tweak the lists for sourcerer-archive or sourcerer-buildd. Sourcerer-archive itself build a debian archive and you just point apt (or other frontend) to use it as additional (or only) source for packages. I recognise that I may need to use your package to select packages that should be built from source as opposed to installed from binary packages, but can they be still be upgraded (via source) though my regular package manager? In my current setup I have sourcerer-watcher follow my apt usage completly so everything goes automatic. Here is what happens: 1) I : apt-get install foobar - download foobar from debian and install 2) watcher: add foobar to list of sources to carry and build 3) archive: mirror foobar sources 4) buildd : build local foobar 5) I : apt-get update; apt-get upgrade - install local build foobar 6) time : when a new source comes out go back to 3 My usual pattern is to update and select using dselect, download using a cron job with apt-get -d dselect-upgrade, followed by installing the following day with apt-get dselect-upgrade... What part of that process would need to change to support your package? Nothing. In fact to get your current behaviour you would tell the watcher to add/remove packages from the sourcerer-archive list and leave sourcerer-buildd alone. And you tell sourcerer-archive to mirror only debs. Every night the archive then downloads any new version of installed debs and the next day you can upgrade them directly. You don't need to run a buildd for this. That is just one of the sources for debs. Fredderic MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Freddie Unpenstein [EMAIL PROTECTED] writes: Your priority are your users, and if Debian has decided to focus only on some key architectures it would be the best for them to help them switching to Gentoo instead of hacking Debian to become some cheap Gentoo clone for most architectures. I don't view this as being a cheap Gentoo clone. In fact, I view srcinst as being the Gentoo idea, done right. Gentoo has a lot of problems, especially relating to difficult upgrades. Because of our superior packaging system, we are in a great position to hit the ground running and, with a little help from something like srcinst, come up with something that works -- and works better than Gentoo -- in a short amount of time. I'm wondering, what happens if you want to install MOST of the deps from source? Wouldn't it be better to have apt-build (using the official apt algorithms) ask on a dep-by-dep basis whether you want it compiled from source or installed from a binary? Even better, would be to allow apt's preferences file to state whether a specific package should be installed from binary or source, and have the stock apt-get install do what's appropriate. With a few options to set the default mode (binary or source), and to overide the mode for specific packages. And of course, last but not least, apt should keep the package comming from the same source unless specifically changed. Now, if only apt-get understood that a pacakge may be available from more than one mirror at a time... Fredderic Which is basically what sourcerer acomplishes in a nice, transparent, round up fashion (upload pending some spare time). sourcerer-archive maintains a local archive of selected sources and packages from various sources (debian mirrors or incoming queue). E.g. mirror all installed packages and sources. sourcerer-watcher watches for apt activity and can update the selection of sourcerer-archive automatically. On apt-get install foobar foobar gets added to the package list, on purge it gets removed. sourcerer-buildd finaly builds selected packages from the archive and uploads it to the archive again. Local gcc options can be enforced, e.g. to optimize for i686, and the version is modified bumping the 4th part of the debian revision (debian only uses 3) to generate a version higher than this one and lower than any debian upload. With this and apts pining it is trivial to install and update a system with only locally build debs or with the prebuild debs as fallback for the first install or as fallback for anything yet unbuild or build failures. This also has one big advantage over apt-get install from source: You do not have to wait for the compile to finish. The nightly cron job will start the build and the new debs will be ready for install in the morning. MfG Goswin -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
I'm wondering, what happens if you want to install MOST of the deps from source? Wouldn't it be better to have apt-build (using the official apt algorithms) ask on a dep-by-dep basis whether you want it compiled from source or installed from a binary? Which is basically what sourcerer acomplishes in a nice, transparent, round up fashion (upload pending some spare time). What if I use dselect, aptitude, or any number of other similar packages as my package manager? Can I select packages to be installed or upgraded? I recognise that I may need to use your package to select packages that should be built from source as opposed to installed from binary packages, but can they be still be upgraded (via source) though my regular package manager? My usual pattern is to update and select using dselect, download using a cron job with apt-get -d dselect-upgrade, followed by installing the following day with apt-get dselect-upgrade... What part of that process would need to change to support your package? Fredderic ___ Join Excite! - http://www.excite.com The most personalized portal on the Web! -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 11:25:23AM -0600, John Goerzen wrote: I'm throwing out a different idea, I propose that we split things along these lines: binary+source (B+S) archs and source-only (SO) archs. SO archs will be handled exactly like we do now, EXCEPT that we will not distribute .debs for most packages. I expect that we will distribute .debs for base and build-essential, mainly -- the minimum someone needs to install a working system and build more packages. * Difficulty of dealing with dependency loops (possible solution: include one .deb from each loop in the dist) And for compilers written in their own language :-) -- Lionel -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Your priority are your users, and if Debian has decided to focus only on some key architectures it would be the best for them to help them switching to Gentoo instead of hacking Debian to become some cheap Gentoo clone for most architectures. I don't view this as being a cheap Gentoo clone. In fact, I view srcinst as being the Gentoo idea, done right. Gentoo has a lot of problems, especially relating to difficult upgrades. Because of our superior packaging system, we are in a great position to hit the ground running and, with a little help from something like srcinst, come up with something that works -- and works better than Gentoo -- in a short amount of time. I'm wondering, what happens if you want to install MOST of the deps from source? Wouldn't it be better to have apt-build (using the official apt algorithms) ask on a dep-by-dep basis whether you want it compiled from source or installed from a binary? Even better, would be to allow apt's preferences file to state whether a specific package should be installed from binary or source, and have the stock apt-get install do what's appropriate. With a few options to set the default mode (binary or source), and to overide the mode for specific packages. And of course, last but not least, apt should keep the package comming from the same source unless specifically changed. Now, if only apt-get understood that a pacakge may be available from more than one mirror at a time... Fredderic ___ Join Excite! - http://www.excite.com The most personalized portal on the Web! -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Fri, Mar 18, 2005 at 07:39:06PM -0600, Gunnar Wolf wrote: Matthias Urlichs dijo [Tue, Mar 15, 2005 at 11:14:50PM +0100]: It won't work that well for slower architectures, for the very simple reason that compiling everything would take a long time. m68k (as the admittedly extreme example) doesn't have ten buildd boxes just because we feel like it. :-) Being m68k among the prime candidates for SCC, why shouldn't we ponder using emulated autobuilders? We have some quite decent and reliable emulators in our archive for some architectures (i.e., basilisk2 for m68k [in contrib because of the needed Mac ROMs], hercules for s390), and they do work reliably. It's bad enough already that the toolchain (which usually does work reliably) breaks from time to time; having an emulator which might break in not immediately obvious ways doesn't sound like a particularly appealing idea to me. There is no problem whatsoever for us m68k buildd maintainers in supporting a dozen or so buildd machines -- or more, if required. It works like a charm. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Matthias Urlichs dijo [Tue, Mar 15, 2005 at 11:14:50PM +0100]: It won't work that well for slower architectures, for the very simple reason that compiling everything would take a long time. m68k (as the admittedly extreme example) doesn't have ten buildd boxes just because we feel like it. :-) Being m68k among the prime candidates for SCC, why shouldn't we ponder using emulated autobuilders? We have some quite decent and reliable emulators in our archive for some architectures (i.e., basilisk2 for m68k [in contrib because of the needed Mac ROMs], hercules for s390), and they do work reliably. (BTW: I have had for a couple of months a Mac Quadra 950 sitting in my house... It seems to work, but I have had no time to set it up :-/ ) Greetings, -- Gunnar Wolf - [EMAIL PROTECTED] - (+52-55)1451-2244 / 5554-9450 PGP key 1024D/8BB527AF 2001-10-23 Fingerprint: 0C79 D2D1 2C4E 9CE4 5973 F800 D80E F35A 8BB5 27AF -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Op di, 15-03-2005 te 11:25 -0600, schreef John Goerzen: As I have been reading the discussions about the SCC proposal for etch, it seems that these are the main problems: 1) Difficulty with, and speed of, buildd systems 2) Difficulty of syncing testing across all archs given #1 3) Difficulty getting security releases out in time, given slow archs 4) Space constraints on mirrors I'm throwing out a different idea, and I'm backing it up with code. I have thought about it some, maybe there are huge holes, but let's see. I propose that we split things along these lines: binary+source (B+S) archs and source-only (SO) archs. [...long detailed explanation of how this would work in practice snipped...] I don't think this is a good idea. One of Debian's key selling points is the fact that you /don't/ have to wait for something to build before being able to use it. We're not Gentoo[1]; many of our users come to Debian because they want a community-developed, binary-based distribution. Changing that would change what Debian is, too much. Also it wouldn't help on slower architectures. People usually decline installing NetBSD on m68k (even if that's possible) when it takes two weeks to make the system useful, simply because everything needs to be compiled manually. [1] this is not meant pejoratively; I'm have tried Gentoo in the past and am quite sure it's a great distribution, but it has different goals. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Wed, Mar 16, 2005 at 02:30:29PM +0100, Wouter Verhelst wrote: ... Also it wouldn't help on slower architectures. People usually decline installing NetBSD on m68k (even if that's possible) when it takes two weeks to make the system useful, simply because everything needs to be compiled manually. ... NetBSD also offers binary packages for many years. cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Hello On Tue, Mar 15, 2005 at 04:10:17PM -0600, John Goerzen wrote: On Tue, Mar 15, 2005 at 10:45:45PM +0100, Ola Lundqvist wrote: Hello distribute for a SO arch). Anything past that is there just for QA purposes -- to make sure packages are buildable on these archs, and would be optional. This is the problem. How do you make sure that the package is buildable on the architecture without building it? And if you have built it why not Well, how do you make sure that the package is runnable on the architecture without running it? You have a point here. Yes, it's a bit less testing, but OTOH, if nobody notices that the package can't be installed, that says something. :) The problem of syncing with testing is also reduced; now we need only make sure base is synced across archs. What you really are saying that for some arches we only support base and essential (and some more). Everything else is provided as source debs and not supported, even if it might work. I mean the source is available currently and the only thing you say is that we should only build some parts of that port. No, I'm not commenting on suport. I'm just commenting on what .debs are out there and autobuilt. I want everything else to stay the same. However, the job of supporting n archs for things like security updates is reduced to the job of supporting 1 source package. True. * Difficulty of dealing with dependency loops (possible solution: include one .deb from each loop in the dist) * Disk space requirements to build packages * Not possible to tell if a package is buildable on a specific arch or not. ^ in advance Yes, that is a downside. So, what do you think? Could this work? Nice idea, but I do not really see the benefit, more than on ftp disk space and security update speed. Also with the testing synchronization. But then, these three are the main problems we've been told about, right? :-) I see that we can have a more flexible approach for releases. Full support (i386, etc): all packages are supported (debs, security updates etc). Basic support (some other arches): installer, base and essential pacakges built, maybe with some more things. The rest is up to the user to compile, maybe using something similar that you have here. No support (sh, hurd...): Well some things may work but that is nothing that Debian guarantees. :) This is just what comes to mind. I'm not saying we should do this, and not even that I want it to. It was just an idea, and very similar to yours. Regards, // Ola -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- - Ola Lundqvist --- / [EMAIL PROTECTED] Annebergsslingan 37 \ | [EMAIL PROTECTED] 654 65 KARLSTAD | | +46 (0)54-10 14 30 +46 (0)70-332 1551 | | http://www.opal.dhs.org UIN/icq: 4912500 | \ gpg/f.p.: 7090 A92B 18FE 7994 0C36 4FE4 18A1 B1CF 0FE5 3DD9 / --- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Hello On Wed, Mar 16, 2005 at 02:30:29PM +0100, Wouter Verhelst wrote: Op di, 15-03-2005 te 11:25 -0600, schreef John Goerzen: As I have been reading the discussions about the SCC proposal for etch, it seems that these are the main problems: 1) Difficulty with, and speed of, buildd systems 2) Difficulty of syncing testing across all archs given #1 3) Difficulty getting security releases out in time, given slow archs 4) Space constraints on mirrors I'm throwing out a different idea, and I'm backing it up with code. I have thought about it some, maybe there are huge holes, but let's see. I propose that we split things along these lines: binary+source (B+S) archs and source-only (SO) archs. [...long detailed explanation of how this would work in practice snipped...] I don't think this is a good idea. One of Debian's key selling points is the fact that you /don't/ have to wait for something to build before being able to use it. We're not Gentoo[1]; many of our users come to Debian because they want a community-developed, binary-based distribution. Changing that would change what Debian is, too much. Also it wouldn't help on slower architectures. People usually decline installing NetBSD on m68k (even if that's possible) when it takes two weeks to make the system useful, simply because everything needs to be compiled manually. This is a really a good point. I agree that provide source packages are not really working on slower arches. Regards, // Ola [1] this is not meant pejoratively; I'm have tried Gentoo in the past and am quite sure it's a great distribution, but it has different goals. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED] -- - Ola Lundqvist --- / [EMAIL PROTECTED] Annebergsslingan 37 \ | [EMAIL PROTECTED] 654 65 KARLSTAD | | +46 (0)54-10 14 30 +46 (0)70-332 1551 | | http://www.opal.dhs.org UIN/icq: 4912500 | \ gpg/f.p.: 7090 A92B 18FE 7994 0C36 4FE4 18A1 B1CF 0FE5 3DD9 / --- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Alternative: Source-Centric Approach [w/code]
As I have been reading the discussions about the SCC proposal for etch, it seems that these are the main problems: 1) Difficulty with, and speed of, buildd systems 2) Difficulty of syncing testing across all archs given #1 3) Difficulty getting security releases out in time, given slow archs 4) Space constraints on mirrors I'm throwing out a different idea, and I'm backing it up with code. I have thought about it some, maybe there are huge holes, but let's see. I propose that we split things along these lines: binary+source (B+S) archs and source-only (SO) archs. B+S archs will be essentially like we have now -- .debs and sources for every package in Debian are available. SO archs will be handled exactly like we do now, EXCEPT that we will not distribute .debs for most packages. I expect that we will distribute .debs for base and build-essential, mainly -- the minimum someone needs to install a working system and build more packages. What will that mean for our problems? The speed of buildd systems mostly becomes irrelevant. They will still have to keep up with base (the set of .debs that we do distribute for a SO arch). Anything past that is there just for QA purposes -- to make sure packages are buildable on these archs, and would be optional. The problem of syncing with testing is also reduced; now we need only make sure base is synced across archs. The problem of getting security releases out in time is also reduced; source packages are enough for these archs. If there's a hole in a base package, we'd want to provide .debs for everyone, but these packages aren't going to be the KDE-style mammoth ones. And finally, this would be a huge win for mirrors. We would have far fewer space constraints, and adding a new arch would be easier. What would this mean for users? Basically, packages install slower on these archs. I have developed a demonstration tool called srcinst, available from http://people.debian.org/~jgoerzen/srcinst/. srcinst is capable of filling all of a package's dependencies and build-depencies solely from source. It will use apt-get -b source to build .debs, then evaluate (and build, if necessary) their deps, then install them with dpkg -i. In short, very similar to apt-get install, except it never downloads a .deb from anywhere for any reason. (Most other tools don't go this far, and still require .deb downloads for build-deps, or don't handle deps at all) This gives us the benefits of smaller mirror infrastructure that projects like Gentoo have, while still maintaining all the advantages of dpkg/apt. What are some downsides? I can see a few: * Longer package install times, obviously * Difficulty of dealing with dependency loops (possible solution: include one .deb from each loop in the dist) * Disk space requirements to build packages More on srcinst: Here's an example for epic4, which looks like this: Depends: libc6 (= 2.3.2.ds1-4), libncurses5 (= 5.4-1), libssl0.9.7, epic4-help (= 1:1.1.7.20020907) Build-Depends: debhelper (= 3.4.8), libncurses5-dev, libssl-dev Let's assume I have libssl-dev installed and libc6 installed, but no ncurses5. So, I run srcinst install epic4. Here's what it does: 1. Looks at the build-depends, sees that we are missing libncurses5-dev. So: a. Grab and build the source package corresponding to libncurses5-dev. b. Examine the deps in libncurses5-dev.deb. Notice that it deps on libncurses5 (which we just built), so dkpg -i libncurses5 and then dpkg -i libncurses5-dev. 2. Build epic4 itself. 3. Look at the deps in the epic4 .deb. Notice that it deps on epic4-help. 4. Build and install epic4-help. 5. Install epic4. The srcinst code is very hackish, ugly, and quick. I just wrote it since yesterday as a proof of concept. So: * Make sure yuo rnu it as root (no support for fakeroot/sudo yet) * It can't resolve dep loops * It can't handle !arch strings in deps accurately * It spews debugging info to stdout * It uses /var/cache/srcinst for its working area. Make sure /var is free enough. So, what do you think? Could this work? -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 11:25:23AM -0600, John Goerzen wrote: So, what do you think? Could this work? I like the idea a lot. What I'd like to see is a way to do a cross-platform build for the small system targets. I do a lot of ARM work: low-performance, resource limited targets. Frankly, this is something I'm actively working on. I agree that the Debian infrastructure shouldn't be required to do all of the building. It would be helpful, though, if there was support for other arch's to be built efficiently by the users of those arches. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
* Marc Singer ([EMAIL PROTECTED]) wrote: On Tue, Mar 15, 2005 at 11:25:23AM -0600, John Goerzen wrote: So, what do you think? Could this work? I like the idea a lot. What I'd like to see is a way to do a cross-platform build for the small system targets. I do a lot of ARM work: low-performance, resource limited targets. Frankly, this is something I'm actively working on. I agree that the Debian infrastructure shouldn't be required to do all of the building. It would be helpful, though, if there was support for other arch's to be built efficiently by the users of those arches. I'm not particularly fond of the idea actually, mostly because of the strain it'd put on our users. An alternative: - Mirror only the popular archs. - Support buildds for stable-enough archs that run them. - Try to include everything in a release, but drop archs more quickly than has been done in the past if there's a lack of resources, but do outline what people need to do if they want the arch included. - For archs that weren't able to make the cut, provide .debs for the packages which don't have RC bugs. Don't provide .debs for the packages which have RC bugs on that architecture. Attempt to include the archs in security updates, but if it's not built fast enough or whatever then drop the .deb for that package and provide the source update. A fixed .deb can be done later by whomever. I dunno, just some thoughts. I don't like becoming a mostly source-only distro for less common archs. If someone wants to build the debs and upload them, let them. Stephen signature.asc Description: Digital signature
Re: Alternative: Source-Centric Approach [w/code]
On Tuesday 15 of March 2005 18:25, John Goerzen wrote: [...] More on srcinst: [...] So, what do you think? Could this work? What's a difference between srcinst and apt-build ? ;-) Regards. -- Lech Karol Pawaszek ike You will never see me fall from grace... [KoRn]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 12:42:54PM -0500, Stephen Frost wrote: - Mirror only the popular archs. - Support buildds for stable-enough archs that run them. - Try to include everything in a release, but drop archs more quickly than has been done in the past if there's a lack of resources, but do outline what people need to do if they want the arch included. - For archs that weren't able to make the cut, provide .debs for the packages which don't have RC bugs. Don't provide .debs for the packages which have RC bugs on that architecture. Attempt to include the archs in security updates, but if it's not built fast enough or whatever then drop the .deb for that package and provide the source update. A fixed .deb can be done later by whomever. I dunno, just some thoughts. I don't like becoming a mostly source-only distro for less common archs. If someone wants to build the debs and upload them, let them. This sounds like the idea already going around of, essentially, 'build what we can'. I'm suggesting that we go a step further and enable a new segment of the user base. The turnkey people use x86 (+relatives) and PPC. The people on other arches can either step-up to provide resources to Debian, or they can build what they need. Of course, I'd rather not cut the support. But John draws, IMHO, a reasonable line in the sand. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Hi, John Goerzen wrote:
1) Difficulty with, and speed of, buildd systems
2) Difficulty of syncing testing across all archs given #1
2a) Bugs on small arch which blocks testing migration of big arch
There are not many people who can do in-depth debugging on most small
architectures, arch-specific gcc internals are less-well exercised and
therefore more buggy, just recompiling gcc is a problem ...
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | [EMAIL PROTECTED]
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 06:42:32PM +0100, Lech Karol Paw?aszek wrote: On Tuesday 15 of March 2005 18:25, John Goerzen wrote: [...] More on srcinst: [...] So, what do you think? Could this work? What's a difference between srcinst and apt-build ? ;-) egrep 'apt-get.*install' apt-build | wc -l 8 egrep -r 'apt-get.*install' srcinst | wc -l 0 IOW, apt-build still requires pre-built binary .debs in order to work. srcinst does not. -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 11:25:23AM -0600, John Goerzen wrote: ... SO archs will be handled exactly like we do now, EXCEPT that we will not distribute .debs for most packages. I expect that we will distribute .debs for base and build-essential, mainly -- the minimum someone needs to install a working system and build more packages. ... What would this mean for users? Basically, packages install slower on these archs. I have developed a demonstration tool called srcinst, available from http://people.debian.org/~jgoerzen/srcinst/. srcinst is capable of filling all of a package's dependencies and build-depencies solely from source. It will use apt-get -b source to build .debs, then evaluate (and build, if necessary) their deps, then install them with dpkg -i. In short, very similar to apt-get install, except it never downloads a .deb from anywhere for any reason. (Most other tools don't go this far, and still require .deb downloads for build-deps, or don't handle deps at all) ... So, what do you think? Could this work? Yes, this could work. That's what Gentoo is good at. Both source-only distributions like Gentoo and binary distributions like Debian are possible - and both have their advantages and disadvantages. I'd hate so see Debian drop most of it's architectures, and as I already expressed I think it's neither required for release management reasons nor for ftp space reasons. Your priority are your users, and if Debian has decided to focus only on some key architectures it would be the best for them to help them switching to Gentoo instead of hacking Debian to become some cheap Gentoo clone for most architectures. If the Debian developers intereested in the ports Debian intends to drop would switch to Gentoo for helping Gentoo to support all of their architectures and for providing easy Debian - Gentoo transition paths this would serve your users better. -- John cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 06:57:00PM +0100, Matthias Urlichs wrote: Hi, John Goerzen wrote: 1) Difficulty with, and speed of, buildd systems 2) Difficulty of syncing testing across all archs given #1 2a) Bugs on small arch which blocks testing migration of big arch There are not many people who can do in-depth debugging on most small architectures, arch-specific gcc internals are less-well exercised and therefore more buggy, just recompiling gcc is a problem ... Granted, gcc and libc can cause an inordinate amount of trouble there. But in almost every other case, a bug in a small arch is a genuine bug in the program that should be fixed, and usually is not exceptionally hard to find. -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 07:46:23PM +0100, Adrian Bunk wrote: don't handle deps at all) ... So, what do you think? Could this work? Yes, this could work. That's what Gentoo is good at. [ snip ] Your priority are your users, and if Debian has decided to focus only on some key architectures it would be the best for them to help them switching to Gentoo instead of hacking Debian to become some cheap Gentoo clone for most architectures. I don't view this as being a cheap Gentoo clone. In fact, I view srcinst as being the Gentoo idea, done right. Gentoo has a lot of problems, especially relating to difficult upgrades. Because of our superior packaging system, we are in a great position to hit the ground running and, with a little help from something like srcinst, come up with something that works -- and works better than Gentoo -- in a short amount of time. If the Debian developers intereested in the ports Debian intends to drop would switch to Gentoo for helping Gentoo to support all of their architectures and for providing easy Debian - Gentoo transition paths this would serve your users better. Yes, but I hope that this proposal, or other suggestions, can help us avoid dropping ports. This specific proposal, for instance, is meant to provide us with a way forward that addresses the main concerns while still producing a quality, usable result for our users. -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 02:24:01PM -0600, John Goerzen wrote: On Tue, Mar 15, 2005 at 07:46:23PM +0100, Adrian Bunk wrote: don't handle deps at all) ... So, what do you think? Could this work? Yes, this could work. That's what Gentoo is good at. [ snip ] Your priority are your users, and if Debian has decided to focus only on some key architectures it would be the best for them to help them switching to Gentoo instead of hacking Debian to become some cheap Gentoo clone for most architectures. I don't view this as being a cheap Gentoo clone. In fact, I view srcinst as being the Gentoo idea, done right. Gentoo has a lot of problems, especially relating to difficult upgrades. Because of our superior packaging system, we are in a great position to hit the ground running and, with a little help from something like srcinst, come up with something that works -- and works better than Gentoo -- in a short amount of time. If the Debian developers intereested in the ports Debian intends to drop would switch to Gentoo for helping Gentoo to support all of their architectures and for providing easy Debian - Gentoo transition paths this would serve your users better. Yes, but I hope that this proposal, or other suggestions, can help us avoid dropping ports. This specific proposal, for instance, is meant to provide us with a way forward that addresses the main concerns while still producing a quality, usable result for our users. ...and may encourage development of new architectures as the overhead for each would be much smaller. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 12:53:31PM -0800, Marc Singer wrote: Yes, but I hope that this proposal, or other suggestions, can help us avoid dropping ports. This specific proposal, for instance, is meant to provide us with a way forward that addresses the main concerns while still producing a quality, usable result for our users. ...and may encourage development of new architectures as the overhead for each would be much smaller. Exactly. If we were doing this, there would have been no reason to keep amd64 out of sarge, for instance. -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Hello On Tue, Mar 15, 2005 at 11:25:23AM -0600, John Goerzen wrote: As I have been reading the discussions about the SCC proposal for etch, it seems that these are the main problems: 1) Difficulty with, and speed of, buildd systems 2) Difficulty of syncing testing across all archs given #1 3) Difficulty getting security releases out in time, given slow archs 4) Space constraints on mirrors I'm throwing out a different idea, and I'm backing it up with code. I have thought about it some, maybe there are huge holes, but let's see. I propose that we split things along these lines: binary+source (B+S) archs and source-only (SO) archs. As you probably know it has been expressed before. B+S archs will be essentially like we have now -- .debs and sources for every package in Debian are available. SO archs will be handled exactly like we do now, EXCEPT that we will not distribute .debs for most packages. I expect that we will distribute .debs for base and build-essential, mainly -- the minimum someone needs to install a working system and build more packages. Might be a good alternative, but see below. What will that mean for our problems? The speed of buildd systems mostly becomes irrelevant. They will still have to keep up with base (the set of .debs that we do distribute for a SO arch). Anything past that is there just for QA purposes -- to make sure packages are buildable on these archs, and would be optional. This is the problem. How do you make sure that the package is buildable on the architecture without building it? And if you have built it why not just add it to the archives. :) So you still need a buildd. :( The problem of syncing with testing is also reduced; now we need only make sure base is synced across archs. What you really are saying that for some arches we only support base and essential (and some more). Everything else is provided as source debs and not supported, even if it might work. I mean the source is available currently and the only thing you say is that we should only build some parts of that port. The problem of getting security releases out in time is also reduced; source packages are enough for these archs. If there's a hole in a base package, we'd want to provide .debs for everyone, but these packages aren't going to be the KDE-style mammoth ones. This will help yes. And finally, this would be a huge win for mirrors. We would have far fewer space constraints, and adding a new arch would be easier. What would this mean for users? Basically, packages install slower on these archs. I have developed a demonstration tool called srcinst, available from http://people.debian.org/~jgoerzen/srcinst/. srcinst is capable of filling all of a package's dependencies and build-depencies solely from source. It will use apt-get -b source to build .debs, then evaluate (and build, if necessary) their deps, then install them with dpkg -i. In short, very similar to apt-get install, except it never downloads a .deb from anywhere for any reason. (Most other tools don't go this far, and still require .deb downloads for build-deps, or don't handle deps at all) Nice. This gives us the benefits of smaller mirror infrastructure that projects like Gentoo have, while still maintaining all the advantages of dpkg/apt. What are some downsides? I can see a few: * Longer package install times, obviously * Difficulty of dealing with dependency loops (possible solution: include one .deb from each loop in the dist) * Disk space requirements to build packages * Not possible to tell if a package is buildable on a specific arch or not. More on srcinst: Here's an example for epic4, which looks like this: Depends: libc6 (= 2.3.2.ds1-4), libncurses5 (= 5.4-1), libssl0.9.7, epic4-help (= 1:1.1.7.20020907) Build-Depends: debhelper (= 3.4.8), libncurses5-dev, libssl-dev Let's assume I have libssl-dev installed and libc6 installed, but no ncurses5. So, I run srcinst install epic4. Here's what it does: 1. Looks at the build-depends, sees that we are missing libncurses5-dev. So: a. Grab and build the source package corresponding to libncurses5-dev. b. Examine the deps in libncurses5-dev.deb. Notice that it deps on libncurses5 (which we just built), so dkpg -i libncurses5 and then dpkg -i libncurses5-dev. 2. Build epic4 itself. 3. Look at the deps in the epic4 .deb. Notice that it deps on epic4-help. 4. Build and install epic4-help. 5. Install epic4. The srcinst code is very hackish, ugly, and quick. I just wrote it since yesterday as a proof of concept. So: * Make sure yuo rnu it as root (no support for fakeroot/sudo yet) * It can't resolve dep loops * It can't handle !arch strings in deps accurately * It spews debugging info to stdout * It uses /var/cache/srcinst for its working area. Make sure /var is free enough. So,
Re: Alternative: Source-Centric Approach [w/code]
Ola Lundqvist on 2005-03-15 22:45:45 +0100: This is the problem. How do you make sure that the package is buildable on the architecture without building it? And if you have built it why not just add it to the archives. :) So you still need a buildd. :( Why not add it to the archives? Because there isn't enough disk space. pgpbzpDrgpQQb.pgp Description: PGP signature
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 04:55:08PM -0500, Alec Berryman wrote: Ola Lundqvist on 2005-03-15 22:45:45 +0100: This is the problem. How do you make sure that the package is buildable on the architecture without building it? And if you have built it why not just add it to the archives. :) So you still need a buildd. :( Why not add it to the archives? Because there isn't enough disk space. Where exactly isn't enough space? On Debian servers? - There'd be enough money for new disks or even new servers. On some mirrors? - Not all mirrors have to mirror all ports. cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 10:45:45PM +0100, Ola Lundqvist wrote: The speed of buildd systems mostly becomes irrelevant. They will still have to keep up with base (the set of .debs that we do distribute for a SO arch). Anything past that is there just for QA purposes -- to make sure packages are buildable on these archs, and would be optional. This is the problem. How do you make sure that the package is buildable on the architecture without building it? And if you have built it why not just add it to the archives. :) So you still need a buildd. :( Not necessarily. I think if we make it easy for end users to perform selective builds we have a chance of making it work. So, what do you think? Could this work? Nice idea, but I do not really see the benefit, more than on ftp disk space and security update speed. While I would like to *not* see a change to the build structure, I agree that removing lesser used arch's from the testing guaranteed support infrastructure gives room for higher frequency releases. IMHO, the disk space issue is a red herring. Security updates, too, are not the primary concern, it's getting all of the cats out the door with some frequency. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 10:45:45PM +0100, Ola Lundqvist wrote: Hello distribute for a SO arch). Anything past that is there just for QA purposes -- to make sure packages are buildable on these archs, and would be optional. This is the problem. How do you make sure that the package is buildable on the architecture without building it? And if you have built it why not Well, how do you make sure that the package is runnable on the architecture without running it? Yes, it's a bit less testing, but OTOH, if nobody notices that the package can't be installed, that says something. The problem of syncing with testing is also reduced; now we need only make sure base is synced across archs. What you really are saying that for some arches we only support base and essential (and some more). Everything else is provided as source debs and not supported, even if it might work. I mean the source is available currently and the only thing you say is that we should only build some parts of that port. No, I'm not commenting on suport. I'm just commenting on what .debs are out there and autobuilt. I want everything else to stay the same. However, the job of supporting n archs for things like security updates is reduced to the job of supporting 1 source package. * Difficulty of dealing with dependency loops (possible solution: include one .deb from each loop in the dist) * Disk space requirements to build packages * Not possible to tell if a package is buildable on a specific arch or not. ^ in advance Yes, that is a downside. So, what do you think? Could this work? Nice idea, but I do not really see the benefit, more than on ftp disk space and security update speed. Also with the testing synchronization. But then, these three are the main problems we've been told about, right? :-) -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 11:01:06PM +0100, Adrian Bunk wrote: On some mirrors? - Not all mirrors have to mirror all ports. The mirroring part of the proposal is effectively just a proposal to rearrange the archive in order to make this easy for mirror admins. -- You grabbed my hand and we fell into it, like a daydream - or a fever. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Hi, John Goerzen wrote:
This specific proposal, for instance, is meant to
provide us with a way forward that addresses the main concerns while
still producing a quality, usable result for our users.
It won't work that well for slower architectures, for the very simple
reason that compiling everything would take a long time.
m68k (as the admittedly extreme example) doesn't have ten buildd boxes
just because we feel like it. :-)
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | [EMAIL PROTECTED]
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
On Tue, Mar 15, 2005 at 11:14:50PM +0100, Matthias Urlichs wrote: Hi, John Goerzen wrote: This specific proposal, for instance, is meant to provide us with a way forward that addresses the main concerns while still producing a quality, usable result for our users. It won't work that well for slower architectures, for the very simple reason that compiling everything would take a long time. It may be that m68k is one arch that should ship with .debs, for this very reason. -- John -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Alternative: Source-Centric Approach [w/code]
Mark Brown schrieb: On Tue, Mar 15, 2005 at 11:01:06PM +0100, Adrian Bunk wrote: On some mirrors? - Not all mirrors have to mirror all ports. The mirroring part of the proposal is effectively just a proposal to rearrange the archive in order to make this easy for mirror admins. [-snip-] [EMAIL PROTECTED]:/root/scripts$ cat anonftpsync #! /bin/sh set -e # This script originates from http://www.debian.org/mirror/anonftpsync # Note: You MUST have rsync 2.0.16-1 or newer, which is available in slink # and all newer Debian releases, or at http://rsync.samba.org/ # Set the variables below to fit your site. You can then use cron to have # this script run daily to automatically update your copy of the archive. # Don't forget: # chmod 744 anonftpsync # TO is the destination for the base of the Debian mirror directory # (the dir that holds dists/ and ls-lR). TO=/home/mirrors/debian # RSYNC_HOST is the site you have chosen from the mirrors file. # (http://www.debian.org/mirror/mirrors_full) RSYNC_HOST=mastermirror.stayout.int # RSYNC_DIR is the directory given in the Packages over rsync: line of # the mirrors file for the site you have chosen to mirror. RSYNC_DIR=debian/ # EXCLUDE is a list of parameters listing patterns that rsync will exclude. # With a blank EXCLUDE you will mirror the entire archive. EXCLUDE=\ --exclude binary-alpha --exclude binary-arm \ --exclude binary-m68k \ --exclude binary-ia64 --exclude binary-mips* --exclude binary-hppa --exclude binary-sh \ --exclude *_hurd-i386.deb --exclude *_mipsel.deb \ --exclude *_s390.deb \ --exclude *_alpha.deb --exclude *_arm.deb \ --exclude *_m68k.deb \ --exclude *_ia64.deb --exclude *_hppa.deb --exclude *_mips.deb --exclude *_sh.deb \ --exclude slink --exclude slink-proposed-updates \ # There should be no need to edit anything below this point, unless [-snip-] Were's the problem? greets Uwe -- Jetzt will man das Internet nicht einfach ein paar Leuten wie der IETF überlassen, die wissen, was sie tun. Es ist zu wichtig geworden. - Scott Bradner http://www.highspeed-firewall.de/adamantix/ http://www.x-tec.de
