Re: [gentoo-dev] Re: Re: Versioning the tree
On Friday 01 December 2006 13:47, Chris Gianelloni wrote: Actually, we would have to review the process, since not everything that gets a security bug ends up with a GLSA. My current loose rule is that if it deserves a GLSA, then it deserves and update, but I don't know the exact criteria the security team uses to decide if something warrants a GLSA or not. http://www.gentoo.org/security/en/vulnerability-policy.xml For relation between severity level and GLSA publication see Dispatch. Basically everything that ends up with Trivial severity level will NOT get a GLSA and everything that ends up with Minor severity level will get a vote from the Security team members. Two yes or no votes normally wins. Everything else gets a GLSA. Then you have to add in Security supported architectures, but that's really of no concern to x86. -- Sune Kloppenborg Jeppesen Gentoo Linux Security Team pgp5S2l2N6A2k.pgp Description: PGP signature
Re: [gentoo-dev] Re: Re: Versioning the tree
Steve Long wrote: There'll always be GLSA's to respond to. That's another issue that needs to be handled w/ a slow-moving tree. Are you going to restrict changes in the slow-moving tree only to changes against a GLSA? That's what we've said. I don't have a problem with this at all. The slow-moving tree isn't; it's a release tree. The only question I have, which Stuart also mentioned, is whether all security updates go thru the GLSA process. Are you asking if all security updates that are done to the release will have gone through the GLSA process? I'd say the answer is yes, since the only updates that will go in the release tree are security updates from GLSAs :P -- Andrew Gaffneyhttp://dev.gentoo.org/~agaffney/ Gentoo Linux Developer Installer Project Today's lesson in political correctness: Go asphyxiate on a phallus -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Re: Versioning the tree
On Fri, 2006-12-01 at 07:22 -0600, Andrew Gaffney wrote: Steve Long wrote: There'll always be GLSA's to respond to. That's another issue that needs to be handled w/ a slow-moving tree. Are you going to restrict changes in the slow-moving tree only to changes against a GLSA? That's what we've said. I don't have a problem with this at all. The slow-moving tree isn't; it's a release tree. The only question I have, which Stuart also mentioned, is whether all security updates go thru the GLSA process. Are you asking if all security updates that are done to the release will have gone through the GLSA process? I'd say the answer is yes, since the only updates that will go in the release tree are security updates from GLSAs :P Actually, we would have to review the process, since not everything that gets a security bug ends up with a GLSA. My current loose rule is that if it deserves a GLSA, then it deserves and update, but I don't know the exact criteria the security team uses to decide if something warrants a GLSA or not. -- Chris Gianelloni Release Engineering Strategic Lead Alpha/AMD64/x86 Architecture Teams Games Developer/Council Member/Foundation Trustee Gentoo Foundation signature.asc Description: This is a digitally signed message part
[gentoo-dev] Re: Re: Versioning the tree
There'll always be GLSA's to respond to. That's another issue that needs to be handled w/ a slow-moving tree. Are you going to restrict changes in the slow-moving tree only to changes against a GLSA? That's what we've said. I don't have a problem with this at all. The slow-moving tree isn't; it's a release tree. The only question I have, which Stuart also mentioned, is whether all security updates go thru the GLSA process. -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Re: Versioning the tree
Steve Long wrote: Chris Gianelloni wrote: From your post we need to add: - strip all USE flags that aren't used from use.mask (per-profile) - strip all packages that aren't available from package.mask (per-profile) What language is the script implemented in? The current script is actually 2 scripts: a python script that I wrote that interfaces with the portage API and a bash script that wolf31o2 wrote that takes the output from my script and does the cleanup. Last night, with the help of ferringb, I started working on a replacement script that uses the pkgcore API. The portage-based script takes quite a while to run, and gets it wrong if there are any p.mask'd stable packages on any arch or other weird things. The pkgcore-based script actually uses the profiles properly and runs over the entire tree with all the stable profiles (according to profiles.desc) in 1m1.869s (on my box). I'll probably end up combining the scripts and doing the cleanup in python to make things easier. Wrt security updates, is it possible to tie into GLSAs so that we could automate updating packages that need it? By updating I mean adding the ebuilds and any dependencies (or dependants that might require updating.) I'm not sure this is the best idea. Caleb Cushing wrote: .. isn't it possible only to sync certain parts of the tree using excludes. maybe some additional functionality saying only sync package X for updates. Would that help in terms of having say, up to date GUI packages, or is it just easier to use overlays? Overlays are definitely best here. -- Andrew Gaffneyhttp://dev.gentoo.org/~agaffney/ Gentoo Linux Developer Installer Project Today's lesson in political correctness: Go asphyxiate on a phallus -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Re: Versioning the tree
On Wed, 2006-11-29 at 06:37 +, Steve Long wrote: snip .. The one disadvantage to my design is it needs infra. It needs it's own repository and rsync. What does that entail? Would a co-located server suffice? Sure. It would be much better to simply use what we currently have, though. Honestly, I was pursuing this with Infra a few months back, and have since dropped it, due to time constraints. I plan on picking it back up, as I said, so I don't know what is necessary at this point. Now, the release trees are non-moving. The 2007.0-release tree is *always* the 2007.0-release tree for as long as we decide to support it. Likely, this support period would begin as a single release and get extended as volunteers came around to support it. New releases get their own tree. Well count me in as a volunteer to help set this up and maintain an x86 release. I'm a pretty good coder if that helps. There wouldn't be an x86 release or anything. It would be the whole thing. All or nothing. Users who want to use the current portage tree get what they want. Users who want a more stable tree get what they want. Basically, everyone (hopefully) is happy, or at least as happy as we can feasibly make them. This all sounds great. Respect for the work you've already put in. Well, Andrew Gaffney really helped out, as he's been doing the python work to utilize portage to get the proper information to strip the tree. Since these scripts were originally written and subsequently abandoned, they've needed a little work. Andrew and I are working on this now to try to get everything back to working order. From your post we need to add: - strip all USE flags that aren't used from use.mask (per-profile) - strip all packages that aren't available from package.mask (per-profile) What language is the script implemented in? There are several scripts. They are written in either python or bash. Wrt security updates, is it possible to tie into GLSAs so that we could automate updating packages that need it? By updating I mean adding the ebuilds and any dependencies (or dependants that might require updating.) What were you expecting that we would do? Would that help in terms of having say, up to date GUI packages, or is it just easier to use overlays? I have no desire to support any packages that weren't in the original tree. If you want something we aren't providing, use an overlay. Chris Gianelloni wrote: No version changes on any packages, except those which are necessary due to a security violation, or a vulnerable package's dependencies. I could imagine a situation where a dependant package (ie one using the package updated) would also require an update. It'd be rare though, so I guess it wouldn't need automating. or a vulnerable package's dependencies Something that would be useful would be for package maintainers that wish to participate to perform further testing and validation on packages in the release snapshot prior to its final freeze. This would give us even more eyes on the tree and hopefully provide a much higher quality snapshot once we're done. It sounds like it'd also make the releases in general have better QA. What do package/ herd maintainers think? Absolutely. One thing that I am working on doing with agaffney is building more and more automated systems for doing testing. We have our Release Engineering build box doing weekly builds of all of the stages from scratch for i686. I plan on doing the same for i586/no-nptl, to cover that facet, as well as amd64. Currently, I am testing against the dev/2007.0 profile, but plan on testing against dev/2007.0/desktop and dev/2007.0/server in addition to the dev/2007.0 profile. I will also be adding a test suite on my Alpha and PPC machines at home. Aside from this, I will be running catalyst tinderbox builds on at least x86/amd64 for many packages that aren't included in the LiveCD/LiveDVD sets. At some point, I'll likely be asking for suggestions on packages to add to this testing. -- Chris Gianelloni Release Engineering Strategic Lead Alpha/AMD64/x86 Architecture Teams Games Developer/Council Member/Foundation Trustee Gentoo Foundation signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Re: Re: Versioning the tree
On Wed, 2006-11-29 at 02:21 -0500, Alec Warner wrote: The bonus is that we then need to have a fetcher that runs over the main tree and all supported release trees, and this of course increases mirror size as we now have distfiles in the tree from a year ago to support a year old release. I am unsure how much space this would take up; but this is an obstacle that must be overcome (somehow), preferably by giving infra an estimate on the space reqs. Well, we have to provide distfiles for everything we've ever released a binary of, forever, to comply with the GPL. As of this release, I kept a copy of all of the distfiles for the entire release, and can make a DVD of it, on request. This fulfills our requirements with the GPL. I would bet that for most of the tree, the size increase would be negligible, since quite a bit of the tree doesn't change every six months. Of course, the stuff that does change tends to be the larger things, like desktop environments and such. I wouldn't have a clue how to give you an idea of the space requirements until the next release comes about and I also have distfiles for that release. That will give you an idea, based only on what we release, though. -- Chris Gianelloni Release Engineering Strategic Lead Alpha/AMD64/x86 Architecture Teams Games Developer/Council Member/Foundation Trustee Gentoo Foundation signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Re: Re: Versioning the tree
On 11/29/06, Chris Gianelloni [EMAIL PROTECTED] wrote: As of this release, I kept a copy of all of the distfiles for the entire release, and can make a DVD of it, on request. This fulfills our requirements with the GPL. What are the arrangements should you go under a bus on the way home from work tonight? Best regards, Stu -- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Re: Versioning the tree
On Wed, 2006-11-29 at 16:01 +, Stuart Herbert wrote: On 11/29/06, Chris Gianelloni [EMAIL PROTECTED] wrote: As of this release, I kept a copy of all of the distfiles for the entire release, and can make a DVD of it, on request. This fulfills our requirements with the GPL. What are the arrangements should you go under a bus on the way home from work tonight? You'd like that, wouldn't you? Well, since you asked, the distfiles are stored on Release Engineering's server, under /release/2006.1/distfiles, which is accessible to anyone in Release Engineering, as well as the OSL and Infra. -- Chris Gianelloni Release Engineering Strategic Lead Alpha/AMD64/x86 Architecture Teams Games Developer/Council Member/Foundation Trustee Gentoo Foundation signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Re: Re: Versioning the tree
Stuart Herbert wrote: On 11/29/06, Chris Gianelloni [EMAIL PROTECTED] wrote: As of this release, I kept a copy of all of the distfiles for the entire release, and can make a DVD of it, on request. This fulfills our requirements with the GPL. What are the arrangements should you go under a bus on the way home from work tonight? I believe they're kept on poseidon, which many people have access to. Please stop arguing just for the sake of arguing. Are you trying to become the next ciaranm (no offense to ciaranm :P)? -- Andrew Gaffneyhttp://dev.gentoo.org/~agaffney/ Gentoo Linux Developer Installer Project Today's lesson in political correctness: Go asphyxiate on a phallus -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Re: Versioning the tree
On Wed, 29 Nov 2006 16:01:11 + Stuart Herbert [EMAIL PROTECTED] wrote: | On 11/29/06, Chris Gianelloni [EMAIL PROTECTED] wrote: | As of this release, I kept a copy of all of the distfiles for the | entire release, and can make a DVD of it, on request. This | fulfills our requirements with the GPL. | | What are the arrangements should you go under a bus on the way home | from work tonight? George and Reuben get a visit from the local police department. -- Ciaran McCreesh Mail: ciaranm at ciaranm.org Web : http://ciaranm.org/ as-needed is broken : http://ciaranm.org/show_post.pl?post_id=13 signature.asc Description: PGP signature
Re: [gentoo-dev] Re: Re: Versioning the tree
On 11/29/06, Chris Gianelloni [EMAIL PROTECTED] wrote: What are the arrangements should you go under a bus on the way home from work tonight? You'd like that, wouldn't you? That was _completely_ uncalled for. go under a bus is just a phrase that's commonly used here in the UK (because our major public transport infrastructure consists of buses and trains) when talking about business continuity of key personnel. Our liabilities under the GPL are ultimately the Foundation's responsibility. It's perfectly reasonable to want to know understand how we'd meet that liability if something did happen to you. Well, since you asked, the distfiles are stored on Release Engineering's server, under /release/2006.1/distfiles, which is accessible to anyone in Release Engineering, as well as the OSL and Infra. Thank you. Do we have backups in place covering these files? Have we tested the backups to confirm that they actually work? Best regards, Stu -- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Re: Versioning the tree
Stuart Herbert wrote: [snip] Thank you. Do we have backups in place covering these files? Have we tested the backups to confirm that they actually work? I have a couple of locations where I could store backups, depending on size and projected growth. I suppose it'll have to wait until 2007.0 though so we can actually gage it as opposed to speculating wildly. --Mike -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Re: Versioning the tree
On 11/29/06, Mike Doty [EMAIL PROTECTED] wrote: Stuart Herbert wrote: I have a couple of locations where I could store backups, depending on size and projected growth. I suppose it'll have to wait until 2007.0 though so we can actually gage it as opposed to speculating wildly. If anything happens to poseiden ... does anyone have a backup anywhere that we can use to rebuild the distfiles archive for 2006.1? What's the situation with older releases? Do we have the distfile archives for those on poseiden too? Would that give you the sizing data you need to put backups in place? Best regards, Stu -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Re: Versioning the tree
Stuart Herbert wrote: On 11/29/06, Mike Doty [EMAIL PROTECTED] wrote: Stuart Herbert wrote: I have a couple of locations where I could store backups, depending on size and projected growth. I suppose it'll have to wait until 2007.0 though so we can actually gage it as opposed to speculating wildly. If anything happens to poseiden ... does anyone have a backup anywhere that we can use to rebuild the distfiles archive for 2006.1? What's the situation with older releases? Do we have the distfile archives for those on poseiden too? Would that give you the sizing data you need to put backups in place? Chris had mentioned earlier that we started with 2006.1(iirc the discussion about GPL requirements came up between .0 and .1) having 2+ snapshots allows me to determine which server is best to use. --Mike -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] Re: Re: Versioning the tree
Ciaran McCreesh wrote: On Wed, 29 Nov 2006 16:01:11 + Stuart Herbert [EMAIL PROTECTED] wrote: | On 11/29/06, Chris Gianelloni [EMAIL PROTECTED] wrote: | As of this release, I kept a copy of all of the distfiles for the | entire release, and can make a DVD of it, on request. This | fulfills our requirements with the GPL. | | What are the arrangements should you go under a bus on the way home | from work tonight? George and Reuben get a visit from the local police department. Hahahahaha. How would they narrow it down to us though ;) -- gentoo-dev@gentoo.org mailing list
[gentoo-dev] Re: Re: Versioning the tree
Chris Gianelloni wrote: I have a script which already does several things: #1. grabs best_visible for stable on each arch #2. repeat for each SLOT #3. purge unnecessary files from FILESDIR #4. strip to only stable profiles from profiles.desc #5. purge unnecessary USE from use.local.desc and use.desc #6. strip unused eclasses #7. regenerate Manifest (via ebuild $foo digest) I had also planned on it doing the following, but they haven't been implemented just yet: - strip all USE flags that aren't used from use.mask (per-profile) - strip all packages that aren't available from package.mask (per-profile) What this gives us is a *much* smaller tree, as it only includes stable packages/versions. The smaller tree sounds great in terms of maintenance for the volunteers who maintain the release. Obviously maintaining that list is some work; predominantly watching the announcements from the security team and fixing up dependencies, and masking out new packages (for what that's worth). It could be regenerated on some or all releases, or perhaps on a yearly basis. It would also mean that versions listed there cannot be removed from the tree (until they're bumped in the list). Well, the simpler approach was to simply copy over the newer, secure ebuilds, and any required dependencies, into the release tree. There'd be no need to update mask files and such this way. It would also be generated with the releases, automatically. snip .. The one disadvantage to my design is it needs infra. It needs it's own repository and rsync. What does that entail? Would a co-located server suffice? (If it gets popular, I'd imagine those mirroring current rsyncs etc would want to mirror the releases as well.) Basically, it makes the system act a bit more like some other development models. We end up with the following: rsync://rsync.gentoo.org/gentoo-portage (this is equivalent to -current) rsync://rsync.gentoo.org/2007.0-release (this is the release tree) Now, the release trees are non-moving. The 2007.0-release tree is *always* the 2007.0-release tree for as long as we decide to support it. Likely, this support period would begin as a single release and get extended as volunteers came around to support it. New releases get their own tree. Well count me in as a volunteer to help set this up and maintain an x86 release. I'm a pretty good coder if that helps. I also started writing a tool to upgrade the distribution. The tool reads pre and post scripts for the upgrade, and performs the necessary steps. Basically, a user can dist-upgrade 2007.1 and the system would: - switch to the 2007.1 tree and emerge --sync - perform all pre steps from 2007.1 - update world + revdep-rebuild + whatever else is necessary - perform all post steps With this, there would be a special version called current which would put the user on the gentoo-portage tree, AKA the tree we all know and love/loathe. Release trees wouldn't have arch and ~arch as everything would be stable there. Testing would be done in the main tree. This, in essence, gives us testing, release candidate and stable environments, with the release trees being stable, and the current tree becoming test/release candidate. Anything marked stable in the current tree would be a candidate for stable in the next release tree. This sounds like a much more professional proposition to put to an IT exec. Users who want to use the current portage tree get what they want. Users who want a more stable tree get what they want. Basically, everyone (hopefully) is happy, or at least as happy as we can feasibly make them. This all sounds great. Respect for the work you've already put in. other post: With the release trees, essentially only those interested in supporting the tree are required to work on it. The tree is created entirely by scripts and is tested before it's released on the public. Having it automated is definitely what I wanted to know about in the original discussion. From your post we need to add: - strip all USE flags that aren't used from use.mask (per-profile) - strip all packages that aren't available from package.mask (per-profile) What language is the script implemented in? Wrt security updates, is it possible to tie into GLSAs so that we could automate updating packages that need it? By updating I mean adding the ebuilds and any dependencies (or dependants that might require updating.) Caleb Cushing wrote: .. isn't it possible only to sync certain parts of the tree using excludes. maybe some additional functionality saying only sync package X for updates. Would that help in terms of having say, up to date GUI packages, or is it just easier to use overlays? Chris Gianelloni wrote: No version changes on any packages, except those which are necessary due to a security violation, or a vulnerable package's dependencies. I could imagine a situation where a
Re: [gentoo-dev] Re: Re: Versioning the tree
Steve Long wrote: snip .. The one disadvantage to my design is it needs infra. It needs it's own repository and rsync. What does that entail? Would a co-located server suffice? (If it gets popular, I'd imagine those mirroring current rsyncs etc would want to mirror the releases as well.) One of the problems here (IIRC) is that our mirrors are quite full. Not only must you mirror the release tree (which is trivial in size) you also have to mirror the distfiles for that tree. Right now our distfiles are maintained by a wonderful python script that checks the files in SRC_URI and fetches them to the master when needed; when an ebuild is removed from CVS, the fetcher will not fetch the file and it will get removed (after two weeks, afaik). The bonus is that we then need to have a fetcher that runs over the main tree and all supported release trees, and this of course increases mirror size as we now have distfiles in the tree from a year ago to support a year old release. I am unsure how much space this would take up; but this is an obstacle that must be overcome (somehow), preferably by giving infra an estimate on the space reqs. -- gentoo-dev@gentoo.org mailing list
