Re: [gentoo-dev] upcoming mirror cleansing
On Sat, Apr 23, 2005 at 04:55:14PM -0700, Robin H. Johnson wrote: > On Sat, Apr 23, 2005 at 08:49:59AM -0500, Brian Harring wrote: > > Hola all. > [snip] > > under 'Deletions for Sunday May 01 2005' > unknown: > portage-2.0.51.20.tar.bz2 sandbox-1.2.tar.bz2 > Perhaps a major glitch here, since portage-2.0.51.20 is the latest version? Jason deployed .20 via distfiles-local to get it into the mirrors prior to adding the ebuild. So why is that file in the list of files that are marked for deletion? Becuase at the time of the run, _no_ ebuild claimed that file. We didn't push the portage ebuild into the tree until .20 tarball was in the mirrors. So it's valid. It's also the reason we wait a full week before actually removing any file from the mirror tier. > Also, will the script be re-run before actual deletions take place? (I'm > tracking down instances of nomirror that shouldn't be there). Yes. I'll be restaggering the deletions to run during the first week it's live, so you've got a week. :) What *can* be done, but requires a damn good reason, is that individual files can have their deletion times screwed with- same way I'm staggering the deletes. That said, I don't care to do it unless requested. Mentioning it, because in special cases/circumstances it may be needed (just the same as in special cases/circumstances, cvs->rsync can be turned off if someone breaks the tree). If a file is marked for deletion, you've got a week from detection to either fix the ebuild, or add an ebuild in- this however is valid. The mirror tier isn't a dumping ground :) ~brian -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] upcoming mirror cleansing
> Aside from that, kindly check the failed fetches on the list. If one > of your ebuilds is in that list, either no valid URI could be found, > or you doffed the SRC_URI for it (mirror:// w/out a mirror has been > common). The following from the list report no valid uri although they WORKFORME. app-shells/tcsh-6.14 tcsh-6.14.00.tar.gz fetcher return no uris succeeded ftp://ftp.astron.com/pub/tcsh/tcsh-6.14.00.tar.gz net-analyzer/netperf-2.4.0_rc3 netperf-2.4.0-rc3.tar.gz fetcher return no uris succeeded ftp://ftp.cup.hp.com/dist/networking/benchmarks/netperf/experimental/netperf-2.4.0-rc3.tar.gz net-wireless/aircrack-2.1 aircrack-2.1.tgz fetcher return no uris succeeded http://www.cr0.net:8040/code/network/aircrack-2.1.tgz Problem with ftp/no port 80 parsing? -- Daniel Black <[EMAIL PROTECTED]> Gentoo Crypto/PPC/dev-embedded/Forensics/NetMon pgpqvPbgGVd4T.pgp Description: PGP signature
Re: [gentoo-dev] upcoming mirror cleansing
On Sat, Apr 23, 2005 at 08:49:59AM -0500, Brian Harring wrote: > Hola all. [snip] under 'Deletions for Sunday May 01 2005' unknown: portage-2.0.51.20.tar.bz2 sandbox-1.2.tar.bz2 Perhaps a major glitch here, since portage-2.0.51.20 is the latest version? Also, will the script be re-run before actual deletions take place? (I'm tracking down instances of nomirror that shouldn't be there). -- Robin Hugh Johnson E-Mail : [EMAIL PROTECTED] Home Page : http://www.orbis-terrarum.net/?l=people.robbat2 ICQ# : 30269588 or 41961639 GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgpjnUb8tjjdg.pgp Description: PGP signature
Re: [gentoo-dev] upcoming mirror cleansing
On Sat, 2005-23-04 at 23:29 +0100, Ciaran McCreesh wrote: > On Sat, 23 Apr 2005 08:49:59 -0500 Brian Harring <[EMAIL PROTECTED]> > wrote: > | Why this matters- around 10,000 files out of 28,600 files will be > | removed from the mirrors network. > > I just had a random thought. Have our GLEP 19 people thought about this > at all? Ideally the clearing script would also very that the file isnt needed by something in the GLEP19 tree, right ? But since there is no official GLEP19 tree yet... -- Olivier Crête [EMAIL PROTECTED] x86 Security Liaison signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] upcoming mirror cleansing
On Sat, 23 Apr 2005 08:49:59 -0500 Brian Harring <[EMAIL PROTECTED]> wrote: | Why this matters- around 10,000 files out of 28,600 files will be | removed from the mirrors network. I just had a random thought. Have our GLEP 19 people thought about this at all? -- Ciaran McCreesh : Gentoo Developer (Vim, Fluxbox, shell tools) Mail: ciaranm at gentoo.org Web : http://dev.gentoo.org/~ciaranm pgpCkWw0RIK9j.pgp Description: PGP signature
Re: [gentoo-dev] upcoming mirror cleansing
On Sat, Apr 23, 2005 at 12:18:30PM -0700, Donnie Berkholz wrote: > Brian Harring wrote: > > A quicky report of flies that'll be ixnayed is available at > > http://dev.gentoo.org/~ferringb/failure.xml > > Thanks for making that page! I just searched it and found two packages I > cared about with wrong SRC_URI's, fixed one and filed a bug for the other. > > It would be even cooler if it matched stuff up with metadata.xml! Intending on pulling herd/maintainer information and mapping it in. That said, my intention for it is to bind it to the packages.g.o database. I need persistance, since it only notes when a file is orphaned; need to be able to look at a general tree history, and know what ebuilds owned a file, thus getting to the metadata.xml. I could write this myself, track the data, but I'm not writing Yet Another [EMAIL PROTECTED] Local TreeDB. Would rather unify the script (and a few other ideas/scripts I'm kicking around) against a generalized packages db... So yeah, moving in that direction :) ~brian -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] upcoming mirror cleansing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Brian Harring wrote: > A quicky report of flies that'll be ixnayed is available at > http://dev.gentoo.org/~ferringb/failure.xml Thanks for making that page! I just searched it and found two packages I cared about with wrong SRC_URI's, fixed one and filed a bug for the other. It would be even cooler if it matched stuff up with metadata.xml! Thanks, Donnie -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFCap+GXVaO67S1rtsRAsFqAJ9YazXDJ6iNQ0+Ovy9JoGDjR5UqKwCePEcd cc5x8VYmEPFxgO9YNsu0nVk= =k96f -END PGP SIGNATURE- -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] upcoming mirror cleansing
On Sat, Apr 23, 2005 at 05:54:40PM +0300, Alin Nastac wrote: > Brian Harring wrote: > > >Why this matters- around 10,000 files out of 28,600 files will be > >removed from the mirrors network. Either > > > >A) no ebuild claims that distfile. it's orphaned on our mirrors > >B) RESTRICT="fetch" is set for the ebuild. We don't mirror those > > files. > >C) RESTRICT="mirror" is set for the ebuild. Again, we don't mirror > > those files, in this case we defer to another network (I don't make > > the rules, that's just how it's done) > > > > > > > You should not erase files newer than an arbitrary amount of time (a > week maybe?). > Don't forget about dev.g.o:/space/distfiles-local; a dev first put the > tarball in that folder _then_ submit the ebuild who use it. Files that are orphaned from ebuilds (this includes distfiles-local uploaded files) are marked for death, and have a week till they're removed from the mirrors. They're stored for an additional 2 weeks, then deleted. There's a bit more to it then that, but the short version is that there are reasaonable delays built into the auto cleansing. ~brian -- gentoo-dev@gentoo.org mailing list
Re: [gentoo-dev] upcoming mirror cleansing
Brian Harring wrote: >Why this matters- around 10,000 files out of 28,600 files will be >removed from the mirrors network. Either > >A) no ebuild claims that distfile. it's orphaned on our mirrors >B) RESTRICT="fetch" is set for the ebuild. We don't mirror those > files. >C) RESTRICT="mirror" is set for the ebuild. Again, we don't mirror > those files, in this case we defer to another network (I don't make > the rules, that's just how it's done) > > > You should not erase files newer than an arbitrary amount of time (a week maybe?). Don't forget about dev.g.o:/space/distfiles-local; a dev first put the tarball in that folder _then_ submit the ebuild who use it. signature.asc Description: OpenPGP digital signature
[gentoo-dev] upcoming mirror cleansing
Hola all. So mirror-dist is ready to go, infra side of it being set for a final testing run then switching it live if things go fine after next weekend. Why this matters- around 10,000 files out of 28,600 files will be removed from the mirrors network. Either A) no ebuild claims that distfile. it's orphaned on our mirrors B) RESTRICT="fetch" is set for the ebuild. We don't mirror those files. C) RESTRICT="mirror" is set for the ebuild. Again, we don't mirror those files, in this case we defer to another network (I don't make the rules, that's just how it's done) Bugs may exist, but I'd suggest you take a hard look at the reasons above to verify it's actually a bug. Should you find one, kindly email me. A quicky report of flies that'll be ixnayed is available at http://dev.gentoo.org/~ferringb/failure.xml Sorry about the size, it'll come down after the 10,000 deletion is swallowed. The deletion dates should be ignored. They're accurate for the testing, but aren't the actual dates when the files will be removed from the mirrors (the deletion dates will be off till this is live). Aside from that, kindly check the failed fetches on the list. If one of your ebuilds is in that list, either no valid URI could be found, or you doffed the SRC_URI for it (mirror:// w/out a mirror has been common). That's all. Eat the pudding. ~brian pgpNXLf8Jkird.pgp Description: PGP signature
