Re: [gentoo-dev] Package ranking by number of ebuilds on the portage tree
On Fri, Oct 26, 2012 at 1:16 PM, Theo Chatzimichos wrote: > On Fri, Oct 26, 2012 at 2:02 PM, Francisco Blas Izquierdo Riera > (klondike) wrote: >> So I have been doing some bash scripting out of some comment in a >> conversation to count (and rank) the packages by the number of ebuilds >> they have (and thus of versions of said package). The results can be >> seen at http://dev.gentoo.org/~klondike/ebuildrank.txt and if there is >> interest I can try to automate the generation of the ranking daily >> (though I'd like infra's comments on that). > > We can put the script in qa-reports.g.o I think that kind of stats would be really easy to generate within the new p.g.o. Probably with a single database request. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] SRC_URI in metadata.xml
On Fri, Aug 10, 2012 at 10:12 PM, Diego Elio Pettenò wrote: > On 10/08/2012 13:05, Corentin Chary wrote: >> Right, our proposal is not here to replace SRC_URI, it's here to fix >> the cases where SRC_URI can't be sanely used to guess new upstream >> versions (strange mangling rules, unbrowsable directories, etc...). > > Yes I guess Jeroen was just saying why we shouldn't abandon it as Gilles > proposed. > > FWIW for the rest it feels right to me. Although this starts to add up > to the reasons why at least metadata.xml should be validated by schema, > and not DTD. Maybe .. We plan to use http://euscan.iksaif.net";> to avoid editing metadata.dtd (for now). What do you think about format propositions ? Current format looks like what was given in the examples, but mgorny feels that something more xmlish would be better. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] SRC_URI in metadata.xml
On Fri, Aug 10, 2012 at 4:21 PM, Jeroen Roovers wrote: > On Fri, 10 Aug 2012 14:03:23 +0200 > Gilles Dartiguelongue wrote: > >> Since you are proposing this, a side question is: >> Why should we write SRC_URI in ebuilds if that info is now available >> in metadata.xml ? (granted that we might still want to keep >> over-riding this information in ebuilds) > > 1) The information in metadata.xml is inaccurate, it's a hint. When it >fails, nothing of value is lost since the ebuild (supposedly) has >what you want. > 2) SRC_URI is precise. > 3) SRC_URI can change over time, and across versions (even with all the >variables in place). > 4) Backward compatibility. > 5) The inversion of your question: Why should we start handling SRC_URI >outside ebuilds and eclasses? Or, how would that be practical, >advantageous, an improvement on the current situation. Right, our proposal is not here to replace SRC_URI, it's here to fix the cases where SRC_URI can't be sanely used to guess new upstream versions (strange mangling rules, unbrowsable directories, etc...). -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] [RFC] euscan: Need to add more upstream info in metadata.xml
On Fri, Aug 10, 2012 at 2:03 PM, Gilles Dartiguelongue wrote: > Having done some debian packaging for work, I find watch files from > debian really helpful. Changing the format to a XML compatible one does > not seem like a hard work so I'll probably leave that up for others to > discuss. > > Since you are proposing this, a side question is: > Why should we write SRC_URI in ebuilds if that info is now available in > metadata.xml ? (granted that we might still want to keep over-riding > this information in ebuilds) It's not (only) SRC_URI, sometime it's completly different, sometimes would contain only versionmangle since SRC_URI contains enought informations for euscan... SRC_URI serves a totally different purpose :). -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] euscan GSoC project - requesting feedback
On Fri, Jun 29, 2012 at 8:53 AM, Michał Górny wrote: > On Fri, 29 Jun 2012 14:07:58 +1200 > Kent Fredric wrote: > >> On 27 June 2012 19:51, Federico "fox" Scrinzi wrote: >> > The main question is: what would you like to have on this dashboard? >> > Currently (in the development version) there's the possibility to >> > login and watch/unwatch packages/categories/herds/... and see the >> > watched stuff in the account dashboard. We're planning on >> > implementing a weekly(?) custom newsletter based on the packages >> > you're watching, which features would you like? >> > >> > The project repo for the GSoC is here: >> > https://github.com/volpino/euscan >> > >> > Thanks! >> > >> >> For the most part it seems to get upstream / portage versioning right, >> but occasionally you get miss-matches for some reason. >> >> It would be nice to allow to provide some mapping mechanism that >> existed on the overlay itself to inform euscan how to map upstream >> versions to downstream ones, but implementing that would be far too >> complex I feel. >> >> Instead, it would be nice to have a mechanism in the interface to set >> a "Upstream version is" value for each package if euscan can't tell. >> >> Ie: >> >> http://euscan.iksaif.net/package/dev-perl/HTML-TreeBuilder-LibXML/ >> >> Upstream is 0.71 , portage is ( normalised ) to 0.710.0 , and these >> are in fact the same version. So in 0.710.0 , it would be nice to be >> able to set the upstream version manually to 0.71 so that euscan no >> longer reported it as outdated. > > I think we could actually handle perl pretty easily. I believe euscan > will start using CPAN API to check the package versions, and we can > embed the normalization there. It's already the case: https://github.com/iksaif/euscan/blob/master/pym/euscan/handlers/cpan.py but my mangling functions are probably broken in some cases. If somebody with a better knowledge of CPAN versionning scheme could fix them it would be great ! Thanks,
Re: [gentoo-dev] euscan GSoC project - requesting feedback
On Fri, Jun 29, 2012 at 4:07 AM, Kent Fredric wrote: > On 27 June 2012 19:51, Federico "fox" Scrinzi wrote: >> The main question is: what would you like to have on this dashboard? >> Currently (in the development version) there's the possibility to login >> and watch/unwatch packages/categories/herds/... and see the watched >> stuff in the account dashboard. We're planning on implementing a >> weekly(?) custom newsletter based on the packages you're watching, which >> features would you like? >> >> The project repo for the GSoC is here: https://github.com/volpino/euscan >> >> Thanks! >> > > For the most part it seems to get upstream / portage versioning right, > but occasionally you get miss-matches for some reason. > > It would be nice to allow to provide some mapping mechanism that > existed on the overlay itself to inform euscan how to map upstream > versions to downstream ones, but implementing that would be far too > complex I feel. > > Instead, it would be nice to have a mechanism in the interface to set > a "Upstream version is" value for each package if euscan can't tell. > > Ie: > > http://euscan.iksaif.net/package/dev-perl/HTML-TreeBuilder-LibXML/ > > Upstream is 0.71 , portage is ( normalised ) to 0.710.0 , and these > are in fact the same version. So in 0.710.0 , it would be nice to be > able to set the upstream version manually to 0.71 so that euscan no > longer reported it as outdated. > > http://euscan.iksaif.net/package/dev-perl/Authen-SASL-Cyrus-server/ > > 0.13 == 0.13-serve > > http://euscan.iksaif.net/package/dev-perl/Module-Extract-Namespaces/ > > 0.140.200_rc == 0.14_0.2 > > http://euscan.iksaif.net/package/dev-perl/Math-BaseCnv/ > http://euscan.iksaif.net/package/dev-perl/XML-Tidy/ > > 1.8 == 1.8.B59BrZ > 1.8 == 1.8.B2AMvd > > ( Upstream for those 2 packages have a versioning scheme tantamount to > intolerable cruelty. > https://rt.cpan.org/Public/Bug/Display.html?id=60275 ) > > http://euscan.iksaif.net/package/dev-perl/Perl-Critic-Moose/ > 0.999.2_rc == 0.999._002 > > http://euscan.iksaif.net/package/dev-perl/aliased/ > 0.300.100_rc == 0.30_0.1 > > http://euscan.iksaif.net/package/dev-perl/EV/ > 4.110.0 == 4.11 > > > -- > Kent > > perl -e "print substr( \"edrgmaM SPA NOcomil.ic\\@tfrken\", \$_ * 3, > 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );" > > http://kent-fredric.fox.geek.nz > Something that could help with that: we plan to add something like debian/watch in metadata.xml. This should allow to specify a regexp to mangle versions (sed like syntax: /match/replace/). For some specific packages, we also already have specific handlers (cpan, php, pypi, etc...) which have specific version mangling functions.
Re: [gentoo-dev] remote-id cpan-module
On Sat, Jun 2, 2012 at 7:11 PM, Torsten Veller wrote: > * Corentin Chary : >> On Thu, May 17, 2012 at 2:02 AM, Kent Fredric wrote: >> > On 13 May 2012 07:43, Torsten Veller wrote: >> >> It doesn't even list "Moose" for Moose? >> > >> > Its probably falling outside the initial 10 results, I forgot it did that, >> > >> >> 02packages.details.txt.gz lists 72 package names for Moose-2.0602. >> >> >> > >> > Need to bolt on a { "size": 100 } to the query to expand how may >> > results it will return. >> >> Updated remotesid.py to use that, correctly add Moose in the diff now ! > > metadata.dtd was updated per bug #406287 and it contains the cpan-module > remote-id. > > The current patch for dev-perl/* is roughly 800k big: > http://dev.gentoo.org/~tove/files/devperlremoteids.patch > > I am going to update the files in the next days. > Now it would be a good time to voice your concerns. > > -- > Thanks > Torsten > I pushed some fixes in remoteids.py, it now have the ability to remove dead remote-id. You should regenerate your patch, or use this one: http://xf.iksaif.net/dev/gentoo/remoteids/dev-perl.diff Note that I just made a small change that will make the remote-id sorted (which is great for cpan-module), if you want it, just regenerate a patch with: eix --only-names -C dev-perl | ./remoteids.py --diff --check-all Thanks, -- Corentin Chary
Re: [gentoo-dev] RFC: Add new remote-id types in metadata.dtd
On Sat, Jun 2, 2012 at 7:30 PM, Michael Weber wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Is there any way to verify the data? > > What programs/scripts use these fields btw? euscan will soon, and I guess p.g.o will too. > I could imagine a test like (i.e. type="sourceforge">$a ) > does http://www.fs.net/$a exist. > > Michael That could probably be done in remoteids.py
Re: [gentoo-dev] Re: RFC: Add new remote-id types in metadata.dtd
On Thu, May 17, 2012 at 2:02 AM, Kent Fredric wrote: > On 13 May 2012 07:43, Torsten Veller wrote: >> * Corentin Chary : >>> On Sat, Apr 21, 2012 at 03:33:18PM +1200, Kent Fredric wrote: >>> > { "term": { "status":"latest"} }, >>> > { "term": { >>> > "module.authorized":"true"}} >> >> What does this mean? >> - "latest"? this term looks like maintenance work. >> - what is "authorized"? > > "latest" means that it will fetch metadata for whatever is deemed the > most recent non-dev release, which is really the only sane option to > go for if you want a list of modules that currently pertain to the > distribution. You could request *all* releases and then find a union > of elements ... but that would be both erroneous and very time > consuming. > >> It doesn't even list "Moose" for Moose? > > Its probably falling outside the initial 10 results, I forgot it did that, > >> 02packages.details.txt.gz lists 72 package names for Moose-2.0602. >> > > Need to bolt on a { "size": 100 } to the query to expand how may > results it will return. Updated remotesid.py to use that, correctly add Moose in the diff now ! > curl -XPOST 'http://api.metacpan.org/module/_search' -d ' > { > "fields": [ > "module.name", > "release" > ], > "query": { > "constant_score": { > "filter" : { > "and" : [ > { "term": { "distribution":"Moose" } }, > { "term": { "status":"latest"} }, > { "term": { > "mime":"text/x-script.perl-module"}}, > { "term": { "indexed":"true"}}, > { "term": { "module.authorized":"true"}} > ] > > } > } > }, > "size": 100 > } > > ^ that | grep module.name | wc -l # 83 > > -- > Kent > > perl -e "print substr( \"edrgmaM SPA NOcomil.ic\\@tfrken\", \$_ * 3, > 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );" > > http://kent-fredric.fox.geek.nz >
Re: [gentoo-dev] [RFC] New third party mirrors
On Thu, Apr 26, 2012 at 8:41 PM, Michał Górny wrote: > On Thu, 26 Apr 2012 10:21:36 +0200 > Corentin Chary wrote: > >> Second solution: >> github http://cloud.github.com/downloads >> github-bad-uris -http://github.com/downloads/ > -https://github.com/downloads/ >> >> The good thing with the first one is that it would allow repoman to >> outputs something like "you should use 'mirror://github'". > > Well, we could decide on something common and special like: > > github:bad-uris http://. > > And then let repoman suggest using mirror with ':bad-uris' stripped. Works for me. What would be the next step to push this ? -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] [RFC] New third party mirrors
On Thu, Apr 26, 2012 at 10:07 AM, Zac Medico wrote: > On 04/26/2012 01:03 AM, Corentin Chary wrote: >> On Thu, Apr 26, 2012 at 9:57 AM, Zac Medico wrote: >>> On 04/26/2012 12:30 AM, Corentin Chary wrote: >>>> On Wed, Apr 25, 2012 at 6:41 PM, Michał Górny wrote: >>>>> On Wed, 25 Apr 2012 09:16:05 +0200 >>>>> Corentin Chary wrote: >>>>> >>>>>> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny >>>>>> wrote: >>>>>>> On Tue, 24 Apr 2012 16:19:11 + >>>>>>> "Robin H. Johnson" wrote: >>>>>>> >>>>>>>> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote: >>>>>>>>>>> $ ./mirrors.py --all --count >>>>>>>>>>> 297 ?? ?? http://pear.php.net >>>>>>>>>>> 297 ?? ?? http://pear.php.net/get >>>>>>>>>>> 88 ?? ?? ??http://pecl.php.net >>>>>>>>>>> 88 ?? ?? ??http://pecl.php.net/get >>>>>>>>>> These are already mirror bouncers. If you visit the above, >>>>>>>>>> you'll get the closest mirror for downloading. >>>>>>>>> And since there is already ~10 "mirrors" with only one actual >>>>>>>>> backend, should they go to thirdpartymirrors or not ? If not, >>>>>>>>> what about this pseudo-mirrors already present in >>>>>>>>> thirdpartymirrors ? >>>>>>>> I think we should add the pseudo-mirrors, but explicitly mark them >>>>>>>> as such in the file, so that they don't get duplicate entries >>>>>>>> added (eg adding us.pear, de.pear and the pear bouncer is bad. >>>>>>>> Should have just the bouncer). >>>>>>> >>>>>>> It'd be great if we could add some kind of additional mirror >>>>>>> entries, which would be used by repoman to signal missing mirror:// >>>>>>> entries but won't be used for downloads. >>>>>> >>>>>> Yep, we could put that in it too: >>>>>> github http://github.com/downloads/ >>>>>> https://github.com/downloads/ >>>>> >>>>> Per spec, portage can choose a random mirror of the list. If we put >>>>> entries like that, these two will be equally possible as the preferred >>>>> cloud. URL -- while they redirect one to another. >>>>> >>>>> We might decide on some common syntax like preceding all extra entries >>>>> with '-' but I don't want to be the one deciding here. >>>> >>>> I checked, and current portage code already handle entries starting >>>> with a - gracefully thanks to stack_dictlist (removing them from the >>>> list of mirrors). >>> >>> That means repoman will ignore them too. If you want existing versions >>> of repoman to check for those paths in SRC_URI, you can add a line like >>> this to thirdpartymirrors: >>> >>> github-bad-urls http://github.com/downloads/ https://github.com/downloads/ >> >> Hum, I checked repoman source code, and I didn't find where it checks >> if SRC_URI matches something in thirdpartymirror. Any hint ? > > Search for SRC_URI.mirror in /usr/bin/repoman. Arg.. ok, I only looked in pym/repoman/. So two solutions here: First one: github http://cloud.github.com/downloads -http://github.com/downloads/ -https://github.com/downloads/ + a small patch that would allow repoman to do something like settings.thirdpartymirrors(keep_bad_uris=True) in order to keep uris starting with a - in the list. Second solution: github http://cloud.github.com/downloads github-bad-uris -http://github.com/downloads/ -https://github.com/downloads/ The good thing with the first one is that it would allow repoman to outputs something like "you should use 'mirror://github'". -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] [RFC] New third party mirrors
On Thu, Apr 26, 2012 at 9:57 AM, Zac Medico wrote: > On 04/26/2012 12:30 AM, Corentin Chary wrote: >> On Wed, Apr 25, 2012 at 6:41 PM, Michał Górny wrote: >>> On Wed, 25 Apr 2012 09:16:05 +0200 >>> Corentin Chary wrote: >>> >>>> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny >>>> wrote: >>>>> On Tue, 24 Apr 2012 16:19:11 + >>>>> "Robin H. Johnson" wrote: >>>>> >>>>>> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote: >>>>>>>>> $ ./mirrors.py --all --count >>>>>>>>> 297 ?? ?? http://pear.php.net >>>>>>>>> 297 ?? ?? http://pear.php.net/get >>>>>>>>> 88 ?? ?? ??http://pecl.php.net >>>>>>>>> 88 ?? ?? ??http://pecl.php.net/get >>>>>>>> These are already mirror bouncers. If you visit the above, >>>>>>>> you'll get the closest mirror for downloading. >>>>>>> And since there is already ~10 "mirrors" with only one actual >>>>>>> backend, should they go to thirdpartymirrors or not ? If not, >>>>>>> what about this pseudo-mirrors already present in >>>>>>> thirdpartymirrors ? >>>>>> I think we should add the pseudo-mirrors, but explicitly mark them >>>>>> as such in the file, so that they don't get duplicate entries >>>>>> added (eg adding us.pear, de.pear and the pear bouncer is bad. >>>>>> Should have just the bouncer). >>>>> >>>>> It'd be great if we could add some kind of additional mirror >>>>> entries, which would be used by repoman to signal missing mirror:// >>>>> entries but won't be used for downloads. >>>> >>>> Yep, we could put that in it too: >>>> github http://github.com/downloads/ >>>> https://github.com/downloads/ >>> >>> Per spec, portage can choose a random mirror of the list. If we put >>> entries like that, these two will be equally possible as the preferred >>> cloud. URL -- while they redirect one to another. >>> >>> We might decide on some common syntax like preceding all extra entries >>> with '-' but I don't want to be the one deciding here. >> >> I checked, and current portage code already handle entries starting >> with a - gracefully thanks to stack_dictlist (removing them from the >> list of mirrors). > > That means repoman will ignore them too. If you want existing versions > of repoman to check for those paths in SRC_URI, you can add a line like > this to thirdpartymirrors: > > github-bad-urls http://github.com/downloads/ https://github.com/downloads/ Hum, I checked repoman source code, and I didn't find where it checks if SRC_URI matches something in thirdpartymirror. Any hint ? -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] [RFC] New third party mirrors
On Wed, Apr 25, 2012 at 6:41 PM, Michał Górny wrote: > On Wed, 25 Apr 2012 09:16:05 +0200 > Corentin Chary wrote: > >> On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny >> wrote: >> > On Tue, 24 Apr 2012 16:19:11 + >> > "Robin H. Johnson" wrote: >> > >> >> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote: >> >> > >> $ ./mirrors.py --all --count >> >> > >> 297 ?? ?? http://pear.php.net >> >> > >> 297 ?? ?? http://pear.php.net/get >> >> > >> 88 ?? ?? ??http://pecl.php.net >> >> > >> 88 ?? ?? ??http://pecl.php.net/get >> >> > > These are already mirror bouncers. If you visit the above, >> >> > > you'll get the closest mirror for downloading. >> >> > And since there is already ~10 "mirrors" with only one actual >> >> > backend, should they go to thirdpartymirrors or not ? If not, >> >> > what about this pseudo-mirrors already present in >> >> > thirdpartymirrors ? >> >> I think we should add the pseudo-mirrors, but explicitly mark them >> >> as such in the file, so that they don't get duplicate entries >> >> added (eg adding us.pear, de.pear and the pear bouncer is bad. >> >> Should have just the bouncer). >> > >> > It'd be great if we could add some kind of additional mirror >> > entries, which would be used by repoman to signal missing mirror:// >> > entries but won't be used for downloads. >> >> Yep, we could put that in it too: >> github http://github.com/downloads/ >> https://github.com/downloads/ > > Per spec, portage can choose a random mirror of the list. If we put > entries like that, these two will be equally possible as the preferred > cloud. URL -- while they redirect one to another. > > We might decide on some common syntax like preceding all extra entries > with '-' but I don't want to be the one deciding here. I checked, and current portage code already handle entries starting with a - gracefully thanks to stack_dictlist (removing them from the list of mirrors). -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] [RFC] New third party mirrors
On Tue, Apr 24, 2012 at 6:38 PM, Michał Górny wrote: > On Tue, 24 Apr 2012 16:19:11 + > "Robin H. Johnson" wrote: > >> On Tue, Apr 24, 2012 at 04:50:49PM +0200, Corentin Chary wrote: >> > >> $ ./mirrors.py --all --count >> > >> 297 ?? ?? http://pear.php.net >> > >> 297 ?? ?? http://pear.php.net/get >> > >> 88 ?? ?? ??http://pecl.php.net >> > >> 88 ?? ?? ??http://pecl.php.net/get >> > > These are already mirror bouncers. If you visit the above, you'll >> > > get the closest mirror for downloading. >> > And since there is already ~10 "mirrors" with only one actual >> > backend, should they go to thirdpartymirrors or not ? If not, what >> > about this pseudo-mirrors already present in thirdpartymirrors ? >> I think we should add the pseudo-mirrors, but explicitly mark them as >> such in the file, so that they don't get duplicate entries added (eg >> adding us.pear, de.pear and the pear bouncer is bad. Should have just >> the bouncer). > > It'd be great if we could add some kind of additional mirror entries, > which would be used by repoman to signal missing mirror:// entries but > won't be used for downloads. Yep, we could put that in it too: github http://github.com/downloads/ https://github.com/downloads/ nongnu http://download.savannah.nongnu.org/releases/ (I'm not really sure for nongnu). -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] [RFC] New third party mirrors
On Tue, Apr 24, 2012 at 4:35 PM, Robin H. Johnson wrote: > On Tue, Apr 24, 2012 at 03:10:00PM +0200, Corentin Chary wrote: >> Hello, >> >> As part of my portage-janitor scripts >> (https://github.com/iksaif/portage-janitor) >> I created a tool that counts URIs prefixes and sort them. >> Here is the result of this script on the gentoo-x86 tree: >> >> $ ./mirrors.py --all --count >> 297 http://pear.php.net >> 297 http://pear.php.net/get >> 88 http://pecl.php.net >> 88 http://pecl.php.net/get > These are already mirror bouncers. If you visit the above, you'll get > the closest mirror for downloading. And since there is already ~10 "mirrors" with only one actual backend, should they go to thirdpartymirrors or not ? If not, what about this pseudo-mirrors already present in thirdpartymirrors ? -- Corentin Chary http://xf.iksaif.net
[gentoo-dev] [RFC] New third party mirrors
Hello, As part of my portage-janitor scripts (https://github.com/iksaif/portage-janitor) I created a tool that counts URIs prefixes and sort them. Here is the result of this script on the gentoo-x86 tree: $ ./mirrors.py --all --count 1262http://dev.gentoo.org 483 http://oss.tresys.com 482 http://oss.tresys.com/files 482 http://oss.tresys.com/files/refpolicy 473 http://xorg.freedesktop.org 473 http://xorg.freedesktop.org/releases 473 http://xorg.freedesktop.org/releases/individual 313 http://dev.gentoo.org/~swift 313 http://dev.gentoo.org/~swift/patches 309 http://dev.gentoo.org/~swift/patches/selinux-base-policy 297 http://pear.php.net 297 http://pear.php.net/get 296 http://hackage.haskell.org/packages 296 http://hackage.haskell.org/packages/archive 296 http://hackage.haskell.org 209 http://launchpad.net 206 http://ftp.xemacs.org 197 https://github.com 196 http://ftp.xemacs.org/pub 196 http://ftp.xemacs.org/pub/xemacs 193 http://ftp.xemacs.org/pub/xemacs/packages 179 http://gstreamer.freedesktop.org 179 http://gstreamer.freedesktop.org/src 176 http://github.com 175 http://linuxgazette.net/ftpfiles 175 http://linuxgazette.net 139 http://xorg.freedesktop.org/releases/individual/app 130 http://pear.horde.org/get 130 http://pear.horde.org 126 http://xorg.freedesktop.org/releases/individual/driver 115 http://dev.gentoo.org/~vapier 112 http://dev.gentoo.org/~vapier/dist 96 ftp://sources.redhat.com/pub 96 ftp://sources.redhat.com 94 http://savannah.nongnu.org/download 94 http://savannah.nongnu.org 92 http://xorg.freedesktop.org/releases/individual/lib 88 http://pecl.php.net 88 http://download.gna.org 88 http://pecl.php.net/get 84 http://dev.gentoo.org/~caster/distfiles 84 http://dev.gentoo.org/~caster 80 https://fedorahosted.org 74 http://get.qt.nokia.com 72 http://components.ez.no/get 72 http://components.ez.no 71 http://get.qt.nokia.com/qt 70 http://get.qt.nokia.com/qt/source 67 http://www.phrack.org/archives 67 http://www.phrack.org/archives/tgz And here are some mirrors that can be deduced from that (except freebsd-jp which I found reading the linphone ebuild): freebsd-jp ftp://ports.jp.FreeBSD.org/pub/FreeBSD-jp/ ftp://ftp.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp1.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp2.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp3.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp4.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp5.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp6.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp7.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp8.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp9.jp.freebsd.org/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/FreeBSD-jp/ ftp://ftp.ics.es.osaka-u.ac.jp/pub/mirrors/FreeBSD-jp/ ftp://ftp.allbsd.org/pub/FreeBSD-jp/ ftp://ftp.nagaokaut.ac.jp/pub/FreeBSD-jp/ ftp://ftp.nara.wide.ad.jp/pub/FreeBSD-jp/ ftp://ftp.jaist.ac.jp/pub/FreeBSD-jp/ pearhttp://pear.php.net/ http://de.pear.php.net/ http://us.pear.php.net/ redhat ftp://sources.redhat.com/pub/ ftp://mirrors.kernel.org/sources.redhat.com/ ftp://mirrors.kernel.org/sources.redhat.com/ http://mirrors.kernel.org/sources.redhat.com/ http://sources-redhat.mirrors.airband.net/ ftp://gd.tuwien.ac.at/gnu/sourceware/ http://gd.tuwien.ac.at/gnu/sourceware/ ftp://ftp.gwdg.de/pub/linux/sources.redhat.com/ http://ftp.gwdg.de/pub/linux/sources.redhat.com/ ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/ http://ftp-stud.fht-esslingen.de/pub/Mirrors/sourceware.org/ http://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://bo.mirror.garr.it/mirrors/sourceware.org/ ftp://ftp.mirrorservice.org/sites/sources.redhat.com/pub/ http://www.mirrorservice.org/sites/sources.redhat.com/pub/ xemacs http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/ xorghttp://xorg.freedesktop.org/pub/ http://ftp.x.org/pub/ That would affect ~1000 URIs. Unfortunatly I didn't found any mirror for: - pecl.php.net - hackage.haskell.org - get.qt.nokia.com - launchpad But maybe we should still add them to thirdpartymirrors ? Look at 3dgamers, beyondunreal, bitbucket, github, gnome, kernel, xfce, vdrfiles and vdr-developerorg: they all only have one mirror ! For those afraid to add these mirrors and patch all affected ebuilds: no worry, mirrors.py can do it for you, and I already generated splitted patches generated from various thirdpartymirrors here: http://xf.iksaif.net/dev/gentoo/ Note that as part of https://bugs.gentoo.org/show_bug.cgi?id=405533 mgorny already commited a massive amount of patches adding mirror:// when the URI prefix was already present in thirdpartymirrors. Thanks, -- Corentin Chary http://xf.iksaif.net/ pgp2V9f9HGBZb.pgp Description: PGP signature
Re: [gentoo-dev] RFC: Add new remote-id types in metadata.dtd
On Sat, Apr 21, 2012 at 11:32:26AM +0200, Michał Górny wrote: > On Thu, 19 Apr 2012 21:32:49 +0200 > Corentin Chary wrote: > > > On Thu, Apr 19, 2012 at 6:54 PM, Michał Górny > > wrote: > > > On Thu, 19 Apr 2012 17:31:11 +0200 > > > Corentin Chary wrote: > > > > > >> - > >> (freshmeat|sourceforge|sourceforge-jp|cpan|vim|google-code|ctan|pypi|rubyforge|cran) > > >> #REQUIRED> > > >> + > >> (freshmeat|sourceforge|sourceforge-jp|cpan|vim|google-code|ctan|pypi|rubyforge|cran|rubygems|github|gitorious|pecl|pear|bitbucket) > > >> #REQUIRED> > > > > > > Wouldn't it be better if we kept them sorted? > > > > There were not, so I just appended them, but I can provide another > > patch sorting them too if preferred. > > I'll appreciate if you could do that, along with that cpan-module > addition, and if noone objects, I'll just commit that. > > -- > Best regards, > Michał Górny --- a/metadata/dtd/metadata.dtd 2010-03-02 18:52:11.0 +0100 +++ b/metadata/dtd/metadata.dtd 2012-04-21 13:09:52.557223402 +0200 @@ -61,7 +61,7 @@ - + -- Corentin Chary http://xf.iksaif.net/ pgp65RpvYD2Le.pgp Description: PGP signature
Re: [gentoo-dev] RFC: Add new remote-id types in metadata.dtd
On Sat, Apr 21, 2012 at 03:33:18PM +1200, Kent Fredric wrote: > On 21 April 2012 08:33, Corentin Chary wrote: > > On Fri, Apr 20, 2012 at 9:35 PM, Kent Fredric wrote: > >> On 21 April 2012 01:34, Corentin Chary wrote: > >>> Yeah, not very important, but seems to work with this patch: > >>> https://github.com/iksaif/portage-janitor/commit/972aff94744741e34e99f917337430d245883c48 > > http://api.metacpan.org/release/Scalar-List-Utils > > Will be far better an option than parsing HTML =) > > And even better: > > curl -XPOST 'http://api.metacpan.org/module/_search' -d ' > { > "fields": [ > "module.name", > "release" > ], > "query": { > "constant_score": { > "filter" : { > "and" : [ > { "term": { > "distribution":"Scalar-List-Utils" } }, > { "term": { "status":"latest"} }, > { "term": { > "mime":"text/x-script.perl-module"}}, > { "term": { "indexed":"true"}}, > { "term": { "module.authorized":"true"}} > ] > } > } > } > } > ' > It is ! $ ./remoteids.py --diff WWW-Bugzilla Moose bioperl Scalar-List-Utils libwww-perl --- a/dev-perl/WWW-Bugzilla/metadata.xml +++ b/dev-perl/WWW-Bugzilla/metadata.xml @@ -3,6 +3,8 @@ perl +WWW::Bugzilla::Search +WWW::Bugzilla WWW-Bugzilla --- a/dev-perl/Moose/metadata.xml +++ b/dev-perl/Moose/metadata.xml @@ -3,6 +3,17 @@ perl +Class::MOP +Moose::Meta::Attribute::Native::Trait::Counter +Moose::Meta::Attribute::Native::Trait::String +Moose::Meta::Method::Delegation +Moose::Meta::TypeConstraint::Role +Moose::Meta::Attribute::Custom::Moose +Moose::Meta::Attribute +Moose::Meta::Method +Moose::Error::Croak +Moose::Util::MetaRole +Moose::Role Moose --- a/perl-core/Scalar-List-Utils/metadata.xml +++ b/perl-core/Scalar-List-Utils/metadata.xml @@ -3,6 +3,9 @@ perl +List::Util +Scalar::Util +List::Util::XS Scalar-List-Utils --- a/dev-perl/libwww-perl/metadata.xml +++ b/dev-perl/libwww-perl/metadata.xml @@ -3,6 +3,17 @@ perl +LWP::Protocol::data +LWP::Protocol::gopher +LWP::Debug +LWP::UserAgent +LWP::Authen::Digest +LWP::Protocol::file +LWP::Protocol::loopback +LWP::Protocol::MyFTP +LWP::Protocol::ftp +LWP::Protocol::cpan +LWP::MemberMixin libwww-perl -- Corentin Chary http://xf.iksaif.net/
Re: [gentoo-dev] RFC: Add new remote-id types in metadata.dtd
On Fri, Apr 20, 2012 at 9:35 PM, Kent Fredric wrote: > On 21 April 2012 01:34, Corentin Chary wrote: >> Yeah, not very important, but seems to work with this patch: >> https://github.com/iksaif/portage-janitor/commit/972aff94744741e34e99f917337430d245883c48 >> >> Example: >> $ python remoteids.py --diff WWW-Bugzilla Moose bioperl >> --- a/dev-perl/WWW-Bugzilla/metadata.xml >> +++ b/dev-perl/WWW-Bugzilla/metadata.xml >> @@ -3,6 +3,7 @@ >> > ... >> -- >> Corentin Chary >> http://xf.iksaif.net/ >> > > What does it to for say, Scalar-List-Utils ? or libwww-perl ? Nothing, it's one of the cases you'll have to handle by-hand. Another option is to parse the HTML from http://search.cpan.org/dist/Scalar-List-Utils/ but I'm not a fan of HTML-parsing. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] RFC: Add new remote-id types in metadata.dtd
On Sat, Apr 21, 2012 at 12:39:26AM +1200, Kent Fredric wrote: > If you really want to support Perl Modules, ( which theres not much > need for at present, looks like the team have gone through already for > the most part and added remote-id's where possible already ), anything > that inherits 'perl-module.eclass' has a bit of magic, in that neither > SRC_URI or HOMEPAGE is required in the ebuild, and it just gets the > package name from what gentoo is using. We've tried to be as close to > upstream as possible for the ease of maintenance. Actually the eclass seems to fill SRC_URI and HOMEPAGE, so when you source the ebuild they are available. > However, there are still exception cases, for instance, BioPerl has to > define 'MY_PN' to tell the perl-module eclass to use a different token > ( and when this is present, it should be sufficient to say that that > should be the remote-id instead of the package name: > see dev-perl/Moose # an example with neither src_uri or homepage > see sci-biology/bioperl # an example where the package name has > been forced overridden as its changed upstream > > But resolving module names is much trickier, its easy-ish to map a > module name to a package using the service, but doing it the other way > round is not so straight forward, as one package can have many > modules, and its common in perl to state dependencies in terms of the > module to require, not the package its in, but there's also often a > defacto "main module". > > But I'm myself still working out how to best do that with the service > , so auto-populating a "cpan-module" identifier can be left to later, > its just something I considered "useful" to have metadata wise because > that value is more useful to users. Yeah, not very important, but seems to work with this patch: https://github.com/iksaif/portage-janitor/commit/972aff94744741e34e99f917337430d245883c48 Example: $ python remoteids.py --diff WWW-Bugzilla Moose bioperl --- a/dev-perl/WWW-Bugzilla/metadata.xml +++ b/dev-perl/WWW-Bugzilla/metadata.xml @@ -3,6 +3,7 @@ perl +WWW::Bugzilla WWW-Bugzilla --- a/dev-perl/Moose/metadata.xml +++ b/dev-perl/Moose/metadata.xml @@ -3,6 +3,7 @@ perl +Moose Moose --- a/sci-biology/bioperl/metadata.xml +++ b/sci-biology/bioperl/metadata.xml @@ -8,6 +8,7 @@ Install sci-biology/bioperl-run + BioPerl BioPerl -- Corentin Chary http://xf.iksaif.net/
Re: [gentoo-dev] RFC: Add new remote-id types in metadata.dtd
On Fri, Apr 20, 2012 at 10:26 AM, Kent Fredric wrote: > On 20 April 2012 19:46, Corentin Chary wrote: >> On Fri, Apr 20, 2012 at 9:37 AM, Kent Fredric wrote: >>> On 20 April 2012 03:31, Corentin Chary wrote: >>>> Add rubygems, github, gitorious, pecl, pear, bitbucket. >>>> All of them are handled by my remoteids.py script. >>>> >>>> ref: https://bugs.gentoo.org/show_bug.cgi?id=406287 >>>> ref: https://github.com/iksaif/portage-janitor/blob/master/remoteids.py >>>> >>>> --- a/metadata/dtd/metadata.dtd 2010-03-02 18:52:11.0 +0100 >>>> +++ b/metadata/dtd/metadata.dtd 2012-04-19 14:22:14.077954310 +0200 >>>> @@ -61,7 +61,7 @@ >>>> >>>> >>>> >>>> - >>> (freshmeat|sourceforge|sourceforge-jp|cpan|vim|google-code|ctan|pypi|rubyforge|cran) >>>> #REQUIRED> >>>> + >>> (freshmeat|sourceforge|sourceforge-jp|cpan|vim|google-code|ctan|pypi|rubyforge|cran|rubygems|github|gitorious|pecl|pear|bitbucket) >>>> #REQUIRED> >>>> >>>> >>>> >>>> -- >>>> Corentin Chary >>>> http://xf.iksaif.net/ >>> >>> >>> I suggested last week on #gentoo-perl that it might be nice to have >>> 'cpan' and 'cpan-module' ( or something like that ) to disambiguate 2 >>> queryable terms. ( where 'cpan' => 'the package name on cpan' ) >>> >>> For some purposes, its most convenient to use the distribution name, >>> and for other purposes, (ie: cpan clients) its more convenient to use >>> a Module name, and its not easy to translate between the two, as >>> Module names sometimes switch between packages they're shipped in. >>> >>> For instance, a while ago, the BioPerl module was shipped in a >>> distribution 'bioperl' , which has only recently been changed to >>> BioPerl >>> >>> >>> http://api.metacpan.org/release/_search?q=distribution:bioperl&fields=archive,author,date,download_url >>> >>> http://api.metacpan.org/release/_search?q=distribution:BioPerl&fields=archive,author,date,download_url >>> >>> vs >>> >>> >>> http://api.metacpan.org/module/_search?q=module.name:Bio\:\:Perl&fields=distribution,author,release >> >> Looks sane since the goal of remote-id is being able to identify the >> package upstream. >> Do you think you could patch remotesid.py to generate tags for cpan / >> cpan-modules ? Or at least give me a pseudo-algo that does the trick. >> Thanks :) >> >> -- >> Corentin Chary >> http://xf.iksaif.net >> > > > That is sadly not straight forward. Extracting the package name can > be straight forward if you have the URL, because the package name is > literally the same as the archive name in SRC_URI , sans version > information. > > However, if you look at many perl ebuilds, you'll notice many lack > this field and we've got other things in place, so the current parsing > technique you use to detect uses of SRC_URI wont work there ( I could > be wrong, I don't fully grok your python code ) Currently it uses SRC_URI and HOMEPAGE, but honestly it wouldn't be hard to use any other environment variable and to do some checks on a webservice. Anyway for tricky cases it can still be done by hand. > And more-over, determining the value of 'cpan-module' may be > impossible without access to the tar.gz itself, or querying the > MetaCPAN API. > > Usually, upstream are sensible and have package names which closely > correspond with the module names, ie: "Dist::Zilla" is shipped in > 'Dist-Zilla-$VERSION.tar.gz', but there are many packages which dont > do this, such as this notable example: > https://metacpan.org/release/Scalar-List-Utils , which has no modules > corresponding to the package name, and no way to divine the/a 'main' > module from the package itself. ( and this is exacerbated by packages > changing names, or package joins ( 2 packages becoming 1 via releasing > modules together ), and package splits ( 1 package rips into 2 sets > of modules ). > > Essentially, using a cpan-module as an identifier is somewhat > "forwards only" , and even then, what it will resolve to is governed > by time. > > This is fine for CPAN clients, which do the resolution hot, using the > whole of CPAN as their data, if a user asks for "Foo::Bar", their cpan > client will ask a cpan server ( or regularly (hourly) updated list ) > as to what package that module can be found in ( and this only returns > the most recent package, so name changes and so-forth are invisible to > the user ). > > And being helpful to CPAN clients is one of the reasons we want this > value as a specifiable option in the first place. For us, its easier > to track the package name, and then when that has to change we can > manually resolve the issue > > -- > Kent > > perl -e "print substr( \"edrgmaM SPA NOcomil.ic\\@tfrken\", \$_ * 3, > 3 ) for ( 9,8,0,7,1,6,5,4,3,2 );" > > http://kent-fredric.fox.geek.nz > -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] RFC: Add new remote-id types in metadata.dtd
On Fri, Apr 20, 2012 at 9:37 AM, Kent Fredric wrote: > On 20 April 2012 03:31, Corentin Chary wrote: >> Add rubygems, github, gitorious, pecl, pear, bitbucket. >> All of them are handled by my remoteids.py script. >> >> ref: https://bugs.gentoo.org/show_bug.cgi?id=406287 >> ref: https://github.com/iksaif/portage-janitor/blob/master/remoteids.py >> >> --- a/metadata/dtd/metadata.dtd 2010-03-02 18:52:11.0 +0100 >> +++ b/metadata/dtd/metadata.dtd 2012-04-19 14:22:14.077954310 +0200 >> @@ -61,7 +61,7 @@ >> >> >> >> - > (freshmeat|sourceforge|sourceforge-jp|cpan|vim|google-code|ctan|pypi|rubyforge|cran) >> #REQUIRED> >> + > (freshmeat|sourceforge|sourceforge-jp|cpan|vim|google-code|ctan|pypi|rubyforge|cran|rubygems|github|gitorious|pecl|pear|bitbucket) >> #REQUIRED> >> >> >> >> -- >> Corentin Chary >> http://xf.iksaif.net/ > > > I suggested last week on #gentoo-perl that it might be nice to have > 'cpan' and 'cpan-module' ( or something like that ) to disambiguate 2 > queryable terms. ( where 'cpan' => 'the package name on cpan' ) > > For some purposes, its most convenient to use the distribution name, > and for other purposes, (ie: cpan clients) its more convenient to use > a Module name, and its not easy to translate between the two, as > Module names sometimes switch between packages they're shipped in. > > For instance, a while ago, the BioPerl module was shipped in a > distribution 'bioperl' , which has only recently been changed to > BioPerl > > > http://api.metacpan.org/release/_search?q=distribution:bioperl&fields=archive,author,date,download_url > > http://api.metacpan.org/release/_search?q=distribution:BioPerl&fields=archive,author,date,download_url > > vs > > > http://api.metacpan.org/module/_search?q=module.name:Bio\:\:Perl&fields=distribution,author,release Looks sane since the goal of remote-id is being able to identify the package upstream. Do you think you could patch remotesid.py to generate tags for cpan / cpan-modules ? Or at least give me a pseudo-algo that does the trick. Thanks :) -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] RFC: Add new remote-id types in metadata.dtd
On Thu, Apr 19, 2012 at 6:54 PM, Michał Górny wrote: > On Thu, 19 Apr 2012 17:31:11 +0200 > Corentin Chary wrote: > >> - > (freshmeat|sourceforge|sourceforge-jp|cpan|vim|google-code|ctan|pypi|rubyforge|cran) >> #REQUIRED> >> + > (freshmeat|sourceforge|sourceforge-jp|cpan|vim|google-code|ctan|pypi|rubyforge|cran|rubygems|github|gitorious|pecl|pear|bitbucket) >> #REQUIRED> > > Wouldn't it be better if we kept them sorted? There were not, so I just appended them, but I can provide another patch sorting them too if preferred. -- Corentin Chary http://xf.iksaif.net
[gentoo-dev] RFC: Add new remote-id types in metadata.dtd
Add rubygems, github, gitorious, pecl, pear, bitbucket. All of them are handled by my remoteids.py script. ref: https://bugs.gentoo.org/show_bug.cgi?id=406287 ref: https://github.com/iksaif/portage-janitor/blob/master/remoteids.py --- a/metadata/dtd/metadata.dtd 2010-03-02 18:52:11.0 +0100 +++ b/metadata/dtd/metadata.dtd 2012-04-19 14:22:14.077954310 +0200 @@ -61,7 +61,7 @@ - + -- Corentin Chary http://xf.iksaif.net/ pgplrUeuh7QHW.pgp Description: PGP signature
Re: [gentoo-dev] thirdpartymirrors URL for bitbucket
On Wed, Apr 11, 2012 at 12:44 PM, Michał Górny wrote: > Hello all, > > Similarly to github, bitbucket does enforce SSL by default, and > downloads are redirected to another, non-https URI. Thus, I'd like to > add the following thirdpartymirrors entry: > > bitbucket http://cdn.bitbucket.org > > The path part of URI is consistent with the usual https://bitbucket.org > one, so if the URI stops working, we can replace it with the 'official' > one. > > Any comments? Note that you could use my mirror.py (with some tweaks) script to generate a patch that makes ebuilds use this mirror. - https://github.com/iksaif/portage-janitor (portage-janitor scripts, including mirrors.py and others) - https://bugs.gentoo.org/show_bug.cgi?id=405533 (Fix mirrors in multiple ebuilds) -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] New eclass for Python
On Wed, Apr 4, 2012 at 4:22 PM, Mike Gilbert wrote: > On Wed, Apr 4, 2012 at 4:50 AM, Corentin Chary > wrote: >> I have a package that depends on python-dateutil:python-2 for >> python2_x and python-dateutil:python-3 for python3_x. >> Would it be possible to have virtual targets like "python, python2, >> python3, pypi, jithon" ? >> > > With regards to python-dateutil: As of python-dateutil-2.1, there are > no longer separate slots for python-2 and python-3. As well, I masked > the only version (2.0) with SLOT=python-3. > > For future compatibility, you should remove the slot from your > dependencies and just depend on dev-python/python-dateutil. > Yep, I just saw that. But well, celery is full of examples like thats since it have different dependencies for python2, python2.5, python2.5 + tests, jython, jython + tests, etc.. Having a way to group similar python would be great. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] New eclass for Python
On Mon, Mar 26, 2012 at 6:23 PM, Krzysztof Pawlik wrote: > On 26/03/12 18:11, Krzysztof Pawlik wrote: >> On 26/03/12 09:20, justin wrote: >>> On 25/03/12 20:56, Krzysztof Pawlik wrote: >>>> On 28/02/12 22:13, Krzysztof Pawlik wrote: >>>>> If there are no objections then during the weekend (March 3, 4) I will >>>>> add this >>>>> to portage (after finishing remaining TODO items, PyPy requires 4G of >>>>> RAM(!!)). >>>> >>>> Hello, >>>> >>>> Slightly late due to Real Life™ but finally it's in the main tree :) >>>> >>>> (and yes - I've tested it with pypy - works as expected :) >>>> >>> >>> Hi, >>> >>> is there any documentation beside the man page somewhere? >> >> No. >> >>> I tried to port some ebuilds but as soon I set >>> >>> PYTHON_COMPAT="python2_7 python2_6 python2_5 pypy1_8" >>> >>> inherit python-distutils-ng >>> >>> I get >>> >>> REQUIRED_USE: USE flag 'python_targets_python3_1' is not in IUSE >>> >>> Did I do something wrong, or is there something not straight in the eclass? >> >> Can you send me the whole ebuild off-list? >> >> There are two ebuilds using the eclass that I've used as tests: >> http://git.overlays.gentoo.org/gitweb/?p=dev/nelchael.git;a=tree;f=dev-python;h=f1a8e00e3e6df33806d8972c8898f1187163bd3d;hb=HEAD > > Ok, found a bug: REQUIRED_USE can't contain elements not in USE, so if you > excluded python3_1 from PYTHON_COMPAT it didn't appear in IUSE too -> > REQUIRED_USE contained invalid value. Fixed by below patch: > > nelchael@s-lappy ~/.../gentoo-x86/eclass$ cvs diff > Index: python-distutils-ng.eclass > === > RCS file: /var/cvsroot/gentoo-x86/eclass/python-distutils-ng.eclass,v > retrieving revision 1.2 > diff -u -r1.2 python-distutils-ng.eclass > --- python-distutils-ng.eclass 26 Mar 2012 06:12:53 - 1.2 > +++ python-distutils-ng.eclass 26 Mar 2012 16:20:52 - > @@ -105,11 +105,11 @@ > esac > } > > -required_use_str=" || ( > - python_targets_python2_5 python_targets_python2_6 > python_targets_python2_7 > - python_targets_python3_1 python_targets_python3_2 > - python_targets_jython2_5 > - python_targets_pypy1_7 python_targets_pypy1_8 )" > +required_use_str="" > +for impl in ${PYTHON_COMPAT}; do > + required_use_str="${required_use_str} python_targets_${impl}" > +done > +required_use_str=" || ( ${required_use_str} )" > if [[ "${PYTHON_OPTIONAL}" = "yes" ]]; then > IUSE+="python" > REQUIRED_USE+=" python? ( ${required_use_str} )" > > > -- > Krzysztof Pawlik key id: 0xF6A80E46 > desktop-misc, java, vim, kernel, python, apache... > I have a feature request for distutil-ng (or maybe it's already possible but I don't know how). I have a package that depends on python-dateutil:python-2 for python2_x and python-dateutil:python-3 for python3_x. Would it be possible to have virtual targets like "python, python2, python3, pypi, jithon" ? Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] lurking *.ebuild'less packages
On Thu, Mar 15, 2012 at 10:42 PM, Matt Turner wrote: > On Thu, Mar 15, 2012 at 4:59 PM, Sergei Trofimovich wrote: >> slep noticed and reported an odd thing: >> >> $ euse -i kate >> ... >> ls: cannot access >> /gentoo/portage/metadata/cache/kde-base/kdebindings-perl-*: No such file or >> directory >> ls: cannot access >> /gentoo/portage/metadata/cache/kde-base/kdebindings-ruby-*: No such file or >> directory >> ... >> >> The dirs they don't contain ebuilds. Only metadata. >> KDE team is aware and fixing their orphans. >> >> I've decided to write a hack to find such instances in tree: >> >> $ ./find_empty.sh ~/portage/gentoo-x86/ >> kde-base/kdeaccessibility-colorschemes >> kde-base/kdeaccessibility-iconthemes >> kde-base/kdebase-wallpapers >> kde-base/kdebindings-csharp >> kde-base/kdebindings-perl >> kde-base/kdebindings-ruby >> kde-base/kvtml-data >> kde-base/smoke >> media-plugins/mytharchive >> media-plugins/mythbrowser >> media-plugins/mythgallery >> media-plugins/mythgame >> media-plugins/mythmovies >> media-plugins/mythmusic >> media-plugins/mythnetvision >> media-plugins/mythnews >> media-plugins/mythweather >> x11-themes/mythtv-themes >> >> Is there any reason to leave such ebuildless directories? >> >> Thanks! >> >> -- >> >> Sergei > > Let's get this script added to http://qa-reports.gentoo.org/ > Speaking about qa-reports, do you think "SRC_URI not using mirror:// when they should" and "metadata.xml not containing remote-id when they should" belong there ? If yes, please take a look at: - https://github.com/iksaif/portage-janitor - https://bugs.gentoo.org/show_bug.cgi?id=405533 - https://bugs.gentoo.org/show_bug.cgi?id=406287 Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Gentoo Janitor scripts
On Mon, Feb 27, 2012 at 4:10 PM, "Paweł Hajdan, Jr." wrote: > On 2/20/12 6:03 PM, Corentin Chary wrote: >> Since I plan to use the remote remote-id tag for euscan, and I already >> use SRC_URI but I'd like all ebuild to use mirrors, I've wrote to >> scripts to cleanup your ebuilds and metadata. >> There are available here: https://github.com/iksaif/portage-janitor >> Here is what you can do with them: >> >> python remoteids.py --diff pycuda Test-Tester Alien-SDL ostinato >> --- a/dev-python/pycuda/metadata.xml >> +++ b/dev-python/pycuda/metadata.xml >> @@ -4,4 +4,7 @@ >> >> sp...@gentoo.org >> >> + >> + pycuda >> + >> >> >> Maybe some bits could be integrated to repoman... > > I second that, those remoteids.py changes LGTM (look good to me). > > As always, I second any effort to make those useful things part of > "official Gentoo". > Fix remote ids bug: https://bugs.gentoo.org/show_bug.cgi?id=406287 Fix mirrors bug: https://bugs.gentoo.org/show_bug.cgi?id=405533 -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: Gentoo Janitor scripts
On Wed, Feb 22, 2012 at 7:03 PM, Markos Chandras wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On 02/22/2012 01:33 PM, Ben wrote: >> On 22 February 2012 20:36, Corentin Chary >> wrote: >>> - "960 http://dev.gentoo.org": that's a lot of package >>> hosted here, is that really right ? >> >> That includes patches >20kb >> > Gentoo devs are supposed to put patches, tarballs and whatever they > want in their space. This is the recommended policy documented in > devmanual as well > > http://devmanual.gentoo.org/general-concepts/mirrors/index.html Yes, you're right, I totally forgot about patches. I'll try to make the script smarter and skip those. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: Gentoo Janitor scripts
On Wed, Feb 22, 2012 at 1:20 PM, Mart Raudsepp wrote: > On K, 2012-02-22 at 09:48 +0100, Corentin Chary wrote: >> I did a quick script to count most used prefixes in SRC_URI yesterday >> (https://github.com/iksaif/portage-janitor/blob/master/mirrors.py) >> >> Here is the (filtered) result: >> >> $ eix --only-names | python mirrors.py --count >> 960 http://dev.gentoo.org >> 372 http://xorg.freedesktop.org >> 372 http://xorg.freedesktop.org/releases >> 372 http://xorg.freedesktop.org/releases/individual >> 306 http://pear.php.net >> 306 http://pear.php.net/get >> 256 http://oss.tresys.com >> 255 http://oss.tresys.com/files >> 255 http://oss.tresys.com/files/refpolicy >> 225 http://hackage.haskell.org/packages >> 225 http://hackage.haskell.org/packages/archive >> 225 http://hackage.haskell.org >> 206 http://ftp.xemacs.org >> 201 https://github.com >> 196 http://ftp.xemacs.org/pub >> 196 http://ftp.xemacs.org/pub/xemacs >> 193 http://ftp.xemacs.org/pub/xemacs/packages >> 181 http://gstreamer.freedesktop.org >> 181 http://gstreamer.freedesktop.org/src >> 175 http://launchpad.net >> 175 http://linuxgazette.net >> 143 http://github.com >> 130 http://pear.horde.org >> 130 http://pear.horde.org/get >> 101 http://savannah.nongnu.org/download >> 101 http://savannah.nongnu.org >> 100 http://get.qt.nokia.com >> 97 ftp://sources.redhat.com/pub >> 97 ftp://sources.redhat.com >> 96 http://get.qt.nokia.com/qt >> 95 http://get.qt.nokia.com/qt/source >> 90 http://download.gna.org >> 75 http://pecl.php.net >> 75 http://pecl.php.net/get >> 72 http://components.ez.no/get >> 72 http://components.ez.no >> 69 https://fedorahosted.org >> 67 http://www.phrack.org/archives >> 67 http://www.phrack.org/archives/tgz >> 67 http://www.phrack.org >> >> >> From that output we can easilly find out new entries to >> thirdpartymirrors, for example: >> gentoo-dev http://dev.gentoo.org >> xorg http://xorg.freedesktop.org >> gna http://download.gna.org >> pecl http://pecl.php.net >> pear http://pear.php.net >> github https://github.com http://github.com >> xemacs http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/ >> launchpad http://launchpad.net >> redhat ftp://sources.redhat.com/pub/ (and probably others !) >> etc... >> >> The good part is that once you've modified thirdpartymirrors with new >> mirrors, running mirrors.py --all will generate a big patch for all >> your ebuilds to use those new mirrors ! > > If you want this, then you should better figure out actual upstream > mirroring systems and their list of mirrors they would want us to use. > Until such, this seems to be just for shortening SRC_URI addresses when > an upstream tarball domain name or path repeats, and that's definitely > not what thirdpartymirrors is for. Yes, of course, that was just a quick example, not something definitive. But lets some examples: - http://xorg.freedesktop.org, it's easy to find mirror for that one, http://ftp.x.org/pub/ for example - github: packages seems to use http and https, this script can help to standardize the url used - "960 http://dev.gentoo.org": that's a lot of package hosted here, is that really right ? And still, thirdpartymirrors have some entries with only one "mirror" and I believe factorizing SRC_URIs is a good thing (if something changes, you just patch thirdpartymirros, not hundreds of ebuilds). -- Corentin Chary http://xf.iksaif.net
[gentoo-dev] Re: Gentoo Janitor scripts
I did a quick script to count most used prefixes in SRC_URI yesterday (https://github.com/iksaif/portage-janitor/blob/master/mirrors.py) Here is the (filtered) result: $ eix --only-names | python mirrors.py --count 960 http://dev.gentoo.org 372 http://xorg.freedesktop.org 372 http://xorg.freedesktop.org/releases 372 http://xorg.freedesktop.org/releases/individual 306 http://pear.php.net 306 http://pear.php.net/get 256 http://oss.tresys.com 255 http://oss.tresys.com/files 255 http://oss.tresys.com/files/refpolicy 225 http://hackage.haskell.org/packages 225 http://hackage.haskell.org/packages/archive 225 http://hackage.haskell.org 206 http://ftp.xemacs.org 201 https://github.com 196 http://ftp.xemacs.org/pub 196 http://ftp.xemacs.org/pub/xemacs 193 http://ftp.xemacs.org/pub/xemacs/packages 181 http://gstreamer.freedesktop.org 181 http://gstreamer.freedesktop.org/src 175 http://launchpad.net 175 http://linuxgazette.net 143 http://github.com 130 http://pear.horde.org 130 http://pear.horde.org/get 101 http://savannah.nongnu.org/download 101 http://savannah.nongnu.org 100 http://get.qt.nokia.com 97 ftp://sources.redhat.com/pub 97 ftp://sources.redhat.com 96 http://get.qt.nokia.com/qt 95 http://get.qt.nokia.com/qt/source 90 http://download.gna.org 75 http://pecl.php.net 75 http://pecl.php.net/get 72 http://components.ez.no/get 72 http://components.ez.no 69 https://fedorahosted.org 67 http://www.phrack.org/archives 67 http://www.phrack.org/archives/tgz 67 http://www.phrack.org >From that output we can easilly find out new entries to thirdpartymirrors, for example: gentoo-devhttp://dev.gentoo.org xorg http://xorg.freedesktop.org gna http://download.gna.org pecl http://pecl.php.net pear http://pear.php.net github https://github.com http://github.com xemacs http://ftp.xemacs.org/pub/ ftp://ftp.sa.xemacs.org/pub/ launchpadhttp://launchpad.net redhat ftp://sources.redhat.com/pub/ (and probably others !) etc... The good part is that once you've modified thirdpartymirrors with new mirrors, running mirrors.py --all will generate a big patch for all your ebuilds to use those new mirrors ! -- Corentin Chary http://xf.iksaif.net
[gentoo-dev] Gentoo Janitor scripts
Hi, Since I plan to use the remote remote-id tag for euscan, and I already use SRC_URI but I'd like all ebuild to use mirrors, I've wrote to scripts to cleanup your ebuilds and metadata. There are available here: https://github.com/iksaif/portage-janitor Here is what you can do with them: python remoteids.py --diff pycuda Test-Tester Alien-SDL ostinato --- a/dev-python/pycuda/metadata.xml +++ b/dev-python/pycuda/metadata.xml @@ -4,4 +4,7 @@ sp...@gentoo.org + +pycuda + --- a/dev-perl/Alien-SDL/metadata.xml +++ b/dev-perl/Alien-SDL/metadata.xml @@ -7,4 +7,7 @@ ssuomi...@gentoo.org Samuli Suominen + +Alien-SDL + --- a/net-analyzer/ostinato/metadata.xml +++ b/net-analyzer/ostinato/metadata.xml @@ -7,5 +7,7 @@ + +ostinato + $ eix -C dev-python --only-names | python mirrors.py --diff --- a/dev-python/asciitable/asciitable-0.8.0.ebuild +++ b/dev-python/asciitable/asciitable-0.8.0.ebuild @@ -11,7 +11,7 @@ DESCRIPTION="An extensible ASCII table reader" HOMEPAGE="http://pypi.python.org/pypi/asciitable http://cxc.harvard.edu/contrib/asciitable"; -SRC_URI="http://pypi.python.org/packages/source/a/${PN}/${P}.tar.gz"; +SRC_URI="mirror://pypi/a/${PN}/${P}.tar.gz" LICENSE="GPL-2" SLOT="0" --- a/dev-python/cosmolopy/cosmolopy-0.1.102.ebuild +++ b/dev-python/cosmolopy/cosmolopy-0.1.102.ebuild @@ -15,7 +15,7 @@ DESCRIPTION="Cosmology routines built on NumPy/SciPy" HOMEPAGE="http://roban.github.com/CosmoloPy/ http://pypi.python.org/pypi/CosmoloPy"; -SRC_URI="http://pypi.python.org/packages/source/C/${MY_PN}/${MY_P}.tar.gz"; +SRC_URI="mirror://pypi/C/${MY_PN}/${MY_P}.tar.gz" LICENSE="MIT" SLOT="0" Feel free to test them, and if they are broken I'll gladly accept a patch :). Maybe some bits could be integrated to repoman... Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] RFC: upstream/watch in metadata.xml
On Mon, Feb 13, 2012 at 10:59 AM, Corentin Chary wrote: > On Mon, Feb 13, 2012 at 10:52 AM, Robin H. Johnson wrote: >> On Mon, Feb 13, 2012 at 10:33:11AM +0100, Corentin Chary wrote: >>> Currently, if you run euscan on this package, it doesn't work at all: >>> http://euscan.iksaif.net/package/media-plugins/vdr-softdevice/ >>> 1/ it's hosted on gentoo mirrors, and scanning them takes too long >>> because all files are in the same directory >> I've been wondering if it would help to have a pregenerated index go out >> to the mirrors from our master box, would that be useful for you? > > Would be better, but the index would still be pretty big (and > currently euscan doesn't cache anything, maybe it should). > Note that even with a HTTP cache, scanning it would take a lot of CPU if it is too big :/. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] RFC: upstream/watch in metadata.xml
On Mon, Feb 13, 2012 at 10:52 AM, Robin H. Johnson wrote: > On Mon, Feb 13, 2012 at 10:33:11AM +0100, Corentin Chary wrote: >> Currently, if you run euscan on this package, it doesn't work at all: >> http://euscan.iksaif.net/package/media-plugins/vdr-softdevice/ >> 1/ it's hosted on gentoo mirrors, and scanning them takes too long >> because all files are in the same directory > I've been wondering if it would help to have a pregenerated index go out > to the mirrors from our master box, would that be useful for you? Would be better, but the index would still be pretty big (and currently euscan doesn't cache anything, maybe it should). -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] RFC: upstream/watch in metadata.xml
On Mon, Feb 13, 2012 at 10:50 AM, Dirkjan Ochtman wrote: > On Mon, Feb 13, 2012 at 10:33, Corentin Chary > wrote: >> One other thing, metadata.xml already contain a remote-id tag, which >> would be very great to help euscan do its job, but a lot of package >> are lacking it: >> - Should we patch repoman to scan SRC_URI and issue a warning when it >> looks like an URI that match a well known remote-id >> - Should we write a script to update metadata.xml ? It would be easy >> for rubygem, pypi and pear packages. >> >> Any comment ? Objections ? Ideas ? > > I like the idea for keeping the data somewhere for known-insane cases, > and metadata.xml sounds like it might be fine. But I don't think we > should add anything for the likes of PyPI, if we can easily derive > that we should look on PyPI some other way (i.e. for python, many > packages list a PyPI page in their HOMEPAGE). For pypi (and some others), looking at SRC_URI is enought: it starts with mirror://pypi/. Still for those *must* be set because the package name is not always exactly the same as in gentoo. Currently euscan tries to guess it, but it is not always accurate. Most of the time, if remote-id is set, we don't need "version-scan" because upstream provides a stable API to list versions. -- Corentin Chary http://xf.iksaif.net
[gentoo-dev] RFC: upstream/watch in metadata.xml
As some may know, I'm working on euscan, and currently euscan only use CPV and SRC_URI to find new upstream versions. This works well if upstream url and version scheme is sane or if upstream has an API for that (rubygem, pypi, pecl, pear), but it's far from optimal. Debian use a specific file for that: debian/watch and it looks like that (for media-plugins/vdr-softdevice): opts=downloadurlmangle=s/prdownload/download/ \ http://developer.berlios.de/project/showfiles.php?group_id=2051 \ http://prdownload.berlios.de/softdevice/vdr-softdevice-(.+).tgz opts specify some options to mangle the final url, and then there is a list of url to scan. man uscan for more informations. Currently, if you run euscan on this package, it doesn't work at all: http://euscan.iksaif.net/package/media-plugins/vdr-softdevice/ 1/ it's hosted on gentoo mirrors, and scanning them takes too long because all files are in the same directory 2/ the url doesn't contain the version So, to help euscan (and other tools) for some package, I think we could introduce some hints in metadata.xml. This would extend the existing "upstream" element: http://developer.berlios.de/project/showfiles.php?group_id=2051 \ http://prdownload.berlios.de/softdevice/vdr-softdevice-(.+).tgz The format is not defined yet, but it would probably look like debian/watch, that would allow to write a script to import (valid) debian/watch files into associated metadata.xml when needed. One other thing, metadata.xml already contain a remote-id tag, which would be very great to help euscan do its job, but a lot of package are lacking it: - Should we patch repoman to scan SRC_URI and issue a warning when it looks like an URI that match a well known remote-id - Should we write a script to update metadata.xml ? It would be easy for rubygem, pypi and pear packages. Any comment ? Objections ? Ideas ? Thanks, -- Corentin Chary http://xf.iksaif.net
[gentoo-dev] euscan-0.1.0 released
I released eucsan-0.1.0 last week and hwoarang uploaded the associated ebuild. You can now emerge it instead of euscan-. Note that this is only the standalone utility, not the web interface. I you have an overlay, you can run "eix --in-overlay --only-names | xargs euscan" to scan only your packages (and you can use sys-process/parallel instead of xargs). Nothing really new in 0.1.0 except that this is the first release. I don't know what could be added to the command line utility for the next version, except maybe new site handlers (sourceforge, cpan, mysql, etc..). Feel free to write some, it's really easy even with basic python skills. On the other side I have a long list of thing in my TODO for the web interface, including: - Using celery task queue for on-demand scanning - Better overlay integration (adding an overlay from the admin, etc...) - Custom commands (euscaninit to build a local rootfs with portage and layman stuff, scan a specific category/herd/maintainer/package) - Migration to gentoo-infra ? :) - euscan on iksaif.net: http://euscan.iksaif.net - euscan for chromium-os: http://chromium-os.euscan.iksaif.net/ - euscan on github: https://github.com/iksaif/euscan Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
> Right, having a stable API and a special euscan tag on the wiki could > solve that. Speeking of API, I found some time to play with django-piston this week, and here is the result: - http://euscan.iksaif.net/about/api Some examples: - http://euscan.iksaif.net/api/1.0/packages/by-maintainer/1.json - http://euscan.iksaif.net/api/1.0/packages/by-herd/openoffice.json - http://euscan.iksaif.net/api/1.0/packages/by-category/app-office.xml - http://euscan.iksaif.net/api/1.0/package/app-office/libreoffice-bin.json -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Mon, Sep 26, 2011 at 8:42 PM, Hans de Graaff wrote: > On Tue, 2011-09-20 at 10:00 +0200, Corentin Chary wrote: >> On Mon, Sep 19, 2011 at 9:23 PM, Hans de Graaff wrote: > >> > One integration that might also be useful is what we track on the ruby >> > wiki currently: https://overlays.gentoo.org/proj/ruby/wiki/PendingBumps >> > which is a collection of quick notes on problems encountered while >> > bumping packages. The ability to attach notes to packages and to >> > acknowledge version bumps (and e.g. make them yellow instead of red) >> > could help with larger lists for herds. > >> You can already use that page: http://euscan.iksaif.net/herds/ruby/ > > I think you missed my point. This page is more like what we have at > http://moving-innovations.com/~graaff/outdated-gems/ which is an > automated list based on what is in the tree and upstream. Your page is a > good replacement for that so we may just use that in the future. > > What our wiki page does it keep *notes* on these bumps, so all of that > relies on manual input. (e.g. "we tried to bump this but it needs a new > dependency that we don't have yet.") The euscan page doesn't support > that yet. Arguably I should open bugs for each of these bumps, but that > seems like a lot of overhead for some quick notes. Right, having a stable API and a special euscan tag on the wiki could solve that. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
>> Open a bug, attach your ebuilds ( are there any releases besides the >> git code? ) and I will take it from there > > Here it is: https://bugs.gentoo.org/show_bug.cgi?id=383937 > There are no "released" version, and the project have no homepage > (except http://euscan.iksaif.net). euscan ebuild is now in portage, Thanks Markos ! -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] euscan proof of concept (like debian's uscan)
On Mon, Sep 19, 2011 at 10:53 AM, Michał Górny wrote: > On Mon, 19 Sep 2011 10:39:11 +0200 > Corentin Chary wrote: > >> ## Also update eix database, because we use eix internaly >> ## Bottleneck: disk and cpu >> ##Time: 30mn ~ 1h >> eix-update > > Using egencache to keep caches for overlays will make eix updates much > faster. > > Here's my code for it (it uses overlays in /usr/portage/local): > > cd /usr/portage/local && \ > for O in */; do > echo "${O}" > egencache --jobs=8 --update --update-use-local-desc --rsync \ > --repo=$(cat ${O}profiles/repo_name) > done > Just done that, and indeed, it's much faster. Also I switched to PostgreSQL, and since that avoids a lot of deadlock situations when making queries in parallel, I'm now able to do that: eix --only-names -x | gparallel --eta --load 8 --jobs 400% --max-args=64 python manage.py scan-metadata And now it takes less than 30mn instead of more than 1h, and more importantly, it scales. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Wed, Sep 21, 2011 at 12:58 PM, Markos Chandras wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On 09/21/11 11:57, Corentin Chary wrote: >> On Tue, Sep 20, 2011 at 9:04 PM, Donnie Berkholz >> wrote: >>> On 10:00 Tue 20 Sep , Corentin Chary wrote: >>>> Could someone write ebuilds for euscan and euscanwww ? It >>>> should not take a lot of time, but my ebuilds skills are >>>> probably not good enought to do that. >>> >>> Sounds like good practice for when you become a Gentoo dev. =) >> >> Hehe :). Maybe someday, but honestly I don't have any time right >> now to fill the quizz seriously. >> >> Anyway, here is an ebuild for euscan: >> http://git.iksaif.net/?p=portage.git;a=blob;f=app-portage/euscan/euscan-.ebuild;h=760f3ad025a009e3c336b1e7b08b31b72b6fec1f;hb=45b37c438f3cc20325d03ada666834b9db53d7b1 >> >> > for those who want to test. >> Writing one for euscanwww is probably less interesting since >> anyway you'll have to configure most of it by hand (database, cron, >> etc..) and it's clearly not something more than 2-3 people will >> install. >> > Hi, > > Open a bug, attach your ebuilds ( are there any releases besides the > git code? ) and I will take it from there Here it is: https://bugs.gentoo.org/show_bug.cgi?id=383937 There are no "released" version, and the project have no homepage (except http://euscan.iksaif.net). Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Tue, Sep 20, 2011 at 9:04 PM, Donnie Berkholz wrote: > On 10:00 Tue 20 Sep , Corentin Chary wrote: >> Could someone write ebuilds for euscan and euscanwww ? It should not >> take a lot of time, but my ebuilds skills are probably not good >> enought to do that. > > Sounds like good practice for when you become a Gentoo dev. =) Hehe :). Maybe someday, but honestly I don't have any time right now to fill the quizz seriously. Anyway, here is an ebuild for euscan: http://git.iksaif.net/?p=portage.git;a=blob;f=app-portage/euscan/euscan-.ebuild;h=760f3ad025a009e3c336b1e7b08b31b72b6fec1f;hb=45b37c438f3cc20325d03ada666834b9db53d7b1 for those who want to test. Writing one for euscanwww is probably less interesting since anyway you'll have to configure most of it by hand (database, cron, etc..) and it's clearly not something more than 2-3 people will install. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Mon, Sep 19, 2011 at 9:23 PM, Hans de Graaff wrote: > On Wed, 2011-08-31 at 15:41 +0200, Corentin Chary wrote: >> Hi, >> >> some news about euscan (still available at http://euscan.iksaif.net) >> >> - New design (yay !) >> - Atom feeds available for each herd/category/maintainer/package >> (http://euscan.iksaif.net/maintainers/59/feed/) >> - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check >> http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD >> ). >> >> Now, maybe we should find a way to integrate that with the GSoC >> statistic project and with http://packages.gentoo.org/ (like done at >> http://packages.qa.debian.org/p/php-net-ipv4.html ). A quick way would >> be to host euscan on a gentoo server, and add some webservices to >> publish the data in json or xml, then packages.gentoo.org and others >> could parse that and display it. > > One integration that might also be useful is what we track on the ruby > wiki currently: https://overlays.gentoo.org/proj/ruby/wiki/PendingBumps > which is a collection of quick notes on problems encountered while > bumping packages. The ability to attach notes to packages and to > acknowledge version bumps (and e.g. make them yellow instead of red) > could help with larger lists for herds. > > Kind regards, > > Hans > Yep, You can already use that page: http://euscan.iksaif.net/herds/ruby/ And ruby packages are well tracked by euscan thanks to the rubygem site handler, see how it helped to find new upstream versions while http://euscan.iksaif.net/herds/ruby/charts/versions-monthly.png Could someone write ebuilds for euscan and euscanwww ? It should not take a lot of time, but my ebuilds skills are probably not good enought to do that. Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] euscan proof of concept (like debian's uscan)
On Mon, Sep 19, 2011 at 10:53 AM, Michał Górny wrote: > On Mon, 19 Sep 2011 10:39:11 +0200 > Corentin Chary wrote: > >> ## Also update eix database, because we use eix internaly >> ## Bottleneck: disk and cpu >> ##Time: 30mn ~ 1h >> eix-update > > Using egencache to keep caches for overlays will make eix updates much > faster. > > Here's my code for it (it uses overlays in /usr/portage/local): > > cd /usr/portage/local && \ > for O in */; do > echo "${O}" > egencache --jobs=8 --update --update-use-local-desc --rsync \ > --repo=$(cat ${O}profiles/repo_name) > done Thanks, I'll try that. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] euscan proof of concept (like debian's uscan)
On Mon, Sep 19, 2011 at 9:35 AM, Dirkjan Ochtman wrote: > On Mon, Sep 19, 2011 at 00:27, "Paweł Hajdan, Jr." > wrote: >> Okay, I think this is pretty cool and we should find it a new home in >> the Gentoo infrastructure. >> >> I was thinking about http://qa-reports.gentoo.org/ with the repo at >> http://git.overlays.gentoo.org/gitweb/?p=proj/qa-scripts.git;a=summary >> >> I can act as a proxy committer and reviewer for that code. Could you >> break it up into some smaller parts (preferably backend first) and send >> to me for review (if you're interested)? >> >> How long does it take to generate the reports? > > +1 I think it would be good to run this on Gentoo infra, and I > wouldn't mind helping out. > > Bikeshedding: not sure "reports" is the best name for this, as reports > implies something more static? Here is how it works, each week I launch this script on lt server. I've got ~30 trees installed with layman. The server is an AMD X2 4600+ with 4GB of RAM and two 80GB HD in raid1 using ext4. My network bandwidth is 20Mbps down 1Mbps up. #!/bin/sh ## Setup some vars to use local portage tree export PATH=${HOME}/euscan/bin:${PATH} export PYTHONPATH=${HOME}/euscan/pym:${PYTHONPATH} export PORTAGE_CONFIGROOT=${HOME}/local export ROOT=${HOME}/local export EIX_CACHEFILE=${HOME}/local/var/cache/eix ## Go to euscanwww dir cd ${HOME}/euscan/euscanwww/ ## Update local trees ## Bottleneck: disk and network bandwidth ## Time: less than 30mn emerge --sync --root=${ROOT} --config-root=${PORTAGE_CONFIGROOT} ROOT="/" layman -S --config=${ROOT}/etc/layman/layman.cfg ## Also update eix database, because we use eix internaly ## Bottleneck: disk and cpu ##Time: 30mn ~ 1h eix-update ## Scan portage (packages, versions) ## Bottleneck: disk and cpu ## Time: < 15mn ## Note: this script uses eix to get a list of packages and versions python manage.py scan-portage --all --purge-versions --purge-packages ## Scan metadata (herds, maintainers, homepages, ...) ## Bottleneck: disk ## Time: 1h ~ 1h30 ## Note: this script uses gentoolkit to fetch metadata python manage.py scan-metadata --all --progress ## Scan uptsream packages ## Bottleneck: disk, network bandwidth and latency, cpu ## Time: up to 6h ## Note: euscan is called on each package. euscan has a slow startup caused by gentoolkit/portage. ## gparallel is used here to limit the load caused by euscan, and to launch up to 16 euscan instances at a time on this machine ## this part is the longest, but scale very well eix --only-names -x | gparallel --load 4 --jobs 800% euscan >> ${HOME}/logs/euscan-upstream.log python manage.py scan-upstream --feed --purge-versions < ${HOME}/logs/euscan-upstream.log ## Update counters (6) ## Time: some minutes ## Bottleneck: cpu ## Note: this script could probably be implemented faster using raw SQL queries python manage.py update-counters > Also not sure how much it has to do > with QA. > How much of it constitutes the backend, in your opinion? It seems > there are two parts, right now: > > 1. euscan script, to find new versions for a single package > 2. the django www app, including storage for the version data Yes, exactly. Here is how the tree is structured currently: euscan script bin/ -- contains the euscan python "binary" pym/ -- contains most of the code used by the euscan script pym/euscan/handlers -- contains specific site handlers (rubygems, pypi, pecl, pear, ..) euscanwww django app euscanwww/ -- contains all the stuff for the django application, all the django application needs is a working portage tree and euscan available in the $PATH > IMO it would be nice to have a somewhat generic REST-style service > exposing the data, and build a simple UI on top of that. In > particular, I have different ideas about what the UI should look like, > so it would be nice if different people could experiment (and/or > integrate in other services like znurt.org). I already added some very dummy json formating (note that it also exposes internal key id, which is bad, but I just wanted to experiment). All you need is to append "/json" to an url. For example: - http://euscan.iksaif.net/maintainers/4/json - http://euscan.iksaif.net/package/app-accessibility/brltty/json This could be a lot better, we just need to define an API and the implementation will be easy. A first step would be to make an ebuild for euscan, and another for euscanwww so that anyone can easilly install it and play with it. Feel free to ping me on irc, I'm on #gentoo-sunrise, my nickname is "iksaif". -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Tue, Sep 6, 2011 at 8:20 PM, Matt Turner wrote: > On Sun, Apr 3, 2011 at 1:20 PM, Corentin Chary > wrote: >> Hi again, > > Found a little problem: it's not finding a newer version of > wireless-regdb, which uses a date-based versioning system. If euscan > tried to view/parse the directory index where the distfiles are > located, it would find this new file. > > http://euscan.iksaif.net/package/net-wireless/wireless-regdb/ Hi, If you try to run euscan on that, you'll get: > * Url doesn't seems to depend on version: 20101124 not found in > http://wireless.kernel.org/download/wireless-regdb/wireless-regdb-2010.11.24.tar.bz2 What's happenning here is that the version can be found in the url, so even a directory scan won't find anything because it won't be able to match files. It would work correctly if the version was 2010.11.24 or the file was named wireless-regdb-20101124.tar.bz2. Unfortunatly, making that work would probably introduce a lot of false positive in other cases. Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Wed, Aug 31, 2011 at 3:41 PM, Corentin Chary wrote: > Hi, > > some news about euscan (still available at http://euscan.iksaif.net) > > - New design (yay !) > - Atom feeds available for each herd/category/maintainer/package > (http://euscan.iksaif.net/maintainers/59/feed/) > - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check > http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD > ). A quick example of what a custom site handler can bring (rubygems here): http://euscan.iksaif.net/categories/dev-ruby/charts/versions-weekly.png :) > Now, maybe we should find a way to integrate that with the GSoC > statistic project and with http://packages.gentoo.org/ (like done at > http://packages.qa.debian.org/p/php-net-ipv4.html ). A quick way would > be to host euscan on a gentoo server, and add some webservices to > publish the data in json or xml, then packages.gentoo.org and others > could parse that and display it. > > Thanks, > > -- > Corentin Chary > http://xf.iksaif.net > -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
2011/9/1 Tomáš Chvátal : > Dne 1.9.2011 09:55, Corentin Chary napsal(a): >>> >>> Btw I have feature request, could it remember the sorting method i set? >>> (so I don't have to click and reorder it every time i refresh) >> >> Per-page or globally ? >> > I would say globaly i smore sane here I did that per-page, as it was only one line to add, I'll try to do it globally later. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Thu, Sep 1, 2011 at 10:31 AM, Michał Górny wrote: > On Wed, 31 Aug 2011 15:41:51 +0200 > Corentin Chary wrote: > >> - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check >> http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD >> ). > > AFAICS that specific handlers are required to grab a list of versions. > Is it or will it be possible to add a kind of semi-handlers which would > just grab a list of all URIs (e.g. on github project) and let euscan > match them with SRC_URI? Yep, it's possible, you can do some specific stuff and import functions from euscan.helpers for euscan.handlers.generic to do a "generic" scan, then return a correct list of version. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
> Btw I have feature request, could it remember the sorting method i set? > (so I don't have to click and reorder it every time i refresh) Per-page or globally ? -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Thu, Sep 1, 2011 at 9:23 AM, Alex Legler wrote: > On Wednesday 31 August 2011 15:41:51 Corentin Chary wrote: >> Hi, >> >> some news about euscan (still available at http://euscan.iksaif.net) >> >> - New design (yay !) > > Glad you like it. Be sure to credit where you got it from, though. Sorry, that was done in the dev version, but I forgot to push it (http://euscan.iksaif.net/about/). Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
Hi, some news about euscan (still available at http://euscan.iksaif.net) - New design (yay !) - Atom feeds available for each herd/category/maintainer/package (http://euscan.iksaif.net/maintainers/59/feed/) - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD ). Now, maybe we should find a way to integrate that with the GSoC statistic project and with http://packages.gentoo.org/ (like done at http://packages.qa.debian.org/p/php-net-ipv4.html ). A quick way would be to host euscan on a gentoo server, and add some webservices to publish the data in json or xml, then packages.gentoo.org and others could parse that and display it. Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
New dynamic charts: - http://euscan.iksaif.net/maintainers/2/charts/versions-weekly.png - http://euscan.iksaif.net/categories/app-accessibility/charts/packages-weekly.png - http://euscan.iksaif.net/categories/app-accessibility/charts/packages-weekly-small.png Feel free to use them in a plasma-applet, or whatever you want !
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Thu, Apr 28, 2011 at 2:34 PM, Corentin Chary wrote: > 2011/4/27 Tomáš Chvátal : >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> Dne 27.4.2011 10:04, Corentin Chary napsal(a): >>> I updated http://euscan.iksaif.net yesterday. >>> >>> I also added some overlays, and I now use local portage portage trees >>> instead of system trees (a script to do that is available in >>> euscanwww/script). >>> >>> New features: >>> - Handle overlays correctly (special column for overlays) >>> - Add some charts (small bar charts in table, pies at the bottom) >>> - Less false positives with euscan >>> >>> I'll try to add "all time" line charts this week. >>> >>> Thanks, >> Found another issue, package stays there even if it was removed from the >> portage tree (kde-misc/plasmaboard is a good example). > > Good catch, I'll add a --purge option to scan-portage. Done, http://git.iksaif.net/?p=euscan.git;a=commitdiff;h=120ae425af82061fee4e0d7f18104595c551a144;hp=244e7d64fdc6ddeb31ad54dfcc415c50672bdebb -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
2011/4/27 Tomáš Chvátal : > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Dne 27.4.2011 10:04, Corentin Chary napsal(a): >> I updated http://euscan.iksaif.net yesterday. >> >> I also added some overlays, and I now use local portage portage trees >> instead of system trees (a script to do that is available in >> euscanwww/script). >> >> New features: >> - Handle overlays correctly (special column for overlays) >> - Add some charts (small bar charts in table, pies at the bottom) >> - Less false positives with euscan >> >> I'll try to add "all time" line charts this week. >> >> Thanks, > Found another issue, package stays there even if it was removed from the > portage tree (kde-misc/plasmaboard is a good example). Good catch, I'll add a --purge option to scan-portage. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
I updated http://euscan.iksaif.net yesterday. I also added some overlays, and I now use local portage portage trees instead of system trees (a script to do that is available in euscanwww/script). New features: - Handle overlays correctly (special column for overlays) - Add some charts (small bar charts in table, pies at the bottom) - Less false positives with euscan I'll try to add "all time" line charts this week. Thanks, -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
2011/4/19 Tomáš Chvátal : > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Dne 16.4.2011 08:29, Corentin Chary napsal(a): >> New website is up and running at >> >> http://euscan.iksaif.net/ >> >> The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary >> >> TODO: >> - Make some charts to see how it's going >> - Finish the "scan my world" feature >> - Add a way to subsribe to herds/maintainer/packages in order to >> receive weekly/monthly reports >> >> I'll gladly accept any patch ! :) >> > Hello found another issue, > > http://euscan.iksaif.net/package/media-video/kmplayer/ > > 0.11.2c IS newer than 0.11.2 :) > > Cheers Should be fixed this evening. I'll now use portage.vercmp() by default, and fallback to pkg_resources. Thanks for the report. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Tue, Apr 19, 2011 at 9:29 AM, Dirkjan Ochtman wrote: > On Wed, Apr 13, 2011 at 08:47, Corentin Chary > wrote: >> Better, what about a per-herd/per-maintainer rss feed ? > > This is what I'd most want, would be very cool to have. > > Perhaps it would be possible for some classes of packages to fine-tune > the algorithm, e.g. check PyPI for Python things, PECL for PHP things, > etc. > Thanks, Added to TODO, but if you got some time, feel free to give it a try :). -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
> The list is missing amarok-2.4.0.90. You need to look both on stable and > unstable trees to find all versions. > ftp://ftp.kde.org/pub/kde/unstable/amarok/2.4.0.90/src Hum, it's hard to guess the "unstable" url from the "stable" one automatically. But maybe a custom "site handler" for ftp.kde.org would do it. > http://euscan.iksaif.net/package/dev-db/mariadb/ > > You're missing quite a few versions of mariadb. They can be found in the > official mysql overlay ( > http://git.overlays.gentoo.org/gitweb/?p=proj/mysql.git;a=tree;f=dev-db/mariadb > ) I'll add some "officials" overlay soon. > http://downloads.askmonty.org/mariadb/5.1/ > http://downloads.askmonty.org/mariadb/5.2/ The 5.2 is hard to find, because all mirrors redirect to "http://downloads.askmonty.org/mariadb/"; when trying to list the top directory. And I can't guess the version from this page. Brute force would be able to find it, if 5.2.0 archive was still available. > http://euscan.iksaif.net/package/dev-db/mysql/ > > You're missing many of the new releases of mysql. MySQL case may be easier to deal with than MariaDB; I added these to my TODO. I'll try to find a generic way, but I don't know if there is really one. Thanks ! -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Mon, Apr 18, 2011 at 7:45 PM, Donnie Berkholz wrote: > On 15:48 Mon 18 Apr , Corentin Chary wrote: >> On Mon, Apr 18, 2011 at 3:32 PM, Donnie Berkholz >> wrote: >> > - Another useful thing would be a way to supply CPV tokens for any >> > "unstable" upstream series that we'll never add to the tree. >> >> There is already some kind of blacklist in euscan. Could you provide >> examples so I can implement that ? > > Sorry, forgot about this bit. I'm just talking about standard package > tokens, the same kind used in package.mask or wherever: > > =x11-drivers/xf86-video-intel-2.14.90* >>=x11-base/xorg-server-1.10.900 > I'll try to implement that, Thanks for the suggestion. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Mon, Apr 18, 2011 at 5:22 PM, Daniel Pielmeier wrote: > Corentin Chary schrieb am 18.04.2011 11:05: >> >> Fyi: I ran a new scan this morning. It only took 3 hours ! >> >> GNU's Parallel is really a great tool: >> `python manage.py list-packages | gparallel --eta --progress --jobs >> 400% euscan | python manage.py scan-upstream --feed > /dev/null` >> >> I'll try to implement charts as soon as I've got enought data. > > I really like it. Keep up the good work. It gives a warm and fuzzy > feeling if there are no unpackaged releases for the own packages :) > > There is one issue I recognized. How do you get the links for the > unpackaged upstream versions and do you verify if the listed tarballs > exist? I am asking because for app-editors/bluefish [1] there are eleven > unpackaged versions listed but only sources for the three 2.0.3 releases > exist and there are no releases for 2.1.x and 2.2.x. > > http://euscan.iksaif.net/package/app-editors/bluefish/ For bluefish, it's because when you try to download http://www.bennewitz.com/bluefish/stable/source/bluefish-2.2.3.tar.bz2 it redirects you to an earlier version. Fixed here: http://git.iksaif.net/?p=euscan.git;a=commitdiff;h=d74b2c505619306ab1ce3d4af3f66084b4faeed9;hp=a7a15c0ac72178bf2f3a73f94a70113ed9856e5c Thanks ! -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Mon, Apr 18, 2011 at 1:48 PM, Corentin Chary wrote: > On Mon, Apr 18, 2011 at 3:32 PM, Donnie Berkholz wrote: >> On 06:29 Sat 16 Apr , Corentin Chary wrote: >>> New website is up and running at >>> >>> http://euscan.iksaif.net/ >>> >>> The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary >>> >>> TODO: >>> - Make some charts to see how it's going >>> - Finish the "scan my world" feature >>> - Add a way to subsribe to herds/maintainer/packages in order to >>> receive weekly/monthly reports >>> >>> I'll gladly accept any patch ! :) >> >> This is cool! I just looked through the x11-team packages and it seems >> very useful already. I have a few suggestions if you have time. =) >> >> - Some teams have "official" overlays. Supporting those as an additional >> source of ebuilds would be pretty nice. > > I'll just checkout these trees on my machine (and only enable them for > euscan). > euscan use portage, gentoolkit and eix internaly, so overlays are not > a problem :). > > Where could I get a list of "official" overlays that should be added ? > >> - Another useful thing would be a way to supply CPV tokens for any >> "unstable" upstream series that we'll never add to the tree. > > There is already some kind of blacklist in euscan. Could you provide > examples so I can implement that ? > >> - This looks like a problem (a versioned tarball named cairo-5c): >> http://euscan.iksaif.net/package/x11-libs/cairo/ > > Hum ... yes, I'll try to use better regex :). Fixed in http://git.iksaif.net/?p=euscan.git;a=commitdiff;h=a7a15c0ac72178bf2f3a73f94a70113ed9856e5c;hp=e5278e0e0f8cf617dff15f84540d0d6604081e82 -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Mon, Apr 18, 2011 at 3:32 PM, Donnie Berkholz wrote: > On 06:29 Sat 16 Apr , Corentin Chary wrote: >> New website is up and running at >> >> http://euscan.iksaif.net/ >> >> The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary >> >> TODO: >> - Make some charts to see how it's going >> - Finish the "scan my world" feature >> - Add a way to subsribe to herds/maintainer/packages in order to >> receive weekly/monthly reports >> >> I'll gladly accept any patch ! :) > > This is cool! I just looked through the x11-team packages and it seems > very useful already. I have a few suggestions if you have time. =) > > - Some teams have "official" overlays. Supporting those as an additional > source of ebuilds would be pretty nice. I'll just checkout these trees on my machine (and only enable them for euscan). euscan use portage, gentoolkit and eix internaly, so overlays are not a problem :). Where could I get a list of "official" overlays that should be added ? > - Another useful thing would be a way to supply CPV tokens for any > "unstable" upstream series that we'll never add to the tree. There is already some kind of blacklist in euscan. Could you provide examples so I can implement that ? > - This looks like a problem (a versioned tarball named cairo-5c): > http://euscan.iksaif.net/package/x11-libs/cairo/ Hum ... yes, I'll try to use better regex :). Thanks ! -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Mon, Apr 18, 2011 at 10:26 AM, Marijn wrote: > Nice work! Thanks ! Fyi: I ran a new scan this morning. It only took 3 hours ! GNU's Parallel is really a great tool: `python manage.py list-packages | gparallel --eta --progress --jobs 400% euscan | python manage.py scan-upstream --feed > /dev/null` I'll try to implement charts as soon as I've got enought data. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
New website is up and running at http://euscan.iksaif.net/ The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary TODO: - Make some charts to see how it's going - Finish the "scan my world" feature - Add a way to subsribe to herds/maintainer/packages in order to receive weekly/monthly reports I'll gladly accept any patch ! :) -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Wed, Apr 13, 2011 at 4:58 PM, Corentin Chary wrote: >> Hi Corentin, >> >> Do you have a public repo for your django code/site available? I would >> really enjoy taking a look at what you have here, sounds cool. >> >> Thanks, >> Matt Just pushed scan-upstream command (with --parallel support !). Now, it's time to do a full scan and implement the views. I staying on #gentoo-unregistered (iksaif) if someone want to get involved in euscan :). -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
> Hi Corentin, > > Do you have a public repo for your django code/site available? I would > really enjoy taking a look at what you have here, sounds cool. > > Thanks, > Matt Hi, Yes, my git repo is here: http://git.iksaif.net/?p=euscan.git;a=summary Current dependencies are: eix-0.22.5, gentoolkit-0.3.0 and portage of course. I didn't took the time to write any kind of documentation, but here is what you'll find in the repo: - euscan standalone utility - euscanwww Currently euscanwww only have basic templates, models and view. But you can alreay run these commands to fill the database: $ emerge --sync $ eix-update $ python manage.py scan-portage --all $ python manage.py scan-metadata --all I'll implement scan-upstream soon. The old index.php is still available in earlier revisions. -- Corentin Chary http://xf.iksaif.net
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Sun, Apr 10, 2011 at 8:43 AM, Hans de Graaff wrote: > On Sun, 2011-04-03 at 17:20 +0000, Corentin Chary wrote: > >> misc: >> - automatic bug report >> - automatic email report for maintainer/herds > > I'm not sure if this makes sense. For example, it gets dev-lang/ruby > wrong, thinking that our old patch files are missing versions. Also, for > exiftool I only add production releases, so there will almost always be > newer upstream intermediate releases. My point here is that there are > often additional considerations in considering new releases, so fully > automating this is going to be hard. Perhaps a weekly/monthly opt-in > mail to the herd/maintainer with an overview would be useful? Better, what about a per-herd/per-maintainer rss feed ? > The website is nice to have as another source of scanning for updates, > bookmarked. The current site won't be updated, I'm working on a new django-based website that should be up and running next week. The url will probably be euscan.iksaif.net, but I'll post it here. -- Corentin Chary http://xf.iksaif.net
[gentoo-dev] Re: euscan proof of concept (like debian's uscan)
Hi again, I found the time to work a little more on euscan: - Git tree: http://git.iksaif.net/?p=euscan.git - Demo web interface: http://xf.iksaif.net/dev/.euscan/web/ To feed the web interface, I created a quick&dirty euscan-update. It currently depends on eix-0.22.5 and gentoolkit-0.3.0. I launched a full portage scan yesterday, and as you can see on web interface, it seems to work. You can browse by category, herd, and maintainer. The scan is not finished yet (currently scanning net-misc) and keep in mind that euscan doesn't support mirror:// urls (although, it would be easy to support some of them). But there is still a lot to do to make something useful with that, and I said before, I don't have a lot of time allocated to that project, so any help would be great. Quick TODO and ideas: euscan: - clean euscan and add options parsing capabilities, with tons of options - support mirror:// (when possible) - fix known bugs (see BUGS) euscan-update: - clean - optimize and parallelize - remove gentoolkit dependency, handle overlays, handle cumulative scans web interface: - display all kind of informations (metadatas, keywords, etc...) - create charts and statistics - mark/report false positives - display euscan-update status - display last euscan output message - show when euscan can/cannot guess upstream version for a given package misc: - automatic bug report - automatic email report for maintainer/herds
Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)
On Sat, Mar 26, 2011 at 5:35 PM, Christian Faulhammer wrote: > Hi, > > Corentin Chary : >> I started this mostly to see if it was possible, and I don't know if >> i'll have the time to continue to work on that project, but I think >> gentoo really needs an automated way to detect outdated packages. >> This could also be a 2011 GSOC project (finishing euscan, and adding a >> web based interface to browse the results). > > Have you applied for a place in GSoC? > > V-Li I can't apply to GSoC, I was suggesting that another student could work on that. -- Corentin Chary http://xf.iksaif.net
[gentoo-dev] euscan proof of concept (like debian's uscan)
Hi, I recently started working on a small gentoo utility named "euscan" (for Ebuild Upstream Scan) For those who don't know debian's uscan, it allows to scan upstream for new versions. It's used by packages.qa.debian.org (example: http://packages.qa.debian.org/p/php-net-ipv4.html ). It's available at: http://xf.iksaif.net/bordel/euscan Currently, it uses two heuristics to find new versions, both based on SRC_URI and PV: - Directory scanning: scan directories to find files with newer version - Brute Force: generate new possible versions, and try to download files Note that it also works when only a part of the version is available in the url. I think that it would be great to have these informations on http://packages.gentoo.org/ and/or unofficial http://gentoo-portage.com/ website. We could also add the ability to browse packages by maintainer to help them see if they have any outdated package. I started this mostly to see if it was possible, and I don't know if i'll have the time to continue to work on that project, but I think gentoo really needs an automated way to detect outdated packages. This could also be a 2011 GSOC project (finishing euscan, and adding a web based interface to browse the results). Examples: $ ./euscan PEAR-Validate-0.8.3 Package: dev-php/PEAR-Validate-0.8.3 Herd: php Maintainer: none Location: /usr/portage/dev-php/PEAR-Validate * Scanning: http://pear.php.net/get/Validate-${PV}.tgz * Scanning: http://pear.php.net/get * Generating version from 0.8.3 * Brute forcing: http://pear.php.net/get/Validate-${PV}.tgz * Trying: http://pear.php.net/get/Validate-0.8.4.tgz ... [ ok ] * Trying: http://pear.php.net/get/Validate-0.8.5.tgz ... [ !! ] * Trying: http://pear.php.net/get/Validate-0.9.0.tgz ... [ !! ] * Trying: http://pear.php.net/get/Validate-0.10.0.tgz ... [ !! ] * Trying: http://pear.php.net/get/Validate-0.8.6.tgz ... [ !! ] New Upstream Version: 0.8.4 http://pear.php.net/get/Validate-0.8.4.tgz $ ./euscan icu4j-4.4.2 Package: dev-java/icu4j-4.4.2 Herd: java Maintainer: none Location: /usr/portage/dev-java/icu4j * Scanning: http://download.icu-project.org/files/icu4j/${PV}/icu4j-4_4_2-src.jar * Scanning: http://download.icu-project.org/files/icu4j New Upstream Version: 4.5.1 http://download.icu-project.org/files/icu4j/4.5.1/ New Upstream Version: 4.5.2 http://download.icu-project.org/files/icu4j/4.5.2/ New Upstream Version: 4.6 http://download.icu-project.org/files/icu4j/4.6/ New Upstream Version: 4.6rc2 http://download.icu-project.org/files/icu4j/4.6rc2/ New Upstream Version: 4.7.1 http://download.icu-project.org/files/icu4j/4.7.1/ New Upstream Version: 4.6.1 http://download.icu-project.org/files/icu4j/4.6.1/ $ ./euscan IceE-1.3.0-r1 Package: dev-cpp/IceE-1.3.0-r1 Herd: no-herd Maintainer: maintainer-nee...@gentoo.org Description: The Internet Communications Engine (Ice) is a modern object-oriented middleware with support for C++, .NET, Java, Python, Ruby, and PHP Location: /usr/portage/dev-cpp/IceE * Scanning: http://www.zeroc.com/download/IceE/${0}.${1}//IceE-${PV}-linux.tar.gz * Scanning: http://www.zeroc.com/download/IceE * Generating version from 1.3.0 * Brute forcing: http://www.zeroc.com/download/IceE/${0}.${1}//IceE-${PV}-linux.tar.gz * Trying: http://www.zeroc.com/download/IceE/1.3//IceE-1.3.1-linux.tar.gz ... [ !! ] * Trying: http://www.zeroc.com/download/IceE/1.3//IceE-1.3.2-linux.tar.gz ... [ !! ] * Trying: http://www.zeroc.com/download/IceE/1.4//IceE-1.4.0-linux.tar.gz ... [ !! ] * Trying: http://www.zeroc.com/download/IceE/1.5//IceE-1.5.0-linux.tar.gz ... [ !! ] Thanks, -- Corentin Chary http://xf.iksaif.net