Re: Trimming the CPAN - Automatic Purging
On Sun, Mar 28, 2010 at 07:28:48AM -0700, dhu...@hudes.org wrote: The danger in a CPAN::Mini and in removing old versions is that one is assuming that the latest and greatest is the one to use. This is false. And this is why I run cp5.6.2an.barnyard.co.uk etc. It wouldn't be difficult for someone to take my code and customise it further to, eg, also pin a few modules that rely on the particular versions of third-party libraries that you use. -- David Cantrell | Bourgeois reactionary pig Eye have a spelling chequer / It came with my pea sea It planely marques four my revue / Miss Steaks eye kin knot sea. Eye strike a quay and type a word / And weight for it to say Weather eye am wrong oar write / It shows me strait a weigh.
Re: Trimming the CPAN - Automatic Purging
On Sun, Mar 28, 2010 at 06:04:03PM -0400, David Golden wrote: As always with perl, it depends. They are laid out just as a normal CPAN repository, so if you have one in your urllist, something specified as author/distribution.tar.gz might well resolve. Not just might well resolve. It *will* work. If you use one of my cpXXXan mirrors, you're hitting a BackPAN mirror with a custom index. *However*, they don't necessarily have up-to-date index files. Compare timestamps on 02packages.details.txt Indeed. I don't imagine that that would be hard for Andreas to keep in sync! -- David Cantrell | even more awesome than a panda-fur coat IMO, the primary historical significance of Unix is that it marks the time in computer history where CPUs became so cheap that it was possible to build an operating system without adult supervision. -- Russ Holsclaw in a.f.c
Re: Tidy up your PAUSE directories
brian d foy wrote: It's time for Spring cleaning again. If you have ancient versions of modules sitting around in your PAUSE directory, consider letting them retire to BackPAN (http://backpan.cpan.org). They don't disappear from the world, but they don't inflate CPAN either. You don't have to do anything, but many mirror operators might be happy that you did. :) Just my one point nine periodic cents: If you got ancient modules sitting around that you wont be updating anymore be good and ask around if someone wants to take over. If the module is simple enough (and possibly an Acme module), you could also ask in beginners: Maintaining a simple module with an existing user base is in my opinion a good learning experience! LG Rene Note: I've taken over three Acme-modules. Among them is Acme-AutoColor. If you lack any non-standard colors (i just added octarine), more features or something like that, just mail me. The other two, Acme::Innuendo and Acme::Mobile::Therbligs need updating too (working on it), ideas are welcome as well.
Re: Trimming the CPAN - Automatic Purging
On Tue, 30 Mar 2010, Matija Grabnar wrote: Er, not exactly. Read http://www.cvsup.org/howsofast.html I had read http://www.cvsup.org/faq.html#features item #3. From what I can see, cvsup uses the rsync algorithm on a file-by-file basis (it uses just the differential send part of the rsync algorithm). It doesn't rsync the whole tree, which was what I understood to be the original problem (wasn't the complaint about the flood of stats?). Sounds like I may have interpreted the FAQ incorrectly, then. Thanks for pointing that out. I have a few question, though: the explanation says: At the same time, the Tree Differ generates a list of the server's files. That seems to infer that it's doing the exact same thing as rsync, so all the stats are still present on the server, right? Nowhere do I see it mentioning that the daemon is maintaining state between requests. The primary speed-ups (beyond special file update handling) is better use of bidirectional bandwidth. Do you have access to a cvsup server so you can verify its behavior? So if you want to make a tool that works fine for large mirrors, your priority apparently should be to reduce the lots of stats part which is used to determine exactly what files need to be considered for checking. (Rsync already makes sure all the *other* I/O operations are minimized). Agreed. Now the key, as I see it, is that unlike all the other use cases where rsync is used, large mirrors are likely to have their directories directly transfered from another mirror. So, the client that pulled the tree update down could store a list of changed files, and the server could then just use that list to determine which files need to be synced to the downstream mirror. (Sure, the original site has to generate the list, but if they use a tool like PAUSE to upload the files, that shouldn't be hard to do). Agreed, but I'm not sure we've gotten past the stat storm on the server, though. --Arthur Corliss Live Free or Die
Re: Trimming the CPAN - Automatic Purging
On Tue, 30 Mar 2010, Rene Schickbauer wrote: snip This could work like any modern, distributed version control systems. That way, the user would also be able to apply local patches and/or deciding which changesets to pull in from the main server. Or have a complete, local mirror and one for the production systems where he/she pulls in changes after they have been reviewed. NOW its time to kick my butt, if you want to. :-) No one can accuse you of not being ambitious. It's a neat idea, but definitely an involved solution. While it could solve a lot of problems I think the human component is going to be your biggest obstacle. As we've seen from the reaction to the heretical notion of ditching rsync I have to imagine getting everyone to ditch their favorite RCS tool would be even worse. Basically, we should just all get onboard with git (disclaimer: I don't use git myself, so my understanding may be deficient), a decentralized distributed RCS. And have developers periodically merge their branches. Tough sell. It probably would solve a bunch of issues, but you're treading into vi versus emacs territory. ;-) --Arthur Corliss Live Free or Die
Re: Trimming the CPAN - Automatic Purging
On Sun, Mar 28, 2010 at 2:32 PM, Elaine Ashton eash...@mac.com wrote: On Mar 28, 2010, at 12:48 PM, Randy Kobes wrote: Has some sort of disk quota system for CPAN author accounts ever been considered? Not specifically, no, at least not that I'm aware of. That would have to be implemented on PAUSE and quotas frequently end up not solving the real problem and create a headache both for the sysadmin and the users. new proposal: Make modules pay rent in order to remain on a mirror. Rent could be in the form of actual user interest, or good reviews. Use as a dependency could count as rent. Or simple downloading. A mirror server that functioned more as a cache than a mirror would also work: only the files that are actually requested need be stored, as long as the mirror server knows how to get something else if requested. If the root cause of The Pain turns out to be full mirroring then do partial mirroring, and automate the partition with a policy instead of trying to plan the partition. -- question doubt
