I agree with Elaine I can't get rsync through the firewall at work. Not even tunneled. For CPAN I use CPAN::Mini. It uses http and it does the job though it does force the local CPAN to blead. My local solution to other things we need such as Blastwave (we run Solaris) I have a special squid proxy with restrictive acls lots of disk space and long retention. That means we only download any package once: what needed when needed. That does create a problem of using bandwidth during the day but it works out ok in the end.
Rsync is better than having to hack reverse caching proxies for each site of interest. Sent from my BlackBerry® smartphone with Nextel Direct Connect -----Original Message----- From: Elaine Ashton <eash...@mac.com> Date: Sat, 27 Mar 2010 21:38:16 To: Arthur Corliss<corl...@digitalmages.com> Cc: Elaine Ashton<eash...@mac.com>; <cpan-work...@perl.org>; <module-authors@perl.org> Subject: Re: Trimming the CPAN - "Automatic Purging" On Mar 27, 2010, at 2:52 PM, Arthur Corliss wrote: > > Don't be such an arrogant prick. You guys made baseless assumptions about > people's experience with storage management in an attempt to diregard their > opinions. That's being a dick by any metric. Actually, I thought I was merely offering my opinion both as the sysadmin for the canonical CPAN mothership and as an end-user. If that makes me a prick, well, I suppose I should go out and buy one :) > I think I was quite explicit in saying that efficiencies should be pursued > in multiple areas, but the predominant bitch I took away from your thread > dealt with the burden of synchronizing mirrors. What's the easiest way to > address that pain? I don't believe it's your method. I'd look into the > size issue *after* you address the incredible inefficiencies of a simple > rsync. And you're disregarding a considerable problem that rsync is a well-established tool for mirroring that is easy to use and works on a very wide range of platforms. Asking mirror ops to adopt a new tool for mirroring one mirror, when they often have several or more, likely won't be met with much enthusiasm and would create two tiers of CPAN mirrors, those using rsync and those not, which would not only complicate something which should remain simple but, again, doesn't address the size of the archive and the multitude of small files that are always a consideration no matter what you're serving them up with. > Rsync is an excellent tool for smaller file sets. I use it to sync my own > mirrors, those mirrors are typically ~10k files. Am I surprised that it > doesn't scale when you're stat'ing every single file? No. Which is why > alternatives should be considered. A simple FTP client playing a > transaction log forward is trivial. FTP? It's 2010 and very few corp firewalls allow ftp in or out. I can't remember the last time I even used ftp come to think of it. I had to go through 2 layers of network red tape just to get rsync for a particular system I wanted to mirror CPAN to at work. Asking for FTP would have been met with a big no or a cackle, depending on which of the nyetwork masters got the request first. > Try doing a simple cost-benefit analysis. What you guys are proposing will > help. But not as much as simpler alternatives. Like replacing rsync with a > perl script and modifying PAUSE to log the transactions. How is replacing rsync, a standard and widely used tool, simpler for mirror ops? I suppose I don't understand the opposition to trimming off the obvious cruft on CPAN to lighten the load when BackPAN exists to archive them. There is already CPAN::Mini (which was created back when CPAN was an ever-so-tiny 1.2GB) so it's not as though lightening the load is a new idea or an unwelcome one. e.
