OK, aqui vai o spoiler: estou fazendo benchmark (de overhead) de todos os HTTP agents que conheço. E o LWP está *MUITO* feio na fita: https://metacpan.org/module/AnyEvent::Net::Curl::Queued#OVERHEAD
ABS() 2011/10/20 Rafael Prenzier <[email protected]> > inclusive, estou lendo a Doc aqui no Cpan e o exemplo que ele dá é > justamente paralelizando Downloads > > http://search.cpan.org/~dlux/Parallel-ForkManager-0.7.5/ForkManager.pm > > use LWP::Simple; > use Parallel::ForkManager; > > ... > > @links=( > ["http://www.foo.bar/rulez.data","rulez_data.txt";], > ["http://new.host/more_data.doc","more_data.doc";], > ... > ); > > ... > > # Max 30 processes for parallel download > my $pm = new Parallel::ForkManager(30); > > foreach my $linkarray (@links) { > $pm->start and next; # do the fork > > my ($link,$fn) = @$linkarray; > warn "Cannot get $fn from $link" > if getstore($link,$fn) != RC_OK; > > $pm->finish; # do the exit in the child process > } > $pm->wait_all_children; > > > Abraço > > 2011/10/20 Rafael Prenzier <[email protected]> > >> Hummmm se você usar o Parallel::ForkManager com qualquer método de >> Download, resolve? >> >> 2011/10/20 Stanislaw Pusep <[email protected]> >> >>> Perlssoal, alguém conhece agente(s) HTTP, preferencialmente em Perl, e >>> preferencialmente capazes de paralelizar downloads, que não sejam estes? >>> >>> - LWP::UserAgent >>> - HTTP::Tiny >>> - HTTP::Lite >>> - AnyEvent::Curl::Multi >>> - lftp >>> - curl >>> - wget >>> >>> P.S. - vulgo: "complete a lista" :P >>> >>> ABS() >>> >>> >>> =begin disclaimer >>> Sao Paulo Perl Mongers: http://sao-paulo.pm.org/ >>> SaoPaulo-pm mailing list: [email protected] >>> L<http://mail.pm.org/mailman/listinfo/saopaulo-pm> >>> =end disclaimer >>> >>> >> >> >> -- >> Rafael Prenzier dos Santos | Engenheiro e Perl Monger >> >> [email protected] >> [email protected] >> <[email protected]> >> (13) 8131-8212 >> >> <http://gplus.to/prenzier> <http://www.facebook.com/rafaelprenzier> >> <http://twitter.com/prenzier/> >> <http://www.orkut.com.br/Main#Profile?uid=13496291347057199080> >> <http://cid-eacb613d80ce8dba.profile.live.com/> >> <http://www.delicious.com/rafaelprenzier/> >> <http://www.google.com/profiles/rafaelprenzier> >> <http://br.linkedin.com/pub/rafael-prenzier-dos-santos/29/6a2/478> >> <http://www.youtube.com/prenzier> >> <http://www.diigo.com/user/prenzier> >> <http://www.shelfari.com/rprenzier/shelf> >> <http://flavors.me/prenzier> >> <http://www.diigo.com/user/prenzier> >> >> >> > > > -- > Rafael Prenzier dos Santos | Engenheiro e Perl Monger > > [email protected] > [email protected] > <[email protected]> > (13) 8131-8212 > > <http://gplus.to/prenzier> <http://www.facebook.com/rafaelprenzier> > <http://twitter.com/prenzier/> > <http://www.orkut.com.br/Main#Profile?uid=13496291347057199080> > <http://cid-eacb613d80ce8dba.profile.live.com/> > <http://www.delicious.com/rafaelprenzier/> > <http://www.google.com/profiles/rafaelprenzier> > <http://br.linkedin.com/pub/rafael-prenzier-dos-santos/29/6a2/478> > <http://www.youtube.com/prenzier> > <http://www.diigo.com/user/prenzier> > <http://www.shelfari.com/rprenzier/shelf> > <http://flavors.me/prenzier> > <http://www.diigo.com/user/prenzier> > > > > =begin disclaimer > Sao Paulo Perl Mongers: http://sao-paulo.pm.org/ > SaoPaulo-pm mailing list: [email protected] > L<http://mail.pm.org/mailman/listinfo/saopaulo-pm> > =end disclaimer > >
=begin disclaimer Sao Paulo Perl Mongers: http://sao-paulo.pm.org/ SaoPaulo-pm mailing list: [email protected] L<http://mail.pm.org/mailman/listinfo/saopaulo-pm> =end disclaimer
