From: perl-win32-users-boun...@listserv.activestate.com [mailto:perl-win32-users-boun...@listserv.activestate.com] On Behalf Of Tobias Hoellrich Sent: 21 November 2012 14:45 To: Daniel Burgaud Cc: Perl-Win32-Users Subject: RE: In need to efficiently retrieve HTTP
> Hi Daniel - > http://stackoverflow.com/questions/6473785/improving-lwpsimple-perl-performance > should bring you > on the right track. If you want to avoid multiple threads/processes then > "Keep-Alive" most likely will give > you the biggest performance gain. The TCP connection stays open after you > made the first request and you can > send more requests over the same connection. > > Cheers - T > > From: perl-win32-users-boun...@listserv.activestate.com > [mailto:perl-win32-users- > boun...@listserv.activestate.com] On Behalf Of Daniel Burgaud > Sent: Wednesday, November 21, 2012 7:37 AM > To: Perl-Win32-Users > Subject: In need to efficiently retrieve HTTP > > Hi All > > Basically, I need to fetch thousands and thousands of small 200~4000 byte > files (map files). Opening and > closing a socket connection is too slow a process so much so a single file > would take as much as 10 seconds! > > Is there any perl script out there that can be used to efficiently fetch HTTP > files? A non-closing script? I > sure wanna do Threaded on this one, but my win32 perl does not have thread > capability. > > Or perhaps, is there a win32 API for this purpose and can be called from a > perl script? That should probably be 'keep_alive', to be pedantic. However, that will only work as long as the files are to be downloaded from the same site (or a small number of sites), and that site doesn't drop the connection, which it may well do if it doesn't like the idea of 'thousands and thousands' of its files being hoovered. If the OP has permission from the site owner, then that's fine, otherwise what he is proposing could be considered rude, or even abusive. Were the OP downloading from google maps, for example, he might find that they don't like the idea of anybody downloading a lot of files in a short period (see https://developers.google.com/maps/faq#usagelimits). HTH -- Brian Raven ________________________________ Please consider the environment before printing this e-mail. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error, please advise the sender immediately by reply e-mail and delete this message and any attachments without retaining a copy. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. _______________________________________________ Perl-Win32-Users mailing list Perl-Win32-Users@listserv.ActiveState.com To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
