Parallel UserAgent
What's the state of play with LWP::Parallel::UserAgent? It hasn't worked for sometime and it hasn't been worked on either, yet it hasn't been deleted from CPAN. Is it likely to be fixed soon? -Nigel
LWP::Parallel::UserAgent default headers patch
Hello!
I was playing around LWP::Parallel::UserAgent and noticed that if I set
default_headers() which is LWP::UserAgent's method, the parallel UA
never used them for requests.
This little patch ensures that the default headers are added to each
parallel ua's request.
P.Krumins
--- UserAgent.original.pm 2004-02-10 17:19:20.0 +0200
+++ UserAgent.pm2007-07-04 06:03:24.890625000 +0300
@@ -1493,6 +1493,13 @@
$request-init_header('Range' = bytes=0-$max_size) if $max_size;
$cookie_jar-add_cookie_header($request) if $cookie_jar;
+my $def_headers = $self-default_headers;
+if ($def_headers) {
+for my $h ($def_headers-header_field_names) {
+$request-init_header($h = [$def_headers-header($h)]);
+}
+}
+
# Transfer some attributes to the protocol object
$protocol-parse_head($parse_head);
$protocol-max_size($max_size);
LWP::Parallel::UserAgent; HTTP 1.1, order of requests etc.
Hello! I am using LWP::Parallel::UserAgent for writing some test
scripts. Some questions have come up:
1. Is it possible to make it send HTTP/1.1 as protocol string instead
of HTTP/1.0 (the server in the other end checks the HTTP header for
1.1)? I have tried
my $req = HTTP::Request-new(POST, $url, $hdrs, $postdata);
$req-protocol('HTTP/1.1');
print $req-as_string;
and it prints the request header as if it was 1.1, but when I check with
Ethereal it seems that HTTP/1.0 is sent!
2. The documentation states that the replies collected when doing
wait() are sorted according to URL. What happens if all the requests
are using the same URL (I am connecting to a special server that
does some processing and constructs its replies based on the XML body
of the POSTs my Perl script is sending)? I am using
$pua-in_order (1); # handle requests in order of registration
$pua-nonblock (1); # non-blocking
3. Should I use LWPng instead or is it still to alpha:ish?
Best regards,
--
Erik Alapää
Software Engineer M. Sc., Lic. Ind. Math.
Aurorum Science Park 8
SE 997 75 Luleå
Sweden
+46 70 535 17 14 (mobile)
+46 920 75510 (fax)
http://www.operax.com
LWP::Parallel::UserAgent and cookies
Hello everyone!
I have a question about LWP::Parallel::UserAgent and cookie jars using
HTTP::Cookies.
Can I use cookie jars with LWP::Parallel::UserAgent? Can it be used in
the same way as I do with LWP::UserAgent?
I have a script whick I know definitely works because I can see the
cookies being stored in the cookies.txt file.
use LWP::UserAgent;
require HTTP::Cookies;
my $jar = new HTTP::Cookies(file = 'cookies.txt', autosave = 1);
my $ua = new LWP::UserAgent;
$ua-cookie_jar( $jar );
for (1 .. 10)
{
my $req = new
HTTP::Request(GET = 'http://www.something.com/index.html');
$ua-request($req);
}
However, when I do the same with LWP::Parallel::UserAgent as in the
script below, nothing gets saved to the cookies.txt file:
use LWP::Parallel::UserAgent;
require HTTP::Cookies;
my $jar = new HTTP::Cookies(file = 'cookies.txt', autosave = 1);
my $ua = new LWP::Parallel::UserAgent;
$ua-cookie_jar( $jar );
for (1 .. 10)
{
my $req = new
HTTP::Request(GET = 'http://www.something.com/index.html');
$ua-register($req);
$ua-wait(100);
}
Can someone please shed some light on how the 2 modules treat
cookies/cookie jars differently?
Regards,
Ashisih
Re: Broken Pipe while using LWP::Parallel::UserAgent
On Wed, 23 May 2001, Sougata Mukherjea wrote: I am using LWP::Parallel::UserAgent to download about 3000 documents. In the middle of the downloading the program is exiting with Broken Pipe. Any idea what can be wrong? have you tried using LWP instead of LWP::Parallel? Does it end with Broken Pipe, too? If it doesn't, let me know which sites you connect to and any more info that'd help me reproduce the problem. -m -- Marc Langheinrich [EMAIL PROTECTED]
Re: Broken Pipe while using LWP::Parallel::UserAgent
No, I don't get the problem with LWP. I connect to a lot of sites - so I can't pinpoint the problem site. Looking through the mailing list I see others have also mentioned this problem. Is there a workaround? Thanks, - Sougata Marc Langheinrich wrote: On Wed, 23 May 2001, Sougata Mukherjea wrote: I am using LWP::Parallel::UserAgent to download about 3000 documents. In the middle of the downloading the program is exiting with Broken Pipe. Any idea what can be wrong? have you tried using LWP instead of LWP::Parallel? Does it end with Broken Pipe, too? If it doesn't, let me know which sites you connect to and any more info that'd help me reproduce the problem. -m -- Marc Langheinrich [EMAIL PROTECTED]
Broken Pipe while using LWP::Parallel::UserAgent
Hi, I am using LWP::Parallel::UserAgent to download about 3000 documents. In the middle of the downloading the program is exiting with Broken Pipe. Any idea what can be wrong? Thanks, - Sougata
LWP::Parallel::UserAgent and MacPerl
I am using the latest MacPerl (v. 5.20r4) and have installed the the LWP::Parallel::UserAgent module with its associated modules. (I have installed lib-www v. 2.48 and Parallel::UserAgent v. 5.48, which are supposed to be mutually compatible.) I have changed the filetypes apporpriately to MacPerl text files, since this was giving me initial difficulty. The problem is, even when I have a script with only the line use LWP::Parallel::UserAgent; the MacPerl syntax checker says "LWP/Parallel/UserAgent did not return a true value" I have not had this problem with the single-thread LWP::UserAgent. I have written scripts with it that work just fine. When I run the syntax checker on LWP::Parallel::UserAgent itself, it checks out just fine. What can be wrong?
parallel useragent request question
I seem to remember this question being brought up a couple of months ago, but can't seem to find the thread in the archives. I'd like to check for success/failure on each of the requests registered in my parallel useragent. After I've registered all the requests and called -wait, can I somehow reuse the request (not just the URL, but the whole request)? It seems the keys of the response hash might contain some info about the request object, but they seem to be empty. Thanks for any help, --Benjamin _ Compare Prices online at http://www.golly.com
Re: parallel useragent and perl-5.6.0
It seems the parallel version does not work with the perl5.6. yeah, I released the latest version without realizing that there had been a new perl version out there :-( I'll try to upload an interim release which would include a Makefile that tested for perl's version number (and fail if it detects 5.6). Hopefully I'll get perl5.6 installed myself sometime within the next few weeks so I can try to upload a 5.6-compatible version! cheers, Marc -- Marc Langheinrich [EMAIL PROTECTED]
