|
I have the following short bit of code:
use LWP;
use LWP::UserAgent; use LWP::Simple; use Socket; $INC{'Socket.pm'} || ($IgnoreExternals = 1);
unless ($URL_Timeout) { $URL_Timeout = 20; } unless ($getpwd) { $getpwd = "pwd"; } unless ($gethostname) { $gethostname = "hostname"; } $ua = LWP::UserAgent->new;
$ua->agent("Northcote Spider v$spider_ver/0.1 "); # Create a request # First of all try GET method
my $req = HTTP::Request->new(GET => "$PageToFetch"); $req->content_type('x-www-form-urlencoded');
$req->content('match=www&errors=0'); # Pass request to the user agent
$res= $ua->request($req); # Check the outcome of the response
$PageError=$res->error_as_HTML();
$PageError=&Strip($PageError); printf LOG "Result: $PageError\n";
Although this works on some sites, I have found some webpages which return a $PageError of either :
411 Length Required
or
500 EOF When Chunk Header Expected
Does anyone know why these errors are being generated (the pages display correctly in a browser) and how to avoid them???
--
Rich Mellor RWAP Services 35 Chantry Croft, Kinsley, Pontefract, West Yorkshire, WF9 5JH TEL: 01977 610509 Visit our website at: URL:http://www.rwapsoftware.co.uk In need of adventure? URL: http://www.rwapadventures.com Stuck with ordinary dial up internet connection ?? Read our review of internet accelerators and broadband at: URL:http://www.rwapservices.co.uk |
