I have just spent a few hours wrestling with this bug, and came up with
a workaround.
After looking at the LWP source code I realized that it doesn't look at
the
Content-Length header returned by the server, so I added a request
callback routine
to read the number of bytes specified in the Content-Length header:
sub read_chunk {
my($chunk, $res, $prot) = @_;
$res->{_content} .= $chunk;
if ($totlength == 0) {
$hlength = scalar $res->header("Content-Length");
print "Content-Length: $hlength\n";
}
$totlength += length($chunk);
print "Length: " . length($chunk) . " $totlength $hlength\n";
die "All data read" if (($hlength>0) && ($totlength>=$hlength));
}
$link = "http://web.pfc.forestry.ca:80/climate/cidet/controls.html";;
$request = new HTTP::Request('GET',$link,$http_headers);
$hlength = 0;
$totlength = 0;
$response = $agent->request($request,\&read_chunk);
This code seems to work reliably during testing, but I haven't tried it
on a
large number of URLs yet. Also, I still seem to have a problem with
HEAD
requests, which don't seem to call the callback routine.
Why doesn't LWP do this already? (use the Content-Length header if
found)
- Bob Fillmore
