Re: High GET- retrieval failure rate

[Peter Stevens]

Hello Philippe,
I have been using Mechanize, rather than LWP directly, so I am not sure 
I can be of much help. When I was debugging my connections, one useful 
tip I got was to turn on LWP's debugging. Include this line in your script:

   use LWP::Debug qw(+);
This way you can see what is happening at the protocol level.
I run queries on several different servers on a daily basis. My problems 
all appear to be time of day issues ( = scheduled maintenance). When 
logging into some sites, I have had some problems with the site not 
being ready for the followup request until after a few seconds have gone 
by, but this is a server problem, not a connectivity issue, and was 
solved by sleeping for a few seconds.

The actual URL would be helpful to reproce the problem.
Good Luck!
Peter
Philippe de Rochambeau wrote:
Hello,
I am using libwww-perl 5.80 with perl 5.80 and get a high GET- 
retrieval failure rate (70-80%) when I connect to a particular server 
located in Israel (I am based in Europe).

When I use curl instead of LWP the failure rate is almost 0%.
Has anyone had this problem before? Do I need to upgrade both Perl and 
LWP?

Philippe
--------- My script ----------------
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
if ( ! @ARGV ) {
    die "Usage: $0 <number> <time out>";
}
my $number_connections = $ARGV[0];
my $timeout = $ARGV[1];
my $get_this = 'http://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
my $dateext = `date +%s`;
chomp $dateext;
my $file = "lwptest$dateext.log";
my $count_successes = 0;
my $count_failures = 0;
print "Connecting to $get_this.\n";
print FH "Connecting to $get_this.\n";
print "Number of connections is $number_connections. Time-out is 
$timeout second(s).\n";
print FH "Number of connections is $number_connections. Time-out is 
$timeout second(s).\n";

for (my $i = 0; $i < $number_connections; $i++) {
    my $ua = LWP::UserAgent->new;
    $ua->agent("INA picker v0.1");
    $ua->timeout($timeout);
    my $req = HTTP::Request->new(GET => $get_this);
    #$req->authorization_basic($partner_retrieve_pointer->{'username'},
    #    $partner_retrieve_pointer->{'password'});
    my $res = $ua->request($req);
    my $date = `date +'%H:%M:%S'`;
    chomp $date;
    printf( "Connection %d/%d (%3.2f%%) ", ($i+1), 
$number_connections, ((($i+1)/$number_connections)*100) );
    if ($res->is_success) {
        print "succeeded.\n";
        $count_successes++;
    } else {
        print "failed. Reason: " . $res->status_line . "\n";
        $count_failures++;
    }
}

my $result = (($count_failures/$number_connections)*100);
printf("Failure rate: %3.2f%%.\n", $result);