On Mon, 23 Nov 2009, Ken Irwin wrote:

Hi all,

I'm moving to a new web server and struggling to get it configured properly. The problem of the 
moment: having a Perl CGI script call another web page in the background and make decisions 
based on its content. On the old server I used an antique Perl script called "hcat" 
(from the Pelican book<http://oreilly.com/openbook/webclient/ch04.html>); I've also tried 
curl and LWP::Simple.

In all three cases, I get the same behavior: it works just fine on the command 
line, but when called by the web server through a CGI script, the LWP (or other 
socket connection) gets no results. It sounds like a permissions thing, but I 
don't know what kind of permissions setting to tinker with. In the test script 
below, my command line outputs:

Content-type: text/plain
Getting URL: http://www.npr.org
885 lines

Whereas the web output just says "Getting URL: http://www.npr.org"; - and doesn't even get 
to the "Couldn't get" error message.

Any clue how I can make use of a web page's contents from w/in a CGI script? 
(The actual application has to do with exporting data from our catalog, but I 
need to work out the basic mechanism first.)

Here's the script I'm using.

use LWP::Simple;
print "Content-type: text/plain\n\n";
my $url = "http://www.npr.org";;
print "Getting URL: $url\n";
my $content = get $url;
die "Couldn't get $url" unless defined $content;
@lines = split (/\n/, $content);
foreach (@lines) { $i++; }
print "\n\n$i lines\n\n";

Any ideas?

I'd suggest testing the results of the call, rather than just looking for content, as an empty response could be a result of the server you're connecting to. (unlikely in this case, but it happens once in a while, particularly if you turn off redirection, or support caching). Unfortunately, you might have to use LWP::UserAgent, rather than LWP::Simple:

        #!/bin/perl --

        use strict; use warnings;
        use LWP::UserAgent;

        my $ua = LWP::UserAgent->new( timeout => 60 );

        my $response = $ua->get('http://www.npr.org/');
        if ( $response->is_success() ) {
                my $content = $response->decoded_content();
        } else {
                print "HTTP Error : ",$response->status_line(),"\n";


(and changing the shebang line for my location of perl, your version worked via both CGI and command line)

oh ... and you don't need the foreach loop:

        my $i = @lines;


Reply via email to