Hi! It's the first of the month and I thought I'd publish the FAQ in
here. If once a month's too frequent, let me know!
- Kenneth
--------------------------
Perl LWP (libwww) FAQ
Version 1.0.3 20000229
The libwww-perl distribution is a collection of Perl modules which provides
a simple and consistent programming interface (API) to the World-Wide Web.
The main focus of the library is to provide classes and functions that allow
you to write WWW clients, thus libwww-perl said to be a WWW client library.
The library also contains modules that are of more general use.
The latest version of this FAQ is available at
http://people.we.mediaone.net/kfrankel/lwpfaq.txt
FAQ Questions and Answers:
1) Where can I find some examples?
Did you read lwpcook? Type: perldoc lwpcook
Try Randal's Web Techniques columns:
http://web.stonehenge.com/merlyn/WebTechniques/
Books:
+ Web Client Programming with Perl by Clinton Wong.
(http://www.ora.com/catalog/webclient/)
+ Web Programming With Perl5 by Bill Middleton, Brian Deng, Chris Kemp.
(http://www.amazon.com/exec/obidos/ISBN=1575211122/3060-8620893-901942)
+ The Perl Cookbook <- also has some examples
2) How do I unsubscribe from the [EMAIL PROTECTED] mailing list?
Send any kind of email to: [EMAIL PROTECTED]
BUT DONT send mail to [EMAIL PROTECTED] to tell everybody about it!
3) Searching on CPAN for the LWP module finds the mailing list. Where is
LWP?
Look in CPAN for libwww.
4) Is LWP Mod-Perl Compatible? (19991210 - [EMAIL PROTECTED])
Yes. (Scott Gifford - [EMAIL PROTECTED])
5) How come LWP::UserAgent timeout does not work?
ActiveState/Win32 Perl: Does not work in ActiveState yet
All Other Perls: Upgrade to IO-1.20 or better.
6) What do I do when 'make test' fails?
This usually means that you can't even run 'ping $(hostname)'
because the hostname configured for the machine can not be looked
up and connected to. Fix your host configuration or, if only the
host configuration tests have failed, simply 'make install' anyway.
7) Where is HTML::Parse?
HTML::Parse has been replaced by HTML::TreeBuilder.
8) Can the XS version HTML-Parser be installed in Win NT ?
Yes. At least one person (Michael A. Chase" <[EMAIL PROTECTED]>) has
built and installed all the alpha and beta versions using Borland C++
under WinNT.
9) How do I install the latest libwww-perl module through PPM?
use search in ppm to show names of modules.
PPM> search www
use install in ppm to install it
PPM> install www
10) How can I post to a form?
my $request = new HTTP::Request("POST", \%HashOfFields);
Read the section on 'POST' in lwpcook for more details.
11) how do I do SSL via LWP?
This is answered in lwpcook. The SSL is done transparently.
If the SSL server is on a non-standard port (not 443), then
you have to append a :XXX where XXX is the port number of the
SSL server.
The libwww-perl HTTPS support is based on SSLeay.
There is also a README.SSL to look at.
12) How can I use the XS version of LWP on Windows?
It is not correct totalk about the XS version of LWP. LWP has always
relied on XS modules (MIME::Base64, Digest::MD5), so the fact that
HTML::Parser now is XS should not be a new problem. (Gisle Aas)
13) How can I use the XS version of LWP if I don't have a C compiler?
Win32: ActiveState Perl comes with a compiled version, however old.
Others: ???
14) Can I still use the non-XS version of LWP?
<< NOT ANSWERED YET >>
<Can we test to see if we are using an XS or non-XS version?!>
15) How do I use Cookies?
use HTTP::Cookies;
my $ua = LWP::UserAgent->new;
my $cookies = HTTP::Cookies->new; # Create a cookie jar
$ua->cookie_jar($cookies); # Enable cookies
...
my $response = $ua->request($req);
Note that the following calls:
$cookies->extract_cookies($response); # Add new cookies to jar
$cookies->add_cookie_header($request); # Add cookies to request
are not necessary, as they are called automatically by
LWP::UserAgent once $ua->cookie_jar is set up.
16) How do use Authentication (login + password)?
use HTTP::Request::Common;
my $request = new HTTP::Request("GET", $URI) or die "$!";
$request->authorization_basic($uname, $password);
17) How do I use LWP when I am behind a proxy server?
Look at lwpcook for more detail (Type: perldoc lwpcook)
Code:
my $ua = new LWP::UserAgent;
$ua->proxy(['http','ftp'] => "YOUR.PROXY.COM");
my $req = new HTTP::Request 'GET',"http://www.perl.com";
$req->proxy_authorization_basic("PROXY_USERNAME","PROXY_PASSWORD");
# Where YOUR.PROXY.COM is your proxy server,and PROXY_USERNAME
# and PROXY_PASSWORD are needed to log into the proxy server.
# You can skip the proxy_auth_basic step if your server is open)
$response = $ua->request($req);
18) How can I tell the ua to stop the connection after a given amount of data?
Use LWP::UserAgent::max_size() to set a cutoff point.
Code:
my $ua = LWP::UserAgent->new;
$ua->max_size($maxsiteinbytes);
my $response = $ua->request(...); # will stop after >= max bytes
19) How can a request be aborted after a given time?
19) or How can I timeout "running" connections?
19) or How can I tell the ua to stop the connection after a given period of
time?
Use alarm() if you are not Win32. See "perldoc -f alarm." Alarm
currently
doesn't work in Win32. See following question.
Also, you can use $ua->timeout() but read the following following
question.
Code:
require LWP::UserAgent;
my $timeout = 60; # in seconds
my $ua = new LWP::UserAgent;
my $request = new HTTP::Request('GET', 'http://host.com/big/file.txt');
eval{
local $SIG{ALRM} = sub { die "download took too long\n" };
alarm $timeout
$response = $ua->request($request);
alarm 0;
}
if ($@) {
if ($@ =~ /^download took too long/) {
# Download timed out
} else {
# some other error?!
}
} else {
# no timeout, got the entire request.
}
(19991213 - Marc Langheinrich [EMAIL PROTECTED])
20) In Win32 / ActiveState Perl, how can I tell the ua to stop the connection
after a given amount of time? (or the other two versions of this
question)
To do this, use the callback in which you manually append incoming
data to the current content of the response object, until the elapsed
time is too big and then simply call "die()" in the callback sub,
as explained in "perldoc LWP::UserAgent": The die message will be
available as the "X-Died" special response header field. Note that this
will not work if there is NO data coming in, the callback routine only
gets called when a chunk is filled.
Code:
require LWP::UserAgent;
use HTTP::Request::Common;
my $ua = LWP::UserAgent->new;
my $request = new HTTP::Request('GET', 'http://host.com/big/file.txt');
$endtime = time + 60; # One minute into the futa
$response = $ua->request($myrequest,
sub {
my($chunk, $res) = @_;
$res->{_content} .= $chunk;
if (time >= $endtime) {
# Aborts the connection
die ("Max Time Exceeded");
}
}
);
print "Response: " . $response->code . " " . $response->message . "\n";
print "Response->content length = " . length($response->content) . "\n";
21) I set the $ua->timeout() to 1 minute, but LWP might still uses a long time
to fetch a document. Is this a bug?
No, $ua->timeout sets a timeout on how long LWP allows the connection to
be idle before giving up. If only a single byte arrives each 50 seconds,
LWP will keep on listening but will probably not finish downloading for
a long, long time. If you want an absolute timeout you can do it with
alarm() or by setting up a callback and measuring the time yourself. Take
a look at preceeding two questions. (Gisle Aas)
22) What is the status of LWPng?
Stable alpha :-) (Gisle Aas)
---------
