Check your usage of cookie jar. The following simple script works fine for me:
$ cat foo.pl #!/usr/local/bin/perl use LWP::UserAgent; use HTTP::Cookies; $cookie = new HTTP::Cookies; $ua = new LWP::UserAgent; $ua->cookie_jar(HTTP::Cookies->new(file=>'lwp-cookies',autosave=>1)); $base_url='http://cgi.ebay.com/aw-cgi/eBayISAPI.dll?ViewItem&item=1304138736'; my $response=$ua->request(new HTTP::Request 'GET' => $base_url); if ( $response->is_success ) { print $response->content; } else { print "Error: " . $response->status_line . "\n"; } $ foo.pl >/dev/null $ cat lwp-cookies #LWP-Cookies-1.0 Set-Cookie3: nonsession="AQAAAAEAAAAIAAAAWAAAAA6ZCjwOJjI8MDEwMDczMjc1MDJ4NjVodHRwOi8vY2dpLmViYXkuY29tL2F3LWNnaS9lQmF5SVNBUEkuZGxsP1ZpZXdJdGVtJml0ZW09MTMwNDEzODczNk4*f"; path="/"; domain=".ebay.com"; path_spec; expires="2002-12-02 21:11:42Z"; version=0 Hope it helps. Richard On Sun, Dec 02, 2001 at 03:20:53PM -0500, [EMAIL PROTECTED] wrote: > Richard, > > Oh yes this fix works nicely. You the man!! Thank you very much. > I will mention however, that this function: read_header_lines() > has been moved to Net/HTTP/Methods.pm in libwww-perl 5.62 > > As to Erik's question.. I am using RedHat 7.2 (newly installed) > with libwww-perl 5.62. > > This bug is most definitely related to eBay's system, and > they probably dont even know the problem exists because > all of the browsers continue to work without a hitch. > I suspect the eBay problem is a recent problem (and perhaps > it was intentional) because "eBayAgent" which is a publically > available (written in perl) auction sniper stopped working > as well. And somewhere I read a thread that the eBay programmers > where trying to defeat "wget" to stop folks from recursively > slurping their web space. > > Although this eBay header doesn't strictly conform to HTTP specs, > I hope that this particular fix makes it into the next version of > libwww-perl. > > Another problem, may or may not be related to this deformed header... > I tried to setup a cookie_jar to save this eBay nonsession cookie, but it > didn't work as described in: perldoc lwpcook. I get only 1 line: > #LWP-Cookies-1.0 > in lwpcookies.txt and nothing else. Not a big deal, but I thought I > would mention it anyway. > > Thanks again for your help, > > Chris Stuber (mapsurfer) > Silicon Mapping Solutions, Inc. > EMail: [EMAIL PROTECTED] > > > Richard Chen wrote: > > > > The problem can be seen from a network sniffer tool: > > > > GET /aw-cgi/eBayISAPI.dll?ViewItem&item=1304138736 HTTP/1.1 > > Connection: close > > Host: cgi.ebay.com:80 > > Accept: image/gif,image/x-xbitmap,image/jpeg,image/pjpeg,image/png,*/* > > Accept-Charset: iso-8859-1,*,utf-8 > > Accept-Encoding: gzip > > Accept-Language: en > > User-Agent: Mozilla/4.78 [en] (X11; U; Linux 2.4.7-10 i686) > > > > HTTP/1.0 200 OK > > Server: Microsoft-IIS/4.0 > > Date: Sun, 02 Dec 2001 15:03:28 GMT > > Set-Cookie: >nonsession=AQAAAAEAAAAIAAAAWAAAAMBCCjzAzzE8MDEwMDczMDU0MDh4NjVodHRwOi8vY2dpLmViYXkuY29tL2F3LWNnaS9lQmF5SVNBUEkuZGxsP1ZpZXdJdGVtJml0ZW09MTMwNDEzODczNk4*i; > path=/; domain=.ebay.com; Expires=Mon, 02-Dec-2002 15:03:28 GMT > > HTTP/1.0 200 OK > > Server: Microsoft-IIS/4.0 > > Date: Sun, 02 Dec 2001 15:03:28 GMT > > ... > > > > Notice that the line > > HTTP/1.0 200 OK > > is sent back twice. This does not follow the http protocol. > > LWP considers this as a bad header in sub read_header_lines of > > Net/HTTP.pm and dies from it: > > > > sub read_header_lines { > > my $self = shift; > > my @headers; > > while (my $line = my_readline($self)) { > > if ($line =~ /^(\S+)\s*:\s*(.*)/s) { > > push(@headers, $1, $2); > > } > > elsif (@headers && $line =~ s/^\s+//) { > > $headers[-1] .= " " . $line; > > } > > else { > > die "Bad header: $line\n"; > > } > > } > > return @headers; > > } > > > > To fix this, juse add the line > > next if ( $line =~ /^HTTP/ ); > > immediately before the die call line. > > > > Hope this helps. > > > > Richard > > > > On Sat, Dec 01, 2001 at 11:12:32PM -0500, [EMAIL PROTECTED] wrote: > > > > > > I am having extreme problems trying to fetch eBay items > > > using LWP. Try to fetch an auction item like: > > > > > > http://cgi.ebay.com/aw-cgi/eBayISAPI.dll?ViewItem&item=1304138736 > > > > > > Note: This item number expires 12/9/2001 > > > > > > perl -MLWP::Simple -e 'getprint > > > "http://cgi.ebay.com/aw-cgi/eBayISAPI.dll?ViewItem&item=1304138736";' > > > 500 Bad header: HTTP/1.0 200 OK > > > <URL:http://cgi.ebay.com/aw-cgi/eBayISAPI.dll?ViewItem&item=1304138736> > > > > > > The weird thing is that it works fine with all versions of IE, Netscape, > > > lynx, etc, but fails for LWP and WGET. I poke around and found a document > > > that describes the bug for wget > > > http://www.mail-archive.com/[email protected]/msg01562.html > > > > > > To summarize the report, it says that some of eBay's page (like the one > > > mentioned above) dont output a "Content-type" header, it uses the > > > "<META HTTP-EQUIV="Content-Type"" tag instead. Why? I dunno... > > > Wayne Schlitt developed a patch for wget-1.7.1-pre1 to assume "Content-type: > > > text/html" > > > if no header is found. I am hoping that the author of LWP will follow suit. > > > > > > Like I say, most all browsers make this assumption and work fine. Apparently > > > this is a fairly recent issue with these eBay pages. Does anyone know of a > > > quickfix > > > or patch for LWP to deal with this? > > > > > > Thanks, > > > > > > Chris Stuber (mapsurfer) > > > [EMAIL PROTECTED]
