Hi! I've been searching the net looking for a solution to my problem for a couple of days, and I haven't been able to find anything. If someone could give me a hand, I would really appreciate it.
I regularly login to a site, copy a bunch of information into a text file, and then manipulate the text with Perl for some other purpose. I would like to be able to automate this process by having my Perl script get the information too. The server uses a form on an SSL protected page to get the username and password in a POST transaction. It seems that my post request is working properly because I get the header that I expect, a redirection header, to the secure home page. I've checked the header I get back (the first one in getpage.out) with the one I see in Live HTTP Headers in Firefox, and they are the same. So then I have some code that is supposed to get the location of the page to which I'm being redirected and execute a GET request to this location. At this point, I receive an error "501 Not implemented." I haven't been able to find anything yet that explains what's going on. Below I'm including my code listing, called getpage. Following that you'll see the headers that I have being printed to standard out (under getpage.out) as well as the debugging information (under getpage.err). I've removed any sensitive information from the listings, of course. Thanks so much for your help! **** getpage **** #!/usr/bin/perl use LWP; use HTTP::Cookies; use HTTP::Request; use LWP::Debug qw(+); $ua = LWP::UserAgent->new; $cookie_jar = HTTP::Cookies->new(file => "cookies.dat", ignore_discard => 1); $base_url = "https://secure.site.org";; $url = "$base_url/dir"; $response = $ua->get($url); if ($response->status_line =~ /OK/) { $req = HTTP::Request->new(POST => "$base_url/page.html"); $req->content_type('application/x-www-form-urlencoded'); $req->content ('username=value&password=value2'); $cookie_jar->add_cookie_header($req); $res = $ua->request($req); $cookie_jar->extract_cookies($res); print $res->status_line, "\n\n"; print $res->headers_as_string; if ($res->is_redirect) { $loc = $res->header('Location'); print ("Being redirected to $base_url$loc...\n\n"); $red_url = "$base_url$loc"; $req = HTTP::Request->new(Get => $red_url); $cookie_jar->add_cookie_header($req); $res = $ua->request($req); print $res->status_line, "\n\n"; print $res->headers_as_string; open (OUT, ">securehome.html"); print OUT $res->content; close OUT; } } **** getpage.out **** 302 Moved Temporarily Cache-Control: MAX-AGE=260 Connection: close Date: Fri, 18 Mar 2005 21:23:27 GMT Location: /dir Server: Netscape-Enterprise/4.1 Content-Length: 0 Content-Type: text/html; charset=utf8 Expires: Tue, 20 Jun 1995 04:13:09 GMT Client-Date: Fri, 18 Mar 2005 21:54:13 GMT Client-Response-Num: 1 Client-SSL-Cert-Issuer: /O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign Client-SSL-Cert-Subject: <stuff> Client-SSL-Cipher: RC4-MD5 Client-SSL-Warning: Peer certificate not verified Set-Cookie: <stuff> Set-Cookie: <stuff> Set-Cookie: <stuff> Set-Cookie: <stuff> Set-Cookie: <stuff> Set-Cookie: <stuff> Set-Cookie: <stuff> Set-Cookie: <stuff> Set-Cookie: <stuff> Being redirected to <https://secure.site.org/newpage.html> https://secure.site.org/newpage.html... (I added this line! See the getpage code) 501 Not Implemented Connection: close Date: Fri, 18 Mar 2005 21:23:27 GMT Server: Netscape-Enterprise/4.1 Content-Length: 148 Content-Type: text/html Client-Date: Fri, 18 Mar 2005 21:54:14 GMT Client-Response-Num: 1 Client-SSL-Cert-Issuer: /O=VeriSign Trust Network/OU=VeriSign, Inc./OU=VeriSign International Server CA - Class 3/OU=www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign Client-SSL-Cert-Subject: <stuff> Client-SSL-Cipher: RC4-MD5 Client-SSL-Warning: Peer certificate not verified Title: Not Implemented **** getpage.err **** LWP::UserAgent::new: () LWP::UserAgent::request: () LWP::UserAgent::send_request: GET https://secure.site.org/dir LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: () LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 1535 bytes LWP::UserAgent::request: Simple response: OK HTTP::Cookies::add_cookie_header: Checking secure.site.org for cookies HTTP::Cookies::add_cookie_header: Checking .site.org for cookies HTTP::Cookies::add_cookie_header: Checking site.org for cookies HTTP::Cookies::add_cookie_header: Checking .org for cookies LWP::UserAgent::request: () LWP::UserAgent::send_request: POST https://secure.site.org/dir/page.html LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: () LWP::UserAgent::request: Simple response: Found HTTP::Cookies::extract_cookies: <stuff> HTTP::Cookies::extract_cookies: <stuff> HTTP::Cookies::extract_cookies: <stuff> HTTP::Cookies::extract_cookies: <stuff> HTTP::Cookies::extract_cookies: <stuff> HTTP::Cookies::extract_cookies: <stuff> HTTP::Cookies::extract_cookies: <stuff> HTTP::Cookies::extract_cookies: <stuff> HTTP::Cookies::extract_cookies: <stuff> HTTP::Cookies::add_cookie_header: Checking secure.site.org for cookies HTTP::Cookies::add_cookie_header: - checking cookie path=/ HTTP::Cookies::add_cookie_header: - checking cookie <stuff> HTTP::Cookies::add_cookie_header: it's a match HTTP::Cookies::add_cookie_header: - checking cookie <stuff> HTTP::Cookies::add_cookie_header: it's a match HTTP::Cookies::add_cookie_header: Checking .site.org for cookies HTTP::Cookies::add_cookie_header: Checking site.org for cookies HTTP::Cookies::add_cookie_header: Checking .org for cookies LWP::UserAgent::request: () LWP::UserAgent::send_request: Get https://secure.site.org/newpage.html LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: () LWP::Protocol::collect: read 148 bytes LWP::UserAgent::request: Simple response: Not Implemented
