[cross post from perlmonks] Hi, I guess this is more a question of how to get around a website blocking LWP::UserAgent. I have been using this code for a while now
#!/usr/bin/perl -w use diagnostics; use strict; use LWP; use Date::Manip; use HTTP::Cookies; use URI; my $cookie_jar; $cookie_jar = HTTP::Cookies->new( 'file' => 'cookies.lwp', 'autosave' => 1, ); my $url = URI->new ('http://www.nseindia.com/marketinfo/indices/indexwatch.jsp'); my $browser = LWP::UserAgent->new(timeout=>'45', agent=>'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.11) Gecko/20101012 Firefox/3.6.11'); $browser->cookie_jar($cookie_jar); push @{ $browser->requests_redirectable }, 'POST'; my $html_page; my $response = $browser->get($url); if ($response->is_error()) { print "error in getting index ".$response->status_line()."\n"; print "try again...\n"; } else { $html_page = $response->content(); if ($html_page =~ m{<a href=#top>Top</a></center>}) { print "got Index\n"; } elsif ($html_page =~ m/Your request could not be processed/) { print "\t\t\tNSE is down. Try again...\n\n"; } else { print "$html_page\n"; print "failed to get index. Try again...\n"; } } This code worked till yesterday. However today I am hitting the 403 error. " error in getting index 403 Forbidden try again... " I used the same UserAgent as that of Mozilla which successfully retrieves the page. I tried it from different IP address to ensure that my IP is not blocked(I run it once a day and am not running a bot) Looking for some help in getting this going again. Thanks Uday