On Saturday, August 24, 2013 4:15:01 PM UTC+5:30, malhar vora wrote: > Hello All, > > > > > > I am simply fetching data from robots.txt of a url. Below is my code. > > > > siteurl = siteurl.rstrip("/")
Sorry for last complete. It was sent by mistake. Here is my code. siteurl = siteurl.rstrip("/") roboturl = siteurl + r'/robots.txt' robotdata = urllib.urlopen(roboturl).read() # Reading robots.txt of given url print robotdata In above code siteurl is fetched simply from local text file. Whenever I run above code. In place of "/" before robots.txt, it writes "\\" in url as I found in error. The error is given below. This is main function Main URL : www.bestrecipes.com.au $$$$$$$$$$:www.bestrecipes.com.au ###########-->www.bestrecipes.com.au/robots.txt Traceback (most recent call last): File "dataintegrator.py", line 104, in <module> main() File "dataintegrator.py", line 81, in main print "Sitemap Url : " + getSiteMapUrl(i) File "D:\Malhar Data\Projects\Data Parsing\My Code\Final Part\libs\datareader. py", line 50, in getSiteMapUrl robotdata = urllib.urlopen(roboturl).read() # Reading robots.txt of given ur l File "C:\Python26\lib\urllib.py", line 87, in urlopen return opener.open(url) File "C:\Python26\lib\urllib.py", line 203, in open return getattr(self, name)(url) File "C:\Python26\lib\urllib.py", line 461, in open_file return self.open_local_file(url) File "C:\Python26\lib\urllib.py", line 475, in open_local_file raise IOError(e.errno, e.strerror, e.filename) IOError: [Errno 2] The system cannot find the path specified: 'www.bestrecipes.c om.au\\robots.txt' I am new to Python and not able to figure out this problem. Please help me. Thank you, Malhar Vora -- http://mail.python.org/mailman/listinfo/python-list