i find that robots.txt file is unavailable in this site,how can i crawl in farsnews.com ? thanks
-- View this message in context: http://lucene.472066.n3.nabble.com/Couldn-t-get-robots-txt-for-site-tp3983633p3983636.html Sent from the Nutch - User mailing list archive at Nabble.com.
