Hi Devang, Thank you so much for your reply. =)
Have a nice day. Best regards, Kim ________________________________ From: Devang Shah <devang.dss...@gmail.com> To: nutch-user@lucene.apache.org Sent: Wed, March 31, 2010 11:28:50 AM Subject: RE: Crawl yahoo search result page No - you won't be able to crawl this page. Nutch will follow robots directive of the domain - see http://search.yahoo.com/robots.txt. -Devang. -----Original Message----- From: Kim Theng Chong [mailto:kimthe...@yahoo.com] Sent: Tuesday, March 30, 2010 10:00 PM To: nutch-user@lucene.apache.org Subject: Crawl yahoo search result page Hi all, Can Nutch crawl Yahoo search result page? eg : http://search.yahoo.com/search?rd=&fp_ip=my&p=ontology&toggle=1&cop=mss&ei=U TF-8&fr=yfp-t-892 (put as seed url) . I was not able to fetch the results in this page. Can someone guide me on this? Thank you. Best regards, Kim
