it is not allowed for robots.
http://search.yahoo.com/robots.txt
User-agent: *
Disallow: /search
Disallow: /bin
Disallow: /myweb
Disallow: /myresults
Disallow: /language
Kim Theng Chong schrieb:
> Hi all,
>
> Can Nutch crawl Yahoo search result page? eg :
> http://search.yahoo.c
; Best regards,
> Kim
>
>
>
>
>
> From: Devang Shah
> To: nutch-user@lucene.apache.org
> Sent: Wed, March 31, 2010 11:28:50 AM
> Subject: RE: Crawl yahoo search result page
>
> No - you won't be able to crawl this page. Nutch will follow robots
>
Hi Devang,
Thank you so much for your reply. =)
Have a nice day.
Best regards,
Kim
From: Devang Shah
To: nutch-user@lucene.apache.org
Sent: Wed, March 31, 2010 11:28:50 AM
Subject: RE: Crawl yahoo search result page
No - you won't be able to crawl
e.org
Subject: Crawl yahoo search result page
Hi all,
Can Nutch crawl Yahoo search result page? eg :
http://search.yahoo.com/search?rd=&fp_ip=my&p=ontology&toggle=1&cop=mss&ei=U
TF-8&fr=yfp-t-892 (put as seed url) . I was not able to fetch the results in
this page. Can som
Hi all,
Can Nutch crawl Yahoo search result page? eg :
http://search.yahoo.com/search?rd=&fp_ip=my&p=ontology&toggle=1&cop=mss&ei=UTF-8&fr=yfp-t-892 (put
as seed url) . I was not able to fetch the results in this page. Can someone
guide me on this?
Thank you.
Best regards,
Kim
