I think, as of 1.12, there is a parameter to disable the robots check, i am not sure. Check nutch-default, it might be there. M.
-----Original message----- > From:Nestor <[email protected]> > Sent: Wednesday 5th October 2016 0:05 > To: [email protected] > Subject: Re: crawling a subfolder > > OK, Thanks for your help > I found out that part of my problem was that there was a robots.txt that > would not allow me to crawl my site. > The lessons and gotchas of learning nutch > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/crawling-a-subfolder-tp4299300p4299593.html > Sent from the Nutch - User mailing list archive at Nabble.com. >
