Hi Markus, I am trying to understand the problem you described. You meant that with the original Nutch's robots parsing code, the robots file below allowed your crawler to crawl stuff:
User-agent: * Disallow: / User-agent: our_crawler Allow: / But now that started using the change from NUTCH-1031 [0], (ie. delegation of robots parsing to crawler commons), it blocked your crawler. To make things work, you had to change your robots file to this: User-agent: our_crawler Allow: / User-agent: * Disallow: / Did I understand the problem correctly ? [0] : https://issues.apache.org/jira/browse/NUTCH-1031 Thanks, Tejas On Fri, Jan 24, 2014 at 7:29 PM, Markus Jelsma <[email protected]>wrote: > Hi, > > I am attempting to merge some Nutch changes back to our own. We aren't > using Nutch' CrawlerCommons impl but the old stuff. But because of > recording of response time and rudimentary SSL support i decided to move it > back to our version. Suddenly i realized a local crawl does not work > anymore, it seems because of the order of the robots definitions. > > For example: > > User-agent: * > Disallow: / > > User-agent: our_crawler > Allow: / > > Does not allow our crawler to fetch URL's. But > > User-agent: our_crawler > Allow: / > > User-agent: * > Disallow: / > > Does! This was not the case before, anyone here aware of this? By design? > Or is it a flaw? > > Thanks > Markus >
