Hi, I am attempting to merge some Nutch changes back to our own. We aren't using Nutch' CrawlerCommons impl but the old stuff. But because of recording of response time and rudimentary SSL support i decided to move it back to our version. Suddenly i realized a local crawl does not work anymore, it seems because of the order of the robots definitions.
For example: User-agent: * Disallow: / User-agent: our_crawler Allow: / Does not allow our crawler to fetch URL's. But User-agent: our_crawler Allow: / User-agent: * Disallow: / Does! This was not the case before, anyone here aware of this? By design? Or is it a flaw? Thanks Markus
