Hi All,
I'm trying to crawl http://www.tvshack.net why nutch default configuration doesn't crawl it ? I change the robots rules to only my bot http.robots.agents is set to "GoogleBot" I tryed also with and without '*' Any idea ? Regards, Louis
Hi All,
I'm trying to crawl http://www.tvshack.net why nutch default configuration doesn't crawl it ? I change the robots rules to only my bot http.robots.agents is set to "GoogleBot" I tryed also with and without '*' Any idea ? Regards, Louis