I decided to use Fetcher2 instead of Fetcher and i noticed that
Fetcher2 doesn't act
on a polite way. I mean it doesn't wait fetcher.server.delay before
doing another
request on the same server.
In Fetcher2 (on the last version of trunk), someone has defined this option:
// set non-blocking & no-robots mode for HTTP protocol plugins.
getConf().setBoolean(Protocol.CHECK_BLOCKING, false);
getConf().setBoolean(Protocol.CHECK_ROBOTS, false);
In this case, the protocol HTTP doesn't wait crawlDelay defore doing
another request.
May I know exactly why ?
Is it normal or a bug ?
