Emmanuel wrote:
I decided to use Fetcher2 instead of Fetcher and i noticed that Fetcher2 doesn't act on a polite way. I mean it doesn't wait fetcher.server.delay before doing another request on the same server.In Fetcher2 (on the last version of trunk), someone has defined this option: // set non-blocking & no-robots mode for HTTP protocol plugins. getConf().setBoolean(Protocol.CHECK_BLOCKING, false); getConf().setBoolean(Protocol.CHECK_ROBOTS, false); In this case, the protocol HTTP doesn't wait crawlDelay defore doing another request. May I know exactly why ? Is it normal or a bug ?
Have you actually observed this wrong behavior during fetching? Fetcher2 performs blocking in a different way than Fetcher - it controls the blocking itself, instead of delegating it to the protocol plugin. These two properties are set to false on purpose.
-- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com
