sebastian-nagel opened a new pull request, #729:
URL: https://github.com/apache/nutch/pull/729
Fetcher: keep state of empty but stateful fetch queues unless queue feeder
is finished in order to ensure politeness
- next fetch time not yet reached
- non-zero exception counter and queue feeder still
adding new fetch items to queues
Only if the the queue feeder is finished and no more new fetch items are
added, these queues can finally removed.
Note: this PR needs to be adapted to #728 (NUTCH-2946) or vice verse
whichever is merged first. The state of queues needs also preserved in case
fetcher.max.exceptions.per.queue == -1 but fetcher.exceptions.per.queue.delay
!= -1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
