This is an automated email from the ASF dual-hosted git repository.
snagel pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/nutch.git
from 568993b90 NUTCH-2948 Upgrade dependencies to Any23 2.7 and Tika 2.3.0
new 42ae2a345 NUTCH-2946 Fetcher: slow down fetching from hosts where
requests fail repeatedly with exceptions or HTTP status codes mapped to
ProtocolStatus.EXCEPTION (HTTP 403 Forbidden, 429 Too many requests, 5xx server
errors, etc.)
new bdbe7b330 NUTCH-2946 Fetcher: optionally slow down fetching from hosts
with repeated exceptions - configure the delay in seconds as a float instead of
milliseconds - use the value of fetcher.server.delay as default - double the
delay with every observed exception (exponential backoff) but cap the growth
at 2**31 to avoid overflows
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
conf/nutch-default.xml | 23 ++++++++++++++++++----
.../org/apache/nutch/fetcher/FetchItemQueues.java | 19 ++++++++++++++++++
2 files changed, 38 insertions(+), 4 deletions(-)
