Markus Jelsma created NUTCH-3096: ------------------------------------ Summary: HostDB ResolverThread can create too many job counters Key: NUTCH-3096 URL: https://issues.apache.org/jira/browse/NUTCH-3096 Project: Nutch Issue Type: Bug Components: hostdb Affects Versions: 1.20 Reporter: Markus Jelsma Assignee: Markus Jelsma Fix For: 1.21
Hadoop will allow no more than 120 distinct counters. If we have a large number of distinct DNS lookup failure counts, we'll exceed the limit, Hadoop will complain, the job will fail. Let's limit the amount of possibilities by grouping the numFailures in just a few buckets. -- This message was sent by Atlassian Jira (v8.20.10#820010)