Markus Jelsma created NUTCH-3096:
------------------------------------

             Summary: HostDB ResolverThread can create too many job counters
                 Key: NUTCH-3096
                 URL: https://issues.apache.org/jira/browse/NUTCH-3096
             Project: Nutch
          Issue Type: Bug
          Components: hostdb
    Affects Versions: 1.20
            Reporter: Markus Jelsma
            Assignee: Markus Jelsma
             Fix For: 1.21


Hadoop will allow no more than 120 distinct counters. If we have a large number 
of distinct DNS lookup failure counts, we'll exceed the limit, Hadoop will 
complain,  the job will fail.

 

Let's limit the amount of possibilities by grouping the numFailures in just a 
few buckets.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to