Sebastian Nagel created NUTCH-3250:
--------------------------------------

             Summary: CommonCrawlDataDumper fails to read segment
                 Key: NUTCH-3250
                 URL: https://issues.apache.org/jira/browse/NUTCH-3250
             Project: Nutch
          Issue Type: Bug
          Components: dumpers
    Affects Versions: 1.23
            Reporter: Sebastian Nagel
             Fix For: 1.24


(seen while testing / reproducing NUTCH-3249)

CommonCrawlDataDumper uses a fixed pattern how SequenceFile partitions are 
named:
{noformat}
part-[0-9]{5}
{noformat}

This is the pattern used in the "old" MapReduce API which has been replaced 
already in 2018 (Nutch 1.15, NUTCH-1219).

Effectively, all segments written with Nutch 1.15 and upwards fail to be 
processed by CommonCrawlDataDumper.

Ideally, no assumptions about naming of partitions should be made and the 
Hadoop classes should be used to read the input.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to