[
https://issues.apache.org/jira/browse/NUTCH-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18057661#comment-18057661
]
ASF GitHub Bot commented on NUTCH-2793:
---------------------------------------
lewismc commented on PR #534:
URL: https://github.com/apache/nutch/pull/534#issuecomment-3880181511
Closing out. If it turns out @pmezard is able to take back up this work, we
can always re-open. I'll also close out the associated Jira ticket. Thank you.
> CSV indexer does not work in distributed mode
> ---------------------------------------------
>
> Key: NUTCH-2793
> URL: https://issues.apache.org/jira/browse/NUTCH-2793
> Project: Nutch
> Issue Type: Improvement
> Components: indexer, plugin
> Affects Versions: 1.17
> Reporter: Patrick Mézard
> Priority: Major
> Fix For: 1.22
>
>
> Reasons are discussed in
> https://issues.apache.org/jira/browse/NUTCH-1541?focusedCommentId=13797768&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13797768
> and following comments.
> To summarize, the indexer interface is not aware of tasks so it cannot
> generate unique output name per reducers.
> But it seems achievable because IndexWriters initialize each writer with
> calls to 2 open functions:
> * One passing the general configuration and a "name"
> * The second to pass indexer parameters
> [https://github.com/apache/nutch/blob/master/src/java/org/apache/nutch/indexer/IndexWriters.java#L214]
> Fortunately, "name" is generated by calling getUniqueFile which does exactly
> what we want:
> [https://github.com/apache/nutch/blob/master/src/java/org/apache/nutch/indexer/IndexerOutputFormat.java#L43]
> I propose we use it instead of "nutch.csv" as CSVIndexWriter output file
> name. This is a breaking change because it modifies the output name but
> allows the indexer to work in distributed mode.
> PR will follow the ticket creation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
