[
https://issues.apache.org/jira/browse/PHOENIX-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700913#comment-14700913
]
Gabriel Reid commented on PHOENIX-2154:
---------------------------------------
I think that writing to HFiles with a single reducer basically eliminates any
performance advantage that you can get via writing to HFiles, as it essentially
turns the job into a serial process. [~tdsilva]'s performance numbers reflect
this.
Note that the work that the reducer does when writing to HFiles isn't simply
deduplication -- it also ensures ordering per region. At a high level it does
the same thing as a compaction in a region server.
Seeing as having a decently-split table doesn't sound like it's too feasible,
the idea of writing via the "normal" HBase API and then having a single reducer
whose only task is to mark the index as active sounds like a great idea.
> Failure of one mapper should not affect other mappers in MR index build
> -----------------------------------------------------------------------
>
> Key: PHOENIX-2154
> URL: https://issues.apache.org/jira/browse/PHOENIX-2154
> Project: Phoenix
> Issue Type: Bug
> Reporter: James Taylor
> Attachments: IndexTool.java
>
>
> Once a mapper in the MR index job succeeds, it should not need to be re-done
> in the event of the failure of one of the other mappers. The initial
> population of an index is based on a snapshot in time, so new rows getting
> *after* the index build has started and/or failed do not impact it.
> Also, there's a 1:1 correspondence between index rows and table rows, so
> there's really no need to dedup. However, the index rows will have a
> different row key than the data table, so I'm not sure how the HFiles are
> split. Will they potentially overlap and is this an issue?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
