[
https://issues.apache.org/jira/browse/HBASE-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031285#comment-13031285
]
Todd Lipcon commented on HBASE-3871:
------------------------------------
I have a few concerns with this:
* If there's an incremental load that requires a lot of splits, it indicates
that the MR job probably was configured incorrectly. (eg with too few reducers,
or with reducers not aligned to existing table boundaries). We shouldn't
endeavour to optimize a case where HFileOutputFormat has been likely misused.
* The splitting on the load side is probably mostly network bound.
Parallelizing it to multiple threads isn't likely to gain us a lot.
> Speedup LoadIncrementalHFiles by parallelizing HFile splitting
> --------------------------------------------------------------
>
> Key: HBASE-3871
> URL: https://issues.apache.org/jira/browse/HBASE-3871
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
> Affects Versions: 0.90.2
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: 3871.patch
>
>
> From Adam w.r.t. HFile splitting:
> There's actually a good number of messages of that type (HFile no longer fits
> inside a single region), unfortunately I didn't take a timestamp on just when
> I was running with the patched jars vs the regular ones, however from the
> logs I can say that this is occurring fairly regularly on this system. The
> cluster I tested this on is our backup cluster, the mapreduce jobs on our
> production cluster output HFiles which are copied to the backup and then
> loaded into HBase on both. Since the regions may be somewhat different on
> the backup cluster I would expect it to have to split somewhat regularly.
> This JIRA complements HBASE-3721 by parallelizing HFile splitting which is
> done in the main thread.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
