[
https://issues.apache.org/jira/browse/HBASE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646994#comment-16646994
]
stack commented on HBASE-21286:
-------------------------------
Related to
https://issues.apache.org/jira/browse/HBASE-20786?focusedCommentId=16603944&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16603944
?
Could we parallelize for all cases, not just for snapshots? Thanks.
> Parallelize computeHDFSBlocksDistribution when getting splits of a
> HBaseSnapshot
> --------------------------------------------------------------------------------
>
> Key: HBASE-21286
> URL: https://issues.apache.org/jira/browse/HBASE-21286
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
> Affects Versions: 1.4.0
> Reporter: Lavinia-Stefania Sirbu
> Priority: Minor
> Attachments: HBASE-21286.branch-1.4.001.patch,
> HBASE-21286.branch-1.4.002.patch
>
>
> Even if this step is called computeHDFSBlocksDistribution, this is executed
> no matter the file system of the snapshot. For example, we have observed an
> important slowness when we have a snapshot in s3 (~26k regions, 5column
> families, 2 files per column family) the getsplits time is ~40min due to the
> calls in s3 for listing the files to get the best locations.
> Parallelizing this operation can reduce the overall setup time. The thread
> pool should be configurable and a good choice could be
> "hbase.snapshot.thread.pool.max" that is also used in RestoreSnapshotHelper.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
