[
https://issues.apache.org/jira/browse/HBASE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646614#comment-16646614
]
Lavinia-Stefania Sirbu commented on HBASE-21286:
------------------------------------------------
[[email protected]]
>From my experiments, the general "formula" is total time ~= (nr of regions *
>nr of column families * nr of hfiles for a region) * latency / nr of threads.
Unfortunately, I do not have results with the improvements for the big snapshot
from s3 mentioned in the description (~26k regions, 5column families, 2 files
per column family), but I have tested with smaller ones, and the results
decreased linear whit the number of threads used.
Another test done was with a big snapshot(~26k regions, 5column families, 2
files per column family) exported to a custom file system over s3 (with a known
latency), and the results were also in compliance with the formula.
> Parallelize computeHDFSBlocksDistribution when getting splits of a
> HBaseSnapshot
> --------------------------------------------------------------------------------
>
> Key: HBASE-21286
> URL: https://issues.apache.org/jira/browse/HBASE-21286
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
> Affects Versions: 1.4.0
> Reporter: Lavinia-Stefania Sirbu
> Priority: Minor
> Attachments: HBASE-21286.branch-1.4.001.patch,
> HBASE-21286.branch-1.4.002.patch
>
>
> Even if this step is called computeHDFSBlocksDistribution, this is executed
> no matter the file system of the snapshot. For example, we have observed an
> important slowness when we have a snapshot in s3 (~26k regions, 5column
> families, 2 files per column family) the getsplits time is ~40min due to the
> calls in s3 for listing the files to get the best locations.
> Parallelizing this operation can reduce the overall setup time. The thread
> pool should be configurable and a good choice could be
> "hbase.snapshot.thread.pool.max" that is also used in RestoreSnapshotHelper.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
