[
https://issues.apache.org/jira/browse/HBASE-18806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309121#comment-16309121
]
Zheng Hu commented on HBASE-18806:
----------------------------------
I've port this patch into our cluster until now. In our online cluster, I
have a table with 512 regions , 4296 hfiles. It tooks about 36 min to
restore the source snapshot and the peer snapshot in total , and after that,
the verify replication start scan rows by reading snapshot .
However, without this patch, then tooks about 17min to restore the source
snapshot , and after the job submitted, each mapper will take about 20 min to
restore the peer snapshot (so many redundant rpc to HDFS cluster), the verify
replication will start to read rows in the final .
[~tedyu], [~zghaobac], FYI
I plan to push those verifyReplication patches into master & branch-2 before
2.0.0-beta-2, other relative issues are : HBASE-18452 , HBASE-18484.
Thanks.
> VerifyRep by snapshot need not to restore snapshot for each mapper
> ------------------------------------------------------------------
>
> Key: HBASE-18806
> URL: https://issues.apache.org/jira/browse/HBASE-18806
> Project: HBase
> Issue Type: Improvement
> Components: Replication
> Affects Versions: 2.0.0-alpha-2
> Reporter: Zheng Hu
> Assignee: Zheng Hu
> Attachments: HBASE-18806.v1.patch, HBASE-18806.v2.patch,
> HBASE-18806.v3.patch, HBASE-18806.v3.patch
>
>
> In following method stack, seems like each mapper task will restore the
> snapshot. If we verify replication by a snapshot which has many hfiles,
> then we will take long time to restore snapshot. In our cluster, we took
> ~30min for the snapshot restoring when verify a big table.
> {code}
> Verifier.map
> |--------> replicatedScanner = new TableSnapshotScanner(...)
> |-------->
> TableSnapshotScanner.init()
>
> |-----> RestoreSnapshotHelper.copySnapshotForScanner
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
