[
https://issues.apache.org/jira/browse/HBASE-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Mi updated HBASE-21098:
-----------------------------
Attachment: HBASE-21098.master.002.patch
> Improve Snapshot Performance with Temporary Snapshot Directory when rootDir
> on S3
> ---------------------------------------------------------------------------------
>
> Key: HBASE-21098
> URL: https://issues.apache.org/jira/browse/HBASE-21098
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 2.1.0
> Reporter: Tyler Mi
> Priority: Major
> Attachments: HBASE-21098.master.001.patch,
> HBASE-21098.master.002.patch
>
>
> When using Apache HBase, the snapshot feature can be used to make a point in
> time recovery. To do this, HBase creates a manifest of all the files in all
> of the Regions so that those files can be referenced again when a user
> restores a snapshot. With HBase storage mode S3, developers can store their
> data off-cluster in Amazon S3. However, utilizing S3 as a FileSystem is
> inefficient in some operations, namely renames. Most Hadoop ecosystem
> applications use an atomic rename as a method of committing data. However,
> with S3, a rename is a separate copy and then a delete of every file which is
> no longer atomic and, in fact, quite costly. In addition, puts and deletes on
> S3 have latency issues that traditional filesystems do not encounter when
> manipulating the region snapshots. When HBase on S3 customers have a
> significant amount of regions, puts, deletes, and renames (the final commit
> stage of the snapshot) become the bottleneck causing snapshots to take many
> minutes or even hours to complete.
> The purpose of this patch is to increase the overall performance of snapshots
> while utilizing HBase on S3 through the use of a temporary directory for the
> snapshots that exists on a traditional filesystem to circumvent the
> bottlenecks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
