[
https://issues.apache.org/jira/browse/HBASE-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16595734#comment-16595734
]
Mingliang Liu commented on HBASE-21098:
---------------------------------------
Making temporary directory for snapshort configurable (to use HDFS) makes sense
to the use case of S3 mode. The rename in patch from {{fs}} to {{rootFs}} is
relevant and good.
# The unit test problem in previous comment is also in
{{TestSnapshotDFSTemporaryDirectory}}.
# {{SnapshotDescriptionUtils::isWithinDefaultWorkingDir()}} is flaky as it
should check is subdirectory of {{getDefaultWorkingSnapshotDir()}} instead of
{{rootdir}}.
{code}
270 /**
271 * Determines if the given workingDir is a subdirectory of the
default working snapshot directory
272 * @param workingDir a directory to check
273 * @param conf configuration for the HBase cluster
274 * @return true if the given workingDir is a subdirectory of the
default working directory for
275 * snapshots, false otherwise
276 */
277 public static boolean isWithinDefaultWorkingDir(final Path
workingDir, Configuration conf) {
278 return isSubDirectoryOf(workingDir, new
Path(conf.get(HConstants.HBASE_DIR)));
279 }
{code}
# The {{FileSystem::exist()}} before {{FileSystem::delete()}} is useless and
redundant per [HADOOP-13427]. I think this is the same case here. We can remove
it from {{SnapshortManager}}.
{code}
if (tmpFs.exists(tmpdir)) {
if (!tmpFs.delete(tmpdir, true)) {
{code}
# Document the newly added configuration "hbase.snapshot.working.dir" to
{{hbase-default.xml|}}
# Is it possible to add this in {{branch-1}}?
# Nit: it's better to put path definition and assertion together for path1 ~
path 10 in {{TestSnapshotDescriptionUtils::testIsSubDirectoryWorks}}
> Improve Snapshot Performance with Temporary Snapshot Directory when rootDir
> on S3
> ---------------------------------------------------------------------------------
>
> Key: HBASE-21098
> URL: https://issues.apache.org/jira/browse/HBASE-21098
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 3.0.0, 2.1.1
> Reporter: Tyler Mi
> Priority: Major
> Attachments: HBASE-21098.master.001.patch,
> HBASE-21098.master.002.patch, HBASE-21098.master.003.patch,
> HBASE-21098.master.004.patch, HBASE-21098.master.005.patch,
> HBASE-21098.master.006.patch
>
>
> When using Apache HBase, the snapshot feature can be used to make a point in
> time recovery. To do this, HBase creates a manifest of all the files in all
> of the Regions so that those files can be referenced again when a user
> restores a snapshot. With HBase's S3 storage mode, developers can store their
> data off-cluster on Amazon S3. However, utilizing S3 as a file system is
> inefficient in some operations, namely renames. Most Hadoop ecosystem
> applications use an atomic rename as a method of committing data. However,
> with S3, a rename is a separate copy and then a delete of every file which is
> no longer atomic and, in fact, quite costly. In addition, puts and deletes on
> S3 have latency issues that traditional filesystems do not encounter when
> manipulating the region snapshots to consolidate into a single manifest. When
> HBase on S3 users have a significant amount of regions, puts, deletes, and
> renames (the final commit stage of the snapshot) become the bottleneck
> causing snapshots to take many minutes or even hours to complete.
> The purpose of this patch is to increase the overall performance of snapshots
> while utilizing HBase on S3 through the use of a temporary directory for the
> snapshots that exists on a traditional filesystem like HDFS to circumvent the
> bottlenecks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
