[
https://issues.apache.org/jira/browse/HBASE-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14275978#comment-14275978
]
Sean Busbey commented on HBASE-12848:
-------------------------------------
+1 reorganizing to subtask. the WAL is the obvious first step towards HBase
leveraging the feature.
patch comments
{code}
diff --git a/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
b/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
index eee5e83..ffc660b 100644
--- a/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
+++ b/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
@@ -918,6 +918,14 @@ public final class HConstants {
public static final String ENABLE_WAL_COMPRESSION =
"hbase.regionserver.wal.enablecompression";
+ /** Configuration name of HLog storage policy */
+ public static final String WAL_STORAGE_POLICY = "hbase.wal.storage.policy";
+ public static final String DEFAULT_WAL_STORAGE_POLICY = "NONE";
+ /** place only one replica in SSD and the remaining in default storage */
+ public static final String WAL_STORAGE_POLICY_ONE_SSD = "ONE_SSD";
+ /** place all replica on SSD */
+ public static final String WAL_STORAGE_POLICY_ALL_SSD = "ALL_SSD";
+
{code}
please put these somewhere other than HConstants so that they can be scoped to
LimitedPrivate(CONFIG) instead of public. Do we want to flag them as unstable
pending some benchmarks?
Maybe in DefaultWALProvider since it already talks about being FS-based? Also
would be a good place to add some javadocs about using it. we should include a
note that we only allow setting one policy for all FS-based WALs (and not e.g.
one policy for meta and another for user-data).
Also, would be good to add NONE as a policy so that folks could expressly stick
to not using SSD for the wal. (agree that for now doing NONE as default is also
good).
{code}
+ DistributedFileSystem dfs = (DistributedFileSystem)fs;
+ Class<? extends DistributedFileSystem> dfsClass = dfs.getClass();
+ Method m = null;
+ try {
+ m = dfsClass.getDeclaredMethod("setStoragePolicy",
+ new Class<?>[] { Path.class, String.class });
+ m.setAccessible(true);
+ } catch (NoSuchMethodException e) {
+ LOG.info("FileSystem doesn't support"
+ + " setStoragePolicy; --HDFS-7228 not available");
+ } catch (SecurityException e) {
+ LOG.info("Doesn't have access to setStoragePolicy on "
+ + "FileSystems --HDFS-7228 not available", e);
+ m = null; // could happen on setAccessible()
+ }
+ if (m != null) {
+ try {
+ m.invoke(dfs, this.fullPathLogDir, storagePolicy);
+ LOG.info("setting " + storagePolicy + " for " + this.fullPathLogDir);
+ } catch (Exception e) {
+ LOG.warn("Unable to set " + storagePolicy + " for " +
this.fullPathLogDir, e);
+ }
+ }
{code}
Can we move the reflection into one of the filesystem utility classes so that
other parts of HBASE-6572 could make use of it later?
> Utilize Flash storage for WAL
> -----------------------------
>
> Key: HBASE-12848
> URL: https://issues.apache.org/jira/browse/HBASE-12848
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: 12848-v1.patch
>
>
> One way to improve data ingestion rate is to make use of Flash storage.
> HDFS is doing the heavy lifting - see HDFS-7228.
> We assume an environment where:
> 1. Some servers have a mix of flash, e.g. 2 flash drives and 4 traditional
> drives.
> 2. Some servers have all traditional storage.
> 3. RegionServers are deployed on both profiles within one HBase cluster.
> This JIRA allows WAL to be managed on flash in a mixed-profile environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
