[
https://issues.apache.org/jira/browse/HBASE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16490487#comment-16490487
]
Yu Li edited comment on HBASE-18118 at 5/25/18 9:53 AM:
--------------------------------------------------------
bq. the HDFS API shouldn't be called with NONE, based on reading the code
Yes, and to be specified, please refer to below codes in {{CommonFSUtils}}. No
real call of HDFS API if {{hbase.wal.storage.policy}} is not set or set to the
same value with {{HConstants.DEFAULT_WAL_STORAGE_POLICY}}:
{code}
public static void setStoragePolicy(final FileSystem fs, final Configuration
conf,
final Path path, final String policyKey, final String defaultPolicy) {
String storagePolicy = conf.get(policyKey,
defaultPolicy).toUpperCase(Locale.ROOT);
if (storagePolicy.equals(defaultPolicy)) {
if (LOG.isTraceEnabled()) {
LOG.trace("default policy of " + defaultPolicy + " requested, exiting
early.");
}
return;
}
setStoragePolicy(fs, path, storagePolicy);
}
{code}
This is also where the problem is w/ patch here, now if user explicitly set
{{hbase.wal.storage.policy}} to HOT, since the default policy is also HOT,
there won't be any real call. However, what if the HDFS default policy is not
HOT (e.g. ONE_SSD) but user exactly want to set the WAL storage to HOT? Current
implementation will make the setting invalid, right?
IMHO setting the default policy to some non-existing one like "NONE" is just
the correct way to handle the above case: if no {{hbase.wal.storage.policy}}
set, we will use the HDFS default policy, otherwise we will always call the API
with whatever user sets.
And it seems to me there won't be any log spew if we change the default back to
"NONE". I'm not sure if any later fix made this happen, but with current codes
it should be working well.
was (Author: carp84):
bq. the HDFS API shouldn't be called with NONE, based on reading the code
Yes, and to be specified, please refer to below codes in {{CommonFSUtils}}. No
real call of HDFS API if {{hbase.wal.storage.policy}} is not set or set to the
same value with {{HConstants.DEFAULT_WAL_STORAGE_POLICY}}:
{code}
public static void setStoragePolicy(final FileSystem fs, final Configuration
conf,
final Path path, final String policyKey, final String defaultPolicy) {
String storagePolicy = conf.get(policyKey,
defaultPolicy).toUpperCase(Locale.ROOT);
if (storagePolicy.equals(defaultPolicy)) {
if (LOG.isTraceEnabled()) {
LOG.trace("default policy of " + defaultPolicy + " requested, exiting
early.");
}
return;
}
setStoragePolicy(fs, path, storagePolicy);
}
{code}
This is also where the problem is w/ patch here, now if user explicitly set
{{hbase.wal.storage.policy}} to HOT, since the default policy is also HOT,
there won't be any real call. However, what if the HDFS default policy is not
HOT (i.e. ONE_SSD) but user exactly want to set the WAL storage to HOT? Current
implementation will make the setting invalid, right?
IMHO setting the default policy to some non-existing one like "NONE" is just
the correct way to handle the above case: if no {{hbase.wal.storage.policy}}
set, we will use the HDFS default policy, otherwise we will always call the API
with whatever user sets.
And it seems to me there won't be any log spew if we change the default back to
"NONE". I'm not sure if any later fix made this happen, but with current codes
it should be working well.
> Default storage policy if not configured cannot be "NONE"
> ---------------------------------------------------------
>
> Key: HBASE-18118
> URL: https://issues.apache.org/jira/browse/HBASE-18118
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 2.0.0
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-18118.patch
>
>
> HBase can't use 'NONE' as default storage policy if not configured because
> HDFS supports no such policy. This policy name was probably available in a
> precommit or early version of the HDFS side support for heterogeneous
> storage. Now the best default is 'HOT'.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
