[jira] [Comment Edited] (HBASE-20145) HMaster start fails with IllegalStateException when HADOOP_HOME is set
[
https://issues.apache.org/jira/browse/HBASE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437565#comment-16437565
]
Artem Ervits edited comment on HBASE-20145 at 4/13/18 5:10 PM:
---
I just got the same issue and can confirm that it's mixing jars on the classpath
{code:java}
/opt/hbase/hbase-2.0.0/lib/hadoop-annotations-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-auth-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-client-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-common-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-common-2.7.4-tests.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-distcp-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-hdfs-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-hdfs-2.7.4-tests.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-app-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-common-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-core-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-hs-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-jobclient-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-shuffle-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-minicluster-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-yarn-api-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-yarn-client-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-yarn-common-2.7.4.jar{code}
the key issue for me was omission of -Dhadoop.profile=3.0 as the stackoverflow
article by [~rohithsharma] specifies. I only used -Dhadoop.version=3.1.0.
was (Author: dbist13):
I just got the same issue and can confirm that it's mixing jars on the classpath
{code:java}
/opt/hbase/hbase-2.0.0/lib/hadoop-annotations-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-auth-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-client-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-common-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-common-2.7.4-tests.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-distcp-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-hdfs-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-hdfs-2.7.4-tests.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-app-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-common-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-core-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-hs-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-jobclient-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-mapreduce-client-shuffle-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-minicluster-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-yarn-api-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-yarn-client-2.7.4.jar:/opt/hbase/hbase-2.0.0/lib/hadoop-yarn-common-2.7.4.jar{code}
> HMaster start fails with IllegalStateException when HADOOP_HOME is set
> --
>
> Key: HBASE-20145
> URL: https://issues.apache.org/jira/browse/HBASE-20145
> Project: HBase
> Issue Type: Bug
> Environment: HBase-2.0-beta1.
> Hadoop trunk version.
> java version "1.8.0_144"
>Reporter: Rohith Sharma K S
>Assignee: Wei-Chiu Chuang
>Priority: Critical
>
> It is observed that HMaster start is failed when HADOOP_HOME is set as env
> while starting HMaster. HADOOP_HOME is pointing to Hadoop trunk version.
> {noformat}
> 2018-03-07 16:59:52,654 ERROR [master//10.200.4.200:16000] master.HMaster:
> Failed to become active master
> java.lang.IllegalStateException: The procedure WAL relies on the ability to
> hsync for proper operation during component failures, but the underlying
> filesystem does not support doing so. Please check the config value of
> 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness
> and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount
> that can provide it.
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532)
> at
> org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The same configs is working in HBase-1.2.6 build properly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20145) HMaster start fails with IllegalStateException when HADOOP_HOME is set
[
https://issues.apache.org/jira/browse/HBASE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437475#comment-16437475
]
Wei-Chiu Chuang edited comment on HBASE-20145 at 4/13/18 4:53 PM:
--
Thanks [~rohithsharma] now I understand why:
Turns out that if you point env HADOOP_HOME to a Hadoop3 path with hbase2.0
convenience binary, the classpath will contain both Hadoop 2.7.4 and Hadoop 3
jar files. Effectively causing class conflicts.
Since Hadoop 3 jars are available, HBase is able to query StreamCapabilities of
the output stream. However, because Hadoop 2.7.4 jars is loaded before Hadoop 3
jars in classpath, it loaded the HdfsDataOutputStream class implemented in
Hadoop 2.7.4, which does not inherit from StreamCapabilities.
{code:java}
if (StreamCapabilities.CLASS.isAssignableFrom(stream.getClass())) {
try {
result = ((Boolean)StreamCapabilities.METHOD.invoke(stream,
capability)).booleanValue();
} catch
(IllegalAccessException|IllegalArgumentException|InvocationTargetException
exception) {
LOG.warn("Your Hadoop installation's StreamCapabilities
implementation doesn't match " +
"our understanding of how it's supposed to work. Please file a
JIRA and include " +
"the following stack trace. In the mean time we're interpreting
this behavior " +
"difference as a lack of capability support, which will probably
cause a failure.",
exception);
}
}
{code}
This is not a problem in the code itself. But the code can be improved to log a
warning message when output stream class does not inherit from
StreamCapabilities.
was (Author: jojochuang):
Thanks [~rohithsharma] now I understand why:
Turns out that if you point env HADOOP_HOME to a Hadoop3 path with hbase2.0
convenience binary, the classpath will contain both Hadoop 2.7.4 and Hadoop 3
jar files. Effectively causing class conflicts.
Since Hadoop 3 jars are available, HBase is able to query StreamCapabilities of
the output stream. However, because Hadoop 2.7.4 jars is loaded before Hadoop 3
jars in classpath, it loaded the HdfsDataOutputStream class implemented in
Hadoop 2.7.4, which does not inherit from StreamCapabilities.
{code:java}
if (StreamCapabilities.CLASS.isAssignableFrom(stream.getClass())) {
try {
LOG.warn("stream type = " +
stream.getWrappedStream().getClass().getCanonicalName());
result = ((Boolean)StreamCapabilities.METHOD.invoke(stream,
capability)).booleanValue();
} catch
(IllegalAccessException|IllegalArgumentException|InvocationTargetException
exception) {
LOG.warn("Your Hadoop installation's StreamCapabilities
implementation doesn't match " +
"our understanding of how it's supposed to work. Please file a
JIRA and include " +
"the following stack trace. In the mean time we're interpreting
this behavior " +
"difference as a lack of capability support, which will probably
cause a failure.",
exception);
}
}
{code}
This is not a problem in the code itself. But the code can be improved to log
warning message when output stream class does not inherit from
StreamCapabilities.
> HMaster start fails with IllegalStateException when HADOOP_HOME is set
> --
>
> Key: HBASE-20145
> URL: https://issues.apache.org/jira/browse/HBASE-20145
> Project: HBase
> Issue Type: Bug
> Environment: HBase-2.0-beta1.
> Hadoop trunk version.
> java version "1.8.0_144"
>Reporter: Rohith Sharma K S
>Assignee: Wei-Chiu Chuang
>Priority: Critical
>
> It is observed that HMaster start is failed when HADOOP_HOME is set as env
> while starting HMaster. HADOOP_HOME is pointing to Hadoop trunk version.
> {noformat}
> 2018-03-07 16:59:52,654 ERROR [master//10.200.4.200:16000] master.HMaster:
> Failed to become active master
> java.lang.IllegalStateException: The procedure WAL relies on the ability to
> hsync for proper operation during component failures, but the underlying
> filesystem does not support doing so. Please check the config value of
> 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness
> and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount
> that can provide it.
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532)
> at
> org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.
[jira] [Comment Edited] (HBASE-20145) HMaster start fails with IllegalStateException when HADOOP_HOME is set
[
https://issues.apache.org/jira/browse/HBASE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436326#comment-16436326
]
Wei-Chiu Chuang edited comment on HBASE-20145 at 4/12/18 10:55 PM:
---
It doesn't seem to reproduce for me – tried hadoop trunk + hbase master, hadoop
3.0.0-beta1 + hbase 2.0.0-beta1. Neither reproduced.
[~rohithsharma] could you share your hadoop and hbase configuration? Also, do
hdfs ec -getPolicy -path /
hdfs ec -getPolicy -path /hbase
See if what the output looks like for both directories.
I would actually think it makes sense to add additional check to see if the
file system directory is erasure coded, and log extra message to avoid
confusion.
was (Author: jojochuang):
It doesn't seem to reproduce for me – tried hadoop trunk + hbase master, hadoop
3.0.0-beta1 + hbase 2.0.0-beta1. Neither reproduced.
[~rohithsharma] could you share your hadoop and hbase configuration?
I would actually think it makes sense to add additional check to see if the
file system directory is erasure coded, and log extra message to avoid
confusion.
> HMaster start fails with IllegalStateException when HADOOP_HOME is set
> --
>
> Key: HBASE-20145
> URL: https://issues.apache.org/jira/browse/HBASE-20145
> Project: HBase
> Issue Type: Bug
> Environment: HBase-2.0-beta1.
> Hadoop trunk version.
> java version "1.8.0_144"
>Reporter: Rohith Sharma K S
>Assignee: Wei-Chiu Chuang
>Priority: Critical
>
> It is observed that HMaster start is failed when HADOOP_HOME is set as env
> while starting HMaster. HADOOP_HOME is pointing to Hadoop trunk version.
> {noformat}
> 2018-03-07 16:59:52,654 ERROR [master//10.200.4.200:16000] master.HMaster:
> Failed to become active master
> java.lang.IllegalStateException: The procedure WAL relies on the ability to
> hsync for proper operation during component failures, but the underlying
> filesystem does not support doing so. Please check the config value of
> 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness
> and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount
> that can provide it.
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532)
> at
> org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The same configs is working in HBase-1.2.6 build properly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20145) HMaster start fails with IllegalStateException when HADOOP_HOME is set
[
https://issues.apache.org/jira/browse/HBASE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436326#comment-16436326
]
Wei-Chiu Chuang edited comment on HBASE-20145 at 4/12/18 10:55 PM:
---
It doesn't seem to reproduce for me – tried hadoop trunk + hbase master, hadoop
3.0.0-beta1 + hbase 2.0.0-beta1, hadoop trunk + hbase 2.0.0-beta1. Neither
reproduced.
[~rohithsharma] could you share your hadoop and hbase configuration? Also, do
hdfs ec -getPolicy -path /
hdfs ec -getPolicy -path /hbase
See if what the output looks like for both directories.
I would actually think it makes sense to add additional check to see if the
file system directory is erasure coded, and log extra message to avoid
confusion.
was (Author: jojochuang):
It doesn't seem to reproduce for me – tried hadoop trunk + hbase master, hadoop
3.0.0-beta1 + hbase 2.0.0-beta1. Neither reproduced.
[~rohithsharma] could you share your hadoop and hbase configuration? Also, do
hdfs ec -getPolicy -path /
hdfs ec -getPolicy -path /hbase
See if what the output looks like for both directories.
I would actually think it makes sense to add additional check to see if the
file system directory is erasure coded, and log extra message to avoid
confusion.
> HMaster start fails with IllegalStateException when HADOOP_HOME is set
> --
>
> Key: HBASE-20145
> URL: https://issues.apache.org/jira/browse/HBASE-20145
> Project: HBase
> Issue Type: Bug
> Environment: HBase-2.0-beta1.
> Hadoop trunk version.
> java version "1.8.0_144"
>Reporter: Rohith Sharma K S
>Assignee: Wei-Chiu Chuang
>Priority: Critical
>
> It is observed that HMaster start is failed when HADOOP_HOME is set as env
> while starting HMaster. HADOOP_HOME is pointing to Hadoop trunk version.
> {noformat}
> 2018-03-07 16:59:52,654 ERROR [master//10.200.4.200:16000] master.HMaster:
> Failed to become active master
> java.lang.IllegalStateException: The procedure WAL relies on the ability to
> hsync for proper operation during component failures, but the underlying
> filesystem does not support doing so. Please check the config value of
> 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness
> and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount
> that can provide it.
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532)
> at
> org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The same configs is working in HBase-1.2.6 build properly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20145) HMaster start fails with IllegalStateException when HADOOP_HOME is set
[
https://issues.apache.org/jira/browse/HBASE-20145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389477#comment-16389477
]
Duo Zhang edited comment on HBASE-20145 at 3/7/18 12:21 PM:
Do you have erasure coding enabled on that directory?
was (Author: apache9):
Do you have ensure coding enabled on that directory?
> HMaster start fails with IllegalStateException when HADOOP_HOME is set
> --
>
> Key: HBASE-20145
> URL: https://issues.apache.org/jira/browse/HBASE-20145
> Project: HBase
> Issue Type: Bug
> Environment: HBase-2.0-beta1.
> Hadoop trunk version.
> java version "1.8.0_144"
>Reporter: Rohith Sharma K S
>Priority: Critical
>
> It is observed that HMaster start is failed when HADOOP_HOME is set as env
> while starting HMaster. HADOOP_HOME is pointing to Hadoop trunk version.
> {noformat}
> 2018-03-07 16:59:52,654 ERROR [master//10.200.4.200:16000] master.HMaster:
> Failed to become active master
> java.lang.IllegalStateException: The procedure WAL relies on the ability to
> hsync for proper operation during component failures, but the underlying
> filesystem does not support doing so. Please check the config value of
> 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness
> and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount
> that can provide it.
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1036)
> at
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:374)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:532)
> at
> org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1232)
> at
> org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1145)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:837)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2026)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The same configs is working in HBase-1.2.6 build properly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
