[
https://issues.apache.org/jira/browse/HBASE-24556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhuqi Jin updated HBASE-24556:
------------------------------
Description:
When we tested upgrading HBase from 2.2 to 2.3, the upgraded node failed to
start due to a known zookeeper failure - ZOOKEEPER-3056.
The error message is shown below:
{code:java}
2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master
exiting2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master
exitingjava.io.IOException: No snapshot found, but there are log entries.
Something is broken! at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at
org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
at
org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
at
org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
at
org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:259)
at
org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:189)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:210)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3035)
{code}
It can be reproduced through the following steps:
* Start a single-node HBase 2.2 with the following configuration.
{code:java}
<property>
<name>hbase.rootdir</name>
<value>/var/lib/hbase</value>
</property> <property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/var/lib/zookeeper</value>
</property>{code}
* Use HBase performance evaluation tool to generate data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
sequentialWrite 1{code}
* Upgrade the node to 2.3 with the same configuration. with the same
configuration. The new version node failed to start because of the zookeeper.
HBase 2.2 is using zookeeper-3.4.10.jar, and Kafka 2.3 are using
zookeeper-3.5.7.jar.
The bug has been fixed by ZOOKEEPER-3056 in zookeeper-3.6.0. Should we upgrade
the dependency of HBase 2.3 to use zookeeper-3.6.0.jar? or upgrade the
configuration file of zookeeper-3.5.7.jar?
was:
When we tested upgrading HBase from 2.2 to 2.3, the upgraded node failed to
start due to a known zookeeper failure - ZOOKEEPER-3056.
The error message is shown below:
{code:java}
2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master
exiting2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master
exitingjava.io.IOException: No snapshot found, but there are log entries.
Something is broken! at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240) at
org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
at
org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
at
org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
at
org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:259)
at
org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:189)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:210)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3035)
{code}
It can be reproduced through the following steps:
# Start a single-node HBase 2.2 with the following configuration.
{code:java}
<property>
<name>hbase.rootdir</name>
<value>/var/lib/hbase</value>
</property> <property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/var/lib/zookeeper</value>
</property>{code}
# Use HBase performance evaluation tool to generate data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
sequentialWrite 1{code}
# Upgrade the node to 2.3 with the same configuration. with the same
configuration. The new version node failed to start because of the zookeeper.
HBase 2.2 is using zookeeper-3.4.10.jar, and Kafka 2.3 are using
zookeeper-3.5.7.jar.
The bug has been fixed by ZOOKEEPER-3056 in zookeeper-3.6.0. Should we upgrade
the dependency of HBase 2.3 to use zookeeper-3.6.0.jar? or upgrade the
configuration file of zookeeper-3.5.7.jar?
Summary: HBase upgrade from 2.2 to 2.3 fails due to failure in
ZooKeeper (was: HBase upgrade fails from 2.2 to 2.3 fails due to failure in
ZooKeeper)
> HBase upgrade from 2.2 to 2.3 fails due to failure in ZooKeeper
> ---------------------------------------------------------------
>
> Key: HBASE-24556
> URL: https://issues.apache.org/jira/browse/HBASE-24556
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
> Affects Versions: 2.3.0
> Reporter: Zhuqi Jin
> Priority: Major
>
> When we tested upgrading HBase from 2.2 to 2.3, the upgraded node failed to
> start due to a known zookeeper failure - ZOOKEEPER-3056.
> The error message is shown below:
>
> {code:java}
> 2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master
> exiting2020-06-12 23:15:11,432 ERROR [main] master.HMasterCommandLine: Master
> exitingjava.io.IOException: No snapshot found, but there are log entries.
> Something is broken! at
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
> at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
> at
> org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
> at
> org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
> at
> org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
> at
> org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:259)
> at
> org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.startup(MiniZooKeeperCluster.java:189)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:210)
> at
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
> at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3035)
> {code}
>
> It can be reproduced through the following steps:
> * Start a single-node HBase 2.2 with the following configuration.
> {code:java}
> <property>
> <name>hbase.rootdir</name>
> <value>/var/lib/hbase</value>
> </property> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/var/lib/zookeeper</value>
> </property>{code}
> * Use HBase performance evaluation tool to generate data.
> {code:java}
> /hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
> sequentialWrite 1{code}
> * Upgrade the node to 2.3 with the same configuration. with the same
> configuration. The new version node failed to start because of the zookeeper.
> HBase 2.2 is using zookeeper-3.4.10.jar, and Kafka 2.3 are using
> zookeeper-3.5.7.jar.
> The bug has been fixed by ZOOKEEPER-3056 in zookeeper-3.6.0. Should we
> upgrade the dependency of HBase 2.3 to use zookeeper-3.6.0.jar? or upgrade
> the configuration file of zookeeper-3.5.7.jar?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
