Zhuqi Jin created HBASE-24430:
---------------------------------
Summary: upgrading standalone-mode hbase from 2.2 to 2.3 fails
because of local filesystem not supporting hflush
Key: HBASE-24430
URL: https://issues.apache.org/jira/browse/HBASE-24430
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 2.3.0
Reporter: Zhuqi Jin
When we tested upgrading HBASE from 2.2 to 2.4/2.5, the upgraded node failed to
start.
The error message is shown below:
{code:java}
2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster]
wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies
on the ability to call hflush for proper operation during component failures,
but the current FileSystem does not support doing so. Please check the config
value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has
suitable capabilities for output streams.2020-05-26 02:46:33,124 ERROR
[master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The
RegionServer async write ahead log provider relies on the ability to call
hflush for proper operation during component failures, but the current
FileSystem does not support doing so. Please check the config value of
'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable
capabilities for output streams.2020-05-26 02:46:33,130 INFO
[master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AbstractFSWAL: Closed WAL:
AsyncFSWAL 9f0ca3fedcda%2C16000%2C1590461189948:(num 1590461193109)2020-05-26
02:46:33,134 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster]
master.HMaster: Failed to become active masterjava.io.IOException: cannot get
log writer at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at
org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193)
at
org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213)
at
org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310)
at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
hflush at
org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105)
... 17 more2020-05-26 02:46:33,135 ERROR
[master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: ***** ABORTING
master 9f0ca3fedcda,16000,1590461189948: Unhandled exception. Starting
shutdown. *****java.io.IOException: cannot get log writer at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at
org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193)
at
org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213)
at
org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310)
at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
hflush at
org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105)
... 17 more
{code}
It can be reproduced through the following steps:
# Start a single-node Hbase branch-2.2.
# Use hbase pe to write data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
sequentialWrite 1{code}
# Upgrade the node to branch-2.3/master. After upgraded, as the log
suggested, hbase failed to start.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
