[
https://issues.apache.org/jira/browse/HBASE-24430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhuqi Jin updated HBASE-24430:
------------------------------
Component/s: (was: wal)
Affects Version/s: 3.0.0-alpha-1
Description:
When we tested upgrading HBASE from 2.2 to 2.4/2.5, the upgraded node failed to
start.
The error message is shown below:
{code:java}
2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster]
wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies
on the ability to call hflush for proper operation during component failures,
but the current FileSystem does not support doing so. Please check the config
value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has
suitable capabilities for output streams.2020-05-26 02:46:33,124 ERROR
[master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The
RegionServer async write ahead log provider relies on the ability to call
hflush for proper operation during component failures, but the current
FileSystem does not support doing so. Please check the config value of
'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable
capabilities for output streams.2020-05-26 02:46:33,130 INFO
[master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AbstractFSWAL: Closed WAL:
AsyncFSWAL 9f0ca3fedcda%2C16000%2C1590461189948:(num 1590461193109)2020-05-26
02:46:33,134 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster]
master.HMaster: Failed to become active masterjava.io.IOException: cannot get
log writer at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at
org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193)
at
org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213)
at
org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310)
at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
hflush at
org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105)
... 17 more2020-05-26 02:46:33,135 ERROR
[master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: ***** ABORTING
master 9f0ca3fedcda,16000,1590461189948: Unhandled exception. Starting
shutdown. *****java.io.IOException: cannot get log writer at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at
org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193)
at
org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213)
at
org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310)
at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
hflush at
org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105)
... 17 more
{code}
It can be reproduced through the following steps:
# Start a single-node Hbase branch-2.2.
# Use hbase pe to write data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
sequentialWrite 1{code}
3. Upgrade the node to branch-2.3/master. After upgraded, as the log
suggested, hbase failed to start.
was:
When we tested upgrading HBASE from 2.2 to 2.4/2.5, the upgraded node failed to
start.
The error message is shown below:
{code:java}
2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster]
wal.AsyncFSWALProvider: The RegionServer async write ahead log provider relies
on the ability to call hflush for proper operation during component failures,
but the current FileSystem does not support doing so. Please check the config
value of 'hbase.wal.dir' and ensure it points to a FileSystem mount that has
suitable capabilities for output streams.2020-05-26 02:46:33,124 ERROR
[master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AsyncFSWALProvider: The
RegionServer async write ahead log provider relies on the ability to call
hflush for proper operation during component failures, but the current
FileSystem does not support doing so. Please check the config value of
'hbase.wal.dir' and ensure it points to a FileSystem mount that has suitable
capabilities for output streams.2020-05-26 02:46:33,130 INFO
[master/9f0ca3fedcda:16000:becomeActiveMaster] wal.AbstractFSWAL: Closed WAL:
AsyncFSWAL 9f0ca3fedcda%2C16000%2C1590461189948:(num 1590461193109)2020-05-26
02:46:33,134 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster]
master.HMaster: Failed to become active masterjava.io.IOException: cannot get
log writer at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at
org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193)
at
org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213)
at
org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310)
at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
hflush at
org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105)
... 17 more2020-05-26 02:46:33,135 ERROR
[master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: ***** ABORTING
master 9f0ca3fedcda,16000,1590461189948: Unhandled exception. Starting
shutdown. *****java.io.IOException: cannot get log writer at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157)
at
org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at
org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193)
at
org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213)
at
org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310)
at org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
hflush at
org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71)
at
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
at
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
at
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105)
... 17 more
{code}
It can be reproduced through the following steps:
# Start a single-node Hbase branch-2.2.
# Use hbase pe to write data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
sequentialWrite 1{code}
# Upgrade the node to branch-2.3/master. After upgraded, as the log
suggested, hbase failed to start.
> upgrading standalone-mode hbase from 2.2 to 2.3 fails because of local
> filesystem not supporting hflush
> -------------------------------------------------------------------------------------------------------
>
> Key: HBASE-24430
> URL: https://issues.apache.org/jira/browse/HBASE-24430
> Project: HBase
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha-1, 2.3.0
> Reporter: Zhuqi Jin
> Priority: Major
>
> When we tested upgrading HBASE from 2.2 to 2.4/2.5, the upgraded node failed
> to start.
> The error message is shown below:
> {code:java}
> 2020-05-26 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster]
> wal.AsyncFSWALProvider: The RegionServer async write ahead log provider
> relies on the ability to call hflush for proper operation during component
> failures, but the current FileSystem does not support doing so. Please check
> the config value of 'hbase.wal.dir' and ensure it points to a FileSystem
> mount that has suitable capabilities for output streams.2020-05-26
> 02:46:33,124 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster]
> wal.AsyncFSWALProvider: The RegionServer async write ahead log provider
> relies on the ability to call hflush for proper operation during component
> failures, but the current FileSystem does not support doing so. Please check
> the config value of 'hbase.wal.dir' and ensure it points to a FileSystem
> mount that has suitable capabilities for output streams.2020-05-26
> 02:46:33,130 INFO [master/9f0ca3fedcda:16000:becomeActiveMaster]
> wal.AbstractFSWAL: Closed WAL: AsyncFSWAL
> 9f0ca3fedcda%2C16000%2C1590461189948:(num 1590461193109)2020-05-26
> 02:46:33,134 ERROR [master/9f0ca3fedcda:16000:becomeActiveMaster]
> master.HMaster: Failed to become active masterjava.io.IOException: cannot get
> log writer at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at
> org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193)
> at
> org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213)
> at
> org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310)
> at
> org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
> hflush at
> org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105)
> ... 17 more2020-05-26 02:46:33,135 ERROR
> [master/9f0ca3fedcda:16000:becomeActiveMaster] master.HMaster: ***** ABORTING
> master 9f0ca3fedcda,16000,1590461189948: Unhandled exception. Starting
> shutdown. *****java.io.IOException: cannot get log writer at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:118)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createAsyncWriter(AsyncFSWAL.java:704)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:710)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:838)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:548)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:489)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:157)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:297) at
> org.apache.hadoop.hbase.master.store.LocalRegion.createWAL(LocalRegion.java:193)
> at
> org.apache.hadoop.hbase.master.store.LocalRegion.bootstrap(LocalRegion.java:213)
> at
> org.apache.hadoop.hbase.master.store.LocalRegion.create(LocalRegion.java:310)
> at
> org.apache.hadoop.hbase.master.store.LocalStore.create(LocalStore.java:148)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:969)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at
> java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException:
> hflush at
> org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:71)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:169)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:166)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:105)
> ... 17 more
> {code}
> It can be reproduced through the following steps:
> # Start a single-node Hbase branch-2.2.
> # Use hbase pe to write data.
> {code:java}
> /hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
> sequentialWrite 1{code}
> 3. Upgrade the node to branch-2.3/master. After upgraded, as the log
> suggested, hbase failed to start.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
