Ji Shen created HBASE-27187:
-------------------------------
Summary: master.HMaster: Failed to become active master , after
reboot the hosts
Key: HBASE-27187
URL: https://issues.apache.org/jira/browse/HBASE-27187
Project: HBase
Issue Type: Bug
Environment: HBASE 2.3.5
Hadoop3.2.2
Reporter: Ji Shen
2022-07-09 20:17:02,408 INFO
[ReadOnlyZKClient-hadoop01:2181,hadoop02:2181,hadoop03:2181@0x23c8e44a-SendThread(hadoop01:2181)]
zookeeper.ClientCnxn: Socket connection established, initiating session,
client: /172.19.21.101:57094, server: hadoop01/172.19.21.101:2181
2022-07-09 20:17:02,409 INFO
[ReadOnlyZKClient-hadoop01:2181,hadoop02:2181,hadoop03:2181@0x23c8e44a-SendThread(hadoop01:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
hadoop01/172.19.21.101:2181, sessionid = 0xff81e2d459f50248, negotiated timeout
= 40000
2022-07-09 20:17:02,434 INFO [master/hadoop01:16000:becomeActiveMaster]
region.MasterRegion: Create or load local region for table 'master:store',
\{NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1',
KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION =>
'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE
=> '65536', REPLICATION_SCOPE => '0'}ZIZIaUaUaU
2022-07-09 20:17:02,437 INFO [master/hadoop01:16000:becomeActiveMaster]
region.MasterRegionFlusherAndCompactor: Injected flushSize=134217728,
flushPerChanges=1000000, flushIntervalMs=900000
2022-07-09 20:17:02,486 INFO [master/hadoop01:16000:becomeActiveMaster]
wal.WALFactory: Instantiating WALProvider of type class
org.apache.hadoop.hbase.wal.AsyncFSWALProvider
2022-07-09 20:17:02,522 ERROR [master/hadoop01:16000:becomeActiveMaster]
master.HMaster: Failed to become active master
java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:222)
at
org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
at
org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:948)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:621)
at java.lang.Thread.run(Thread.java:748)
2022-07-09 20:17:02,523 ERROR [master/hadoop01:16000:becomeActiveMaster]
master.HMaster: ***** ABORTING master hadoop01,16000,1657369019206: Unhandled
exception. Starting shutdown. *****
java.lang.ArrayIndexOutOfBoundsException: 0
at
org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:222)
at
org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
at
org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:948)
at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:621)
at java.lang.Thread.run(Thread.java:748)
2022-07-09 20:17:02,523 INFO [master/hadoop01:16000:becomeActiveMaster]
regionserver.HRegionServer: ***** STOPPING region server
'hadoop01,16000,1657369019206' *****
2022-07-09 20:17:02,523 INFO [master/hadoop01:16000:becomeActiveMaster]
regionserver.HRegionServer: STOPPED: Stopped by
master/hadoop01:16000:becomeActiveMaster
2022-07-09 20:17:02,523 INFO [master/hadoop01:16000]
regionserver.HRegionServer: Stopping infoServer
2022-07-09 20:17:02,529 INFO [master/hadoop01:16000] handler.ContextHandler:
Stopped
o.e.j.w.WebAppContext@648d0e6d\{/,null,UNAVAILABLE}{file:/home/hadoop/app/hbase-2.3.5/hbase-webapps/master}pOpPpP
2022-07-09 20:17:02,534 INFO [master/hadoop01:16000] server.AbstractConnector:
Stopped ServerConnector@512d6e60\{HTTP/1.1,[http/1.1]}qPqPqP\{0.0.0.0:16010}xI
2022-07-09 20:17:02,534 INFO [master/hadoop01:16000] handler.ContextHandler:
Stopped
o.e.j.s.ServletContextHandler@a66e580\{/static,file:///home/hadoop/app/hbase-2.3.5/hbase-webapps/static/,UNAVAILABLE}
2022-07-09 20:17:02,534 INFO [master/hadoop01:16000] handler.ContextHandler:
Stopped
o.e.j.s.ServletContextHandler@352c44a8\{/logs,file:///home/hadoop/app/hbase-2.3.5/logs/,UNAVAILABLE}
2022-07-09 20:17:02,535 INFO [master/hadoop01:16000]
regionserver.HRegionServer: aborting server hadoop01,16000,1657369019206
2022-07-09 20:17:02,535 INFO [master/hadoop01:16000]
regionserver.HRegionServer: stopping server hadoop01,16000,1657369019206; all
regions closed.
2022-07-09 20:17:02,535 INFO [master/hadoop01:16000] hbase.ChoreService: Chore
service for: master/hadoop01:16000 had [] on shutdown
2022-07-09 20:17:02,536 WARN [master/hadoop01:16000]
master.ActiveMasterManager: Failed get of master address: java.io.IOException:
Can't get master address from ZooKeeper; znode data == null
2022-07-09 20:17:02,537 INFO [master/hadoop01:16000] hbase.ChoreService: Chore
service for: hadoop01:16000.splitLogManager. had [ScheduledChore
name=SplitLogManager Timeout Monitor, period=1000, unit=MILLISECONDS] on
shutdown
2022-07-09 20:17:02,537 INFO [master/hadoop01:16000] ipc.NettyRpcServer:
Stopping server on /172.19.21.101:16000
2022-07-09 20:17:02,638 INFO
[ReadOnlyZKClient-hadoop01:2181,hadoop02:2181,hadoop03:2181@0x23c8e44a]
zookeeper.ZooKeeper: Session: 0xff81e2d459f50248 closed
2022-07-09 20:17:02,638 INFO
[ReadOnlyZKClient-hadoop01:2181,hadoop02:2181,hadoop03:2181@0x23c8e44a-EventThread]
zookeeper.ClientCnxn: EventThread shut down for session: 0xff81e2d459f50248
2022-07-09 20:17:02,737 INFO [master/hadoop01:16000] zookeeper.ZooKeeper:
Session: 0xff81e2d459f50246 closed
2022-07-09 20:17:02,737 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down for session: 0xff81e2d459f50246
2022-07-09 20:17:02,737 INFO [master/hadoop01:16000]
regionserver.HRegionServer: Exiting; stopping=hadoop01,16000,1657369019206;
zookeeper connection closed.
2022-07-09 20:17:02,738 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: HMaster Aborted
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:261)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:149)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3085)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
