[
https://issues.apache.org/jira/browse/HBASE-29259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17943799#comment-17943799
]
Duo Zhang commented on HBASE-29259:
-----------------------------------
I dumped the proto output and its timestamp.
{noformat}
2025-04-12T11:29:10,562 INFO [master/meta01:16000:becomeActiveMaster]
region.RegionProcedureStore: =======class_name:
"org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure"
proc_id: 18446744073709551615
submitted_time: 0
owner: "zhangduo"
state: INITIALIZING
last_update: 0
state_message {
type_url: "type.googleapis.com/hbase.pb.RegionRemoteProcedureBaseStateData"
value:
"\nB\b\257\274\231\311\3422\022\'\n\adefault\022\034IntegrationTestBigLinkedList\032\004\330j\215\357\"\004\3422h\303(\0000\0008\000\022\022\n\006data04\020\224}\030\317\266\315\312\3422\030\004"
}
state_message {
type_url: "type.googleapis.com/hbase.pb.CloseRegionProcedureStateData"
value: "\n\022\n\006data04\020\224}\030\317\266\315\312\3422\020\000"
}
executed: false
,
\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF/proc:d/1744450655311/Put/vlen=340/seqid=3942283
{noformat}
According to the timestamp, this log message seems related
{noformat}
2025-04-12T09:37:35,311 INFO [PEWorker-3] procedure.ServerCrashProcedure:
pid=411432, state=RUNNABLE:SERVER_CRASH_ASSIGN, hasLock=true;
ServerCrashProcedure data04,16020,1744450050895, splitWal=true, meta=true found
RIT pid=411426, ppid=411395,
state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, hasLock=true;
TransitRegionStateProcedure table=IntegrationTestBigLinkedList,
region=a13d6f17eba604f7e37d981aefc62212, REOPEN/MOVE; state=CLOSING,
location=data04,16020,1744450050895, table=IntegrationTestBigLinkedList,
region=a13d6f17eba604f7e37d981aefc62212
{noformat}
1744450655311 is exactly 2025-04-12T09:37:35,311.
> Master crash when loading procedures
> ------------------------------------
>
> Key: HBASE-29259
> URL: https://issues.apache.org/jira/browse/HBASE-29259
> Project: HBase
> Issue Type: Bug
> Reporter: Duo Zhang
> Priority: Major
>
> Hit this error when running ITBLL
> {noformat}
> 2025-04-12T10:32:50,541 ERROR [master/meta01:16000:becomeActiveMaster]
> master.HMaster: Failed to become active master
> java.lang.UnsupportedOperationException: Unexpected INITIALIZING state for
> pid=-1, state=INITIALIZING, hasLock=false; CloseRegionProcedure
> a13d6f17eba604f7e37d981aefc62212, server=data04,16020,1744450050895
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.initializeStacks(ProcedureExecutor.java:453)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.loadProcedures(ProcedureExecutor.java:593)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.load(ProcedureExecutor.java:344)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.load(RegionProcedureStore.java:287)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:335)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:688)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1875)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1030)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2554)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:624)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155)
> ~[hbase-common-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:840) ~[?:?]
> 2025-04-12T10:32:50,547 ERROR [master/meta01:16000:becomeActiveMaster]
> master.HMaster: ***** ABORTING master meta01,16000,1744453967314: Unhandled
> exception. Starting shutdown. *****
> java.lang.UnsupportedOperationException: Unexpected INITIALIZING state for
> pid=-1, state=INITIALIZING, hasLock=false; CloseRegionProcedure
> a13d6f17eba604f7e37d981aefc62212, server=data04,16020,1744450050895
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.initializeStacks(ProcedureExecutor.java:453)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.loadProcedures(ProcedureExecutor.java:593)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$1.load(ProcedureExecutor.java:344)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.store.region.RegionProcedureStore.load(RegionProcedureStore.java:287)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.load(ProcedureExecutor.java:335)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:688)
> ~[hbase-procedure-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1875)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1030)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2554)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:624)
> ~[hbase-server-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hbase.trace.TraceUtil.lambda$tracedRunnable$2(TraceUtil.java:155)
> ~[hbase-common-3.0.0-beta-2-SNAPSHOT.jar:3.0.0-beta-2-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:840) ~[?:?]
> {noformat}
> Need to dig more on why this could happen.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
