Sergey Shelukhin created HBASE-21625:
----------------------------------------
Summary: a runnable procedure v2 does not run
Key: HBASE-21625
URL: https://issues.apache.org/jira/browse/HBASE-21625
Project: HBase
Issue Type: Bug
Reporter: Sergey Shelukhin
Haven't looked at the code much yet, but it seems rather fundamental. The
procedure comes from meta replica assignment (HBASE-21624).
There are no other runnable procedures on master - a lot of succeeded
procedures since then, and one unrelated RIT procedure waiting with timeout and
being updated peroodically.
The procedure itself is
{noformat}
157156 157155 RUNNABLE hadoop
org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure Wed Dec 19
17:20:27 PST 2018 Wed Dec 19 17:20:28 PST 2018 [ { region => {
regionId => '1', tableName => { ... }, startKey => '', endKey => '', offline =>
'false', split => 'false', replicaId => '1' }, targetServer => { hostName =>
'server1', port => '17020', startCode => '1545266805778' } }, {} ]
{noformat}
This is in PST so it's been like that for ~19 hours.
The only line involving this PID in the log is {noformat}
2018-12-19 17:20:27,974 INFO [PEWorker-4] procedure2.ProcedureExecutor:
Initialized subprocedures=[{pid=157156, ppid=157155, state=RUNNABLE,
hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}]
{noformat}
There are no other useful logs for either this PID, parent PID, or region in
question since.
All the PEWorker-s are waiting for work:
{noformat}
Thread 158 (PEWorker-16):
State: TIMED_WAITING
Blocked count: 1340
Waited count: 5064
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:171)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:153)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1949)
{noformat}
The main assignment procedure for this region is blocked on it:
{noformat}
157155 WAITING hadoop TransitRegionStateProcedure
table=hbase:meta, region=534574363, ASSIGN Wed Dec 19 17:20:27 PST 2018 Wed
Dec 19 17:20:27 PST 2018 [ { state => [ '1', '2', '3' ] }, {
regionId => '1', tableName => { ... }, startKey => '', endKey => '', offline =>
'false', split => 'false', replicaId => '1' }, { initialState =>
'REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE', lastState =>
'REGION_STATE_TRANSITION_CONFIRM_OPENED', assignCandidate => { hostName =>
'server1', port => '17020', startCode => '1545266805778' }, forceNewPlan =>
'false' } ]
2018-12-19 17:20:27,673 INFO [PEWorker-9] procedure.MasterProcedureScheduler:
Took xlock for pid=157155,
state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false;
TransitRegionStateProcedure table=hbase:meta, region=..., ASSIGN
2018-12-19 17:20:27,809 INFO [PEWorker-9]
assignment.TransitRegionStateProcedure: Starting pid=157155,
state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true;
TransitRegionStateProcedure table=hbase:meta, region=..., ASSIGN; rit=OFFLINE,
location=server1,17020,1545266805778; forceNewPlan=false, retain=false
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
