[
https://issues.apache.org/jira/browse/HBASE-20824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-20824.
---------------------------
Resolution: Won't Fix
Resolving old issue about condition we probably fixed long time ago.
> IllegalArgumentException:
> state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS
> SplitTableRegionProcedure
> ------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-20824
> URL: https://issues.apache.org/jira/browse/HBASE-20824
> Project: HBase
> Issue Type: Bug
> Components: amv2
> Reporter: stack
> Priority: Major
>
> Running tests to check HBASE-20796 STUCK RIT though region successfully
> assigned, I came across this case.
> Log is throwing out these:
> {code}
> STUCK Region-In-Transition rit=CLOSED,
> location=ve0536.X.Y.Z.com,16020,1530233509248,
> table=IntegrationTestBigLinkedList, region=18643f59d7db3689eeb133062ae4bc4f
> {code}
> Backup up I got this:
> {code}
> 2018-06-28 17:57:32,744 INFO [PEWorker-3]
> procedure.MasterProcedureScheduler: pid=2716,
> state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS;
> SplitTableRegionProcedure table=IntegrationTestBigLinkedList,
> parent=18643f59d7db3689eeb133062ae4bc4f,
> daughterA=6b0d2a0d22113c529dd49fc923a35ad9,
> daughterB=be72b5916a6a11d9ddce7d8af24f129f checking lock on
> 18643f59d7db3689eeb133062ae4bc4f
> 2018-06-28 17:57:32,745 WARN [PEWorker-3] procedure2.ProcedureExecutor:
> Worker terminating UNNATURALLY null
>
> java.lang.IllegalArgumentException:
> pid=2716, state=WAITING:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS;
> SplitTableRegionProcedure table=IntegrationTestBigLinkedList,
> parent=18643f59d7db3689eeb133062ae4bc4f,
> daughterA=6b0d2a0d22113c529dd49fc923a35ad9,
> daughterB=be72b5916a6a11d9ddce7d8af24f129f
>
>
> at
> org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:134)
>
>
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1449)
>
>
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
>
>
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
>
>
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {code}
> Just before this I see the below:
> {code}
> 2018-06-28 17:57:32,604 WARN [master/ve0524:16000.Chore.1]
> master.CatalogJanitor: Failed initial scan of catalog table
>
>
> org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
>
>
> at
> org.apache.hadoop.hbase.master.HMaster.isInMaintenanceMode(HMaster.java:2823)
>
>
> at
> org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:281)
>
>
> at
> org.apache.hadoop.hbase.master.CatalogJanitor.initialChore(CatalogJanitor.java:81)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:184)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
> {code}
> We are failing this check
> Preconditions.checkArgument(procedure.getState() ==
> ProcedureState.RUNNABLE,
> procedure.toString());
> Oh, almost in the same millisecond, we get this:
> 2018-06-28 17:57:32,743 INFO [master/ve0524:16000] master.HMaster: Master
> has completed initialization 13.246sec
> Is it a race? The RUNNING state has not been set yet?
> Here was what was in meta. We'd just loaded the region:
> 2018-06-28 17:57:32,443 INFO [master/ve0524:16000]
> assignment.RegionStateStore: Load hbase:meta entry
> region=18643f59d7db3689eeb133062ae4bc4f, regionState=CLOSED,
> lastHost=ve0536.halxg.cloudera.com,16020,1530233509248,
> regionLocation=ve0536.X.Y.Z,16020,1530233509248, openSeqNum=488250
> And before the crash, last thing was
> 2018-06-28 17:57:08,609 INFO [PEWorker-16] assignment.RegionStateStore:
> pid=2717 updating hbase:meta row=18643f59d7db3689eeb133062ae4bc4f,
> regionState=CLOSED
> Before that...
> 2018-06-28 17:57:05,457 INFO [PEWorker-10]
> procedure.MasterProcedureScheduler: pid=2717, ppid=2716,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=IntegrationTestBigLinkedList, region=18643f59d7db3689eeb133062ae4bc4f,
> server=ve0536.X.Y.Z.com,16020,1530233509248 checking lock on
> 18643f59d7db3689eeb133062ae4bc4f
> pid=2716 is the split.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
