Pratyush Bhatt created HBASE-27909:
--------------------------------------
Summary: NotServingRegionException while running
IntegrationTestIngest with ChaosMonkey
Key: HBASE-27909
URL: https://issues.apache.org/jira/browse/HBASE-27909
Project: HBase
Issue Type: Bug
Reporter: Pratyush Bhatt
While Running the command:
{noformat}
hbase org.apache.hadoop.hbase.IntegrationTestIngest --monkey
slowDeterministic{noformat}
Getting below errors:
{noformat}
2023-06-05 08:44:07,362 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=771,
ppid=763, state=SUCCESS; CloseRegionProcedure a5882e880b2ea5eaa33bc972ef98ae9a,
server=ozn-hbnew19-2.ozn-hbnew19.root.hwx.site,22101,1685953635506 in 12.8940
sec
2023-06-05 08:44:07,431 INFO
org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure: Starting
pid=763, ppid=731, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE,
locked=true; TransitRegionStateProcedure table=IntegrationTestIngest,
region=a5882e880b2ea5eaa33bc972ef98ae9a, REOPEN/MOVE; state=CLOSED,
location=ozn-hbnew19-2.ozn-hbnew19.root.hwx.site,22101,1685953635506;
forceNewPlan=false, retain=true
2023-06-05 08:44:07,543 ERROR org.apache.hadoop.hbase.ipc.RpcServer: Unexpected
throwable object
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.NotServingRegionException:
IntegrationTestIngest,182be0c5cdcd5072bb1864cdee4d3d6e-33,1685954003397.e02c6a03f3499091552f84577b27b942.
is closing
at
org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:274)
at
org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:115)
at
org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:126)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:160)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2023-06-05 08:44:07,550 ERROR org.apache.hadoop.hbase.procedure.Procedure:
Procedure 'IntegrationTestIngest' execution failed!
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
ozn-hbnew19-1.ozn-hbnew19.root.hwx.site,22101,1685953632178:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.NotServingRegionException:
IntegrationTestIngest,182be0c5cdcd5072bb1864cdee4d3d6e-33,1685954003397.e02c6a03f3499091552f84577b27b942.
is closing
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:81)
at
org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:382)
at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:204)
at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.NotServingRegionException:
IntegrationTestIngest,182be0c5cdcd5072bb1864cdee4d3d6e-33,1685954003397.e02c6a03f3499091552f84577b27b942.
is closing
at
org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:274)
at
org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:115)
at
org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:126)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:160)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:46)
... 4 more
2023-06-05 08:44:07,552 INFO org.apache.hadoop.hbase.procedure.ZKProcedureUtil:
Clearing all znodes for procedure IntegrationTestIngestincluding nodes
/hbase2/flush-table-proc/acquired /hbase2/flush-table-proc/reached
/hbase2/flush-table-proc/abort
2023-06-05 08:44:07,582 INFO
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer: Reassigned 1 regions.
1 retained the pre-restart assignment.
2023-06-05 08:44:07,583 INFO
org.apache.hadoop.hbase.master.assignment.RegionStateStore: pid=741 updating
hbase:meta row=76a8eaee60cc2aec2af1bbdfba6259a4, regionState=CLOSED
2023-06-05 08:44:07,650 INFO org.apache.hadoop.hbase.master.MasterRpcServices:
Client=hbase//10.64.62.16 procedure request for: flush-table-proc
2023-06-05 08:44:07,659 INFO org.apache.hadoop.hbase.procedure.Procedure:
Starting procedure 'IntegrationTestIngest'
2023-06-05 08:44:08,160 ERROR org.apache.hadoop.hbase.ipc.RpcServer: Unexpected
throwable object
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.NotServingRegionException:
IntegrationTestIngest,07507507,1685953771325.6c59d6c5a3b35231b11d5d4c854729f3.
is closing
at
org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool.waitForOutstandingTasks(RegionServerFlushTableProcedureManager.java:274)
at
org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.flushRegions(FlushTableSubprocedure.java:115)
at
org.apache.hadoop.hbase.procedure.flush.FlushTableSubprocedure.acquireBarrier(FlushTableSubprocedure.java:126)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:160)
at
org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2023-06-05 08:44:08,167 ERROR org.apache.hadoop.hbase.procedure.Procedure:
Procedure 'IntegrationTestIngest' execution failed!
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via
ozn-hbnew19-3.ozn-hbnew19.root.hwx.site,22101,1685953632330:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.NotServingRegionException:
IntegrationTestIngest,07507507,1685953771325.6c59d6c5a3b35231b11d5d4c854729f3.
is closing
at
org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:81)
at
org.apache.hadoop.hbase.procedure.Procedure.waitForLatch(Procedure.java:382)
at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:204)
at org.apache.hadoop.hbase.procedure.Procedure.call(Procedure.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable:
org.apache.hadoop.hbase.NotServingRegionException:
IntegrationTestIngest,07507507,1685953771325.6c59d6c5a3b35231b11d5d4c854729f3.
is closing{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
