stack commented on HBASE-20131:

I'm having a little trouble myself in this area at the moment; i.e. assign of 
regions from disabled tables. See tail of 
 for a deadlock. I have another scenario that is a little similar but upshot 
differs in that we delete assigned regions because we think them unassigned.

Lock on region operation such that only one operation can go through at a time 
helps with the race condition you identify. A check on table state before 
assign/unassign will also help but looking to see if can tease out race 
conditions around table enable/disable.

The test looks good. Will be useful verifying any fix we come up with.

Lets hold on this for now. I think the check you have is too coarse. Hence your 
queasyness I suspect. Will be back.

> NPE in MoveRegionProcedure via IntegrationTestLoadAndVerify with CM
> -------------------------------------------------------------------
>                 Key: HBASE-20131
>                 URL: https://issues.apache.org/jira/browse/HBASE-20131
>             Project: HBase
>          Issue Type: Bug
>          Components: proc-v2
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>            Priority: Critical
>             Fix For: 2.0.0
>         Attachments: HBASE-20131.001.patch
> I believe the error is that a MoveRegionProcedure comes in via ChaosMonkey 
> for an unassigned region that was from a disabled table (also due to CM) 
> which causes an NPE as we try to set a null original location into the 
> protobuf which fails.
> {noformat}
> 2018-03-02 23:07:00,146 ERROR 
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=2,port=20000] ipc.RpcServer: 
> Unexpected throwable object 
> java.lang.NullPointerException
>       at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos$MoveRegionStateData$Builder.setSourceServer(MasterProcedureProtos.java:26127)
>       at 
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.serializeStateData(MoveRegionProcedure.java:133)
>       at 
> org.apache.hadoop.hbase.procedure2.ProcedureUtil.convertToProtoProcedure(ProcedureUtil.java:198)
>       at 
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.writeEntry(ProcedureWALFormat.java:211)
>       at 
> org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.writeInsert(ProcedureWALFormat.java:222)
>       at 
> org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.insert(WALProcedureStore.java:490)
>       at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:863)
>       at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:832)
>       at 
> org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.submitProcedure(ProcedureSyncWait.java:111)
>       at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:561)
>       at org.apache.hadoop.hbase.master.HMaster.move(HMaster.java:1707)
>       at 
> org.apache.hadoop.hbase.master.MasterRpcServices.moveRegion(MasterRpcServices.java:1324)
>       at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
>       at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
>       at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
>       at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
>       at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){noformat}
> IntegrationTestLoadAndVerify also failed, but I'm not sure if it's related to 
> this, or just a problem with the test. The test failed because the table was 
> left offline after it was disabled, and appears to not have been re-enabled. 
> Still debugging that side..

This message was sent by Atlassian JIRA

Reply via email to