[ https://issues.apache.org/jira/browse/HBASE-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390318#comment-16390318 ]
Josh Elser commented on HBASE-20131: ------------------------------------ Parking an initial patch here. I have a UT which fails without my change and passes with it. I'm not positive that this is the right fix, but I'm not sure where a proper fix would go. The fix I've added will guard against the MoveProcedure being submitted when the Region is closed. However, I don't think there's any guard against a race-condition where the Region is being closed when the Move comes in. I don't know if this race condition is something we need to explicitly handle, or if this already fails gracefully. Would greatly appreciate your opinions, mighty [~stack] or some other(s) you might suggest I bother :) > NPE in MoveRegionProcedure via IntegrationTestLoadAndVerify with CM > ------------------------------------------------------------------- > > Key: HBASE-20131 > URL: https://issues.apache.org/jira/browse/HBASE-20131 > Project: HBase > Issue Type: Bug > Components: proc-v2 > Reporter: Josh Elser > Assignee: Josh Elser > Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-20131.001.patch > > > I believe the error is that a MoveRegionProcedure comes in via ChaosMonkey > for an unassigned region that was from a disabled table (also due to CM) > which causes an NPE as we try to set a null original location into the > protobuf which fails. > {noformat} > 2018-03-02 23:07:00,146 ERROR > [RpcServer.default.FPBQ.Fifo.handler=23,queue=2,port=20000] ipc.RpcServer: > Unexpected throwable object > java.lang.NullPointerException > at > org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProcedureProtos$MoveRegionStateData$Builder.setSourceServer(MasterProcedureProtos.java:26127) > at > org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.serializeStateData(MoveRegionProcedure.java:133) > at > org.apache.hadoop.hbase.procedure2.ProcedureUtil.convertToProtoProcedure(ProcedureUtil.java:198) > at > org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.writeEntry(ProcedureWALFormat.java:211) > at > org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.writeInsert(ProcedureWALFormat.java:222) > at > org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.insert(WALProcedureStore.java:490) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:863) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:832) > at > org.apache.hadoop.hbase.master.procedure.ProcedureSyncWait.submitProcedure(ProcedureSyncWait.java:111) > at > org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:561) > at org.apache.hadoop.hbase.master.HMaster.move(HMaster.java:1707) > at > org.apache.hadoop.hbase.master.MasterRpcServices.moveRegion(MasterRpcServices.java:1324) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304){noformat} > IntegrationTestLoadAndVerify also failed, but I'm not sure if it's related to > this, or just a problem with the test. The test failed because the table was > left offline after it was disabled, and appears to not have been re-enabled. > Still debugging that side.. -- This message was sent by Atlassian JIRA (v7.6.3#76005)