Yes, we load org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint
No idea about how we led to this situation :/ We have had some trouble with some RS last week, which lead us to perform multiple hbase hack -repair - not sure it’s related On May 15, 2018 at 7:35:11 PM, rahul gidwani ([email protected]) wrote: Are you using coprocessors? Can you tell us any more about what led to this. Thanks On Tue, May 15, 2018 at 4:27 AM Kevin GEORGES <[email protected]> wrote: > We are running HBASE 1.4.0 > > > On May 15, 2018 at 1:10:15 PM, Kevin GEORGES ([email protected]) wrote: > > Hello, > > We find region server abort with the following exception: > > 2018-05-15 08:23:23,920 ERROR > [RpcServer.default.FPBQ.Fifo.handler=27,queue=7,port=16020] > regionserver.HRegion: Asked to modify this region's > (continuum,R\x0C\xF6\xF2\xBD\xD4L"\xB5\xFC\ > xC6b\x8D\xD7\xC8x$\x7F\xFA\x9F\xA4\x92,1524491062878.117704beb050dcd3920335b4b290a898.) > > memstoreSize to a negative value which is incorrect. Current > memstoreSize=-1533230, delta=480 > java.lang.Exception > at > org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1205) > > at > org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3534) > > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3102) > at > org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3044) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:894) > > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:822) > > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2376) > > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36621) > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2352) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) > 2018-05-15 08:23:23,922 FATAL [regionserver/ > dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420] > > regionserver.HRegionServer: ABORTING region server dn-35.hadoo > p.b.gra.infra.metrics.ovh.net,16020,1525859460041: Assertion failed while > closing store > continuum,RH\xFD\xA6\x88\xD7\xFB5\xBBq\xD9\xE8|\xF2I_r\x7F\xFA\xB0Y\x88,1509095154547.51aea042f53 > > 655350c0d098fd378ab9b. v. flushableSize expected=0, actual= 23429. Current > memstoreSize=-34925. Maybe a coprocessor operation failed and left the > memstore in a partially updated state. > 2018-05-15 08:23:23,922 FATAL [regionserver/ > dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420] > > regionserver.HRegionServer: RegionServer abort: loaded coproce > ssors are: [org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint > > > The error about memstoreSize becoming negative appear at a steady rate > before abort (hundreds/sec) > > Any ideas? > > Thanks, > > Kevin
