Hi all, Below error occurs occasionally. What's the possible cause? I am using Storm 1.1.0. And I have two bolt writing to HBase, both have two tasks. But the other bolt does not have this error. Their writing speeds are approximately the same. And the keyspace is far more larger than the other. Its keys are more than likely to duplicate in one batch write.
Time Error Host Error Port Error 2017-09-21T08:25:54.000Z hadoop182 6702<http://hadoop182:8000/log?file=DPITopology-45-1505979705%2F6702%2Fworker.log> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 376 actions: org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce {-6101012730784879159, 8964218331247407092} on row [c� Q�(E2�] may have already completed at org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4351) at org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4315) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3563) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3496) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29931) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:96) at java.lang.Thread.run(Thread.java:745) : 1 time, org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce {-6101012730784879159, 2091417366752441428} on row [c� Q�" x] may have already completed at org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4351) at org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4315) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3563) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3496) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29931) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:96) at java.lang.Thread.run(Thread.java:745) : 1 time, org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce {-6101012730784879159, 4457764140163160597} on row [c� Qگ�^�] may have already completed at org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4351) at org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4315) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3563) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3496) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29931) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:96) at java.lang.Thread.run(Thread.java:745) : 1 time, org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce {-6101012730784879159, -9075865740596317521} on row [c� Q�$��] may have already completed at org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4351) at org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4315) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3563) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3496) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29931) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:96) at java.lang.Thread.run(Thread.java:745) : 1 time, org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce {-6101012730784879159, -6353626281138241034} on row [c� Q�fu(�] may have already completed at org.apache.hadoop.hbase.regionserver.HRegionServer.startNonceOperation(HRegionServer.java:4351) at org.apache.hadoop.hbase.regionserver.HRegionServer.increment(HRegionServer.java:4315) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3563) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3496) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29931) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:96) at java.lang.Thread.run(Thread.java:745) : 1 time, org.apache.hadoop.hbase.exceptions.OperationConflictException: The operation with nonce {-6101012730784879159, -752554279863447509} on row [c� Q�'լ] may have already completed at ________________________________ [email protected]
