Hi,
I have a customer who use openTSDB. Recently we found that only less than 10%
data are written, rest are are lost. By checking the RS log, there are many row
lock related issues, like below. It seems large amount of write to tsdb that
need row lock caused the problem. Anyone else see similar problem? Is it a bug
of openTSDB? Or it is due to HBase exposed a vulnerable API?
org.apache.hadoop.hbase.UnknownRowLockException: Invalid row lock
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getLockFromId(HRegionServer.java:2732)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2071)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
13/09/18 12:08:30 ERROR regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownRowLockException: -6180307918863136448
at
org.apache.hadoop.hbase.regionserver.HRegionServer.unlockRow(HRegionServer.java:2765)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
Thanks
Tian-Ying
