Re: openTSDB lose large amount of data when the client are writing

2013-09-19 Thread Jean-Daniel Cryans 
Could happen if a region moves since locks aren't persisted, but if I were
you I'd ask on the opentsdb mailing list first.

J-D


On Thu, Sep 19, 2013 at 10:09 AM, Tianying Chang tich...@ebaysf.com wrote:

 Hi,

 I have a customer who use openTSDB. Recently we found that only less than
 10% data are written, rest are are lost. By checking the RS log, there are
 many row lock related issues, like below. It seems large amount of write to
 tsdb that need row lock caused the problem. Anyone else see similar
 problem?  Is it a bug of openTSDB? Or it is due to HBase exposed a
 vulnerable API?

 org.apache.hadoop.hbase.UnknownRowLockException: Invalid row lock
 at
 org.apache.hadoop.hbase.regionserver.HRegionServer.getLockFromId(HRegionServer.java:2732)
 at
 org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2071)
 at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
 at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
 at
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
 13/09/18 12:08:30 ERROR regionserver.HRegionServer:
 org.apache.hadoop.hbase.UnknownRowLockException: -6180307918863136448
 at
 org.apache.hadoop.hbase.regionserver.HRegionServer.unlockRow(HRegionServer.java:2765)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)


 Thanks
 Tian-Ying

Re: openTSDB lose large amount of data when the client are writing

2013-09-19 Thread Stack 
On Thu, Sep 19, 2013 at 10:09 AM, Tianying Chang tich...@ebaysf.com wrote:

 Hi,

 I have a customer who use openTSDB. Recently we found that only less than
 10% data are written, rest are are lost. By checking the RS log, there are
 many row lock related issues, like below. It seems large amount of write to
 tsdb that need row lock caused the problem. Anyone else see similar
 problem?  Is it a bug of openTSDB? Or it is due to HBase exposed a
 vulnerable API?

 org.apache.hadoop.hbase.UnknownRowLockException: Invalid row lock
 at
 org.apache.hadoop.hbase.regionserver.HRegionServer.getLockFromId(HRegionServer.java:2732)
 at
 org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2071)
 at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
 at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
 at
 org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
 13/09/18 12:08:30 ERROR regionserver.HRegionServer:
 org.apache.hadoop.hbase.UnknownRowLockException: -6180307918863136448
 at
 org.apache.hadoop.hbase.regionserver.HRegionServer.unlockRow(HRegionServer.java:2765)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:616)
 at
 org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)


 Thanks
 Tian-Ying



Local filesystem or hdfs?
St.Ack

openTSDB lose large amount of data when the client are writing

2013-09-19 Thread Tianying Chang 
Hi, 

I have a customer who use openTSDB. Recently we found that only less than 10% 
data are written, rest are are lost. By checking the RS log, there are many row 
lock related issues, like below. It seems large amount of write to tsdb that 
need row lock caused the problem. Anyone else see similar problem?  Is it a bug 
of openTSDB? Or it is due to HBase exposed a vulnerable API? 

org.apache.hadoop.hbase.UnknownRowLockException: Invalid row lock
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.getLockFromId(HRegionServer.java:2732)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2071)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at 
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at 
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
13/09/18 12:08:30 ERROR regionserver.HRegionServer: 
org.apache.hadoop.hbase.UnknownRowLockException: -6180307918863136448
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.unlockRow(HRegionServer.java:2765)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at 
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)


Thanks
Tian-Ying