I'm also interested in an answer here. We see this from time to time in our production HBase clusters (non-opentsdb). It seems to be related to contention under heavy reads or heavy writes. But it's not clear what the impact is here.
On Fri, Aug 5, 2016 at 5:14 AM Sterfield <[email protected]> wrote: > Hi, > > I'm currently testing Hbase 1.2.1 + OpenTSDB. For that, I'm generating a > high load of HTTP PUT on OpenTSDB, which then writes in Hbase. Currently, > I'm able to feed 300k data point per seconds, and I'm trying to achieve > higher speed. > > I have also activate JMX on both Master and Region servers in order to grab > a bunch of metrics from Hbase. > > At some point, I'm noticing a bunch of warnings in a region server log file > : > > 2016-08-04 05:12:06,537 WARN > [B.defaultRpcServer.handler=21,queue=3,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778} > 2016-08-04 05:12:06,538 WARN > [B.defaultRpcServer.handler=19,queue=1,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778} > 2016-08-04 05:12:06,538 WARN > [B.defaultRpcServer.handler=40,queue=4,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778} > 2016-08-04 05:12:06,548 WARN > [B.defaultRpcServer.handler=0,queue=0,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778} > 2016-08-04 05:12:06,549 WARN > [B.defaultRpcServer.handler=35,queue=5,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714778} > 2016-08-04 05:12:06,556 WARN > [B.defaultRpcServer.handler=7,queue=1,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779} > 2016-08-04 05:12:06,558 WARN > [B.defaultRpcServer.handler=20,queue=2,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779} > 2016-08-04 05:12:06,559 WARN > [B.defaultRpcServer.handler=14,queue=2,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779} > 2016-08-04 05:12:06,609 WARN > [B.defaultRpcServer.handler=42,queue=0,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779} > 2016-08-04 05:12:06,609 WARN > [B.defaultRpcServer.handler=32,queue=2,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779} > 2016-08-04 05:12:06,619 WARN > [B.defaultRpcServer.handler=2,queue=2,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779} > 2016-08-04 05:12:06,619 WARN > [B.defaultRpcServer.handler=48,queue=0,port=16020] > regionserver.MultiVersionConcurrencyControl: STUCK: > MultiVersionConcurrencyControl{readPoint=7714719, writePoint=7714779} > > Can someone shed some lights on this ? I've googling + searching the > documentation, but found no clue on this subject. > > Is it a problem ? If so, how can I fix it ? > > Thanks a lot for your help, > > Guillaume >
