Re: 回复: Hbase cluster is suddenly unable to respond

2016-08-05 Thread kiran
Hbase client and server are in same version 0.98.7. We are having complete downtime of about 30min and high cpu usage in the node and network in the cluster. On Sat, Aug 6, 2016 at 11:38 AM, kiran wrote: > We are also facing the same issue. Please tell us what is the solution. I > have increased

Re: 回复: Hbase cluster is suddenly unable to respond

2016-08-05 Thread kiran
We are also facing the same issue. Please tell us what is the solution. I have increased the rpc timeout and caching is reduced but with no effect. We are using hbase 0.98.7. Please suggest a work around as we are facing the issue very frequently now and we are having downtime in production. On Fr

Re: How to configure HBase in a HA mode?

2016-08-05 Thread Dima Spivak
Glad you got it sorted out. Happy HBase-ing! :) -Dima On Friday, August 5, 2016, Alexandr Porunov wrote: > Hello Dima, > > I figured out what it was. The problem was with the old znode which hasn't > been configured properly. I've removed /hbase znode (rmr /hbase) and > restarted hbase. Now it

Re: Hbase regionserver.MultiVersionConcurrencyControl Warning

2016-08-05 Thread Bryan Beaudreault
I'm also interested in an answer here. We see this from time to time in our production HBase clusters (non-opentsdb). It seems to be related to contention under heavy reads or heavy writes. But it's not clear what the impact is here. On Fri, Aug 5, 2016 at 5:14 AM Sterfield wrote: > Hi, > > I'm

Re: How to configure HBase in a HA mode?

2016-08-05 Thread Alexandr Porunov
Hello Dima, I figured out what it was. The problem was with the old znode which hasn't been configured properly. I've removed /hbase znode (rmr /hbase) and restarted hbase. Now it works properly. Thanks again for the help Sincerely, Alexandr On Fri, Aug 5, 2016 at 2:35 PM, Alexandr Porunov wro

Re: How to configure HBase in a HA mode?

2016-08-05 Thread Alexandr Porunov
Hello Dima, I have 4 nodes hadoopActiveMaster - Zookeeper, NN active master, journal, zkfc hadoopStandby - Zookeeper, NN standby master, journal, zkfc hadoopSlave1 - Zookeeper, data node, journal hadoopSlave2 - data node /etc/hosts - http://paste.openstack.org/show/550399/ /usr/hadoop/etc/hadoop/

Re: How to configure HBase in a HA mode?

2016-08-05 Thread Dima Spivak
Hey Alexandr, What does your hbase-site and hdfs-site look like? Wanna upload them to Gist or something similar and then paste a link? -Dima On Friday, August 5, 2016, Alexandr Porunov wrote: > Hello Dima, > > Thank you for advice. But the problem haven't disappeared. When I start > HMaster on

Re: How to configure HBase in a HA mode?

2016-08-05 Thread Alexandr Porunov
Hello Dima, Thank you for advice. But the problem haven't disappeared. When I start HMaster on nn1 and nn2 nodes they work but when I try to connect to the nn1 (http://nn1:16010/) HMaster on nn1 crashes. HMaster on nn2 continue be available via http://nn2:16010/ . Don't you know why it is happens?

Hbase regionserver.MultiVersionConcurrencyControl Warning

2016-08-05 Thread Sterfield
Hi, I'm currently testing Hbase 1.2.1 + OpenTSDB. For that, I'm generating a high load of HTTP PUT on OpenTSDB, which then writes in Hbase. Currently, I'm able to feed 300k data point per seconds, and I'm trying to achieve higher speed. I have also activate JMX on both Master and Region servers i