Hi Ted, sorry forget to mention > release of hbase / hadoop you're using hbase hbase-0.98.7-hadoop2, hadoop hadoop-2.5.1
> were region servers doing compaction ? I’ve run major compactions manually earlier today, but it seems that they already completed, looking at the compactionQueueSize. > have you checked region server logs ? The logs of datanode is full of this kind of messages 2015-09-02 16:37:06,950 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.10.8.55:50010, dest: /10.10.8.54:32959, bytes: 19673, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_1225374853_1, offset: 0, srvID: ee7d0634-89a3-4ada-a8ad-7848217327be, blockid: BP-329084760-10.32.0.180-1387281790961:blk_1075277914_1540222, duration: 7881815 p.s. we had to change the ip addresses of the cluster nodes, is it relevant? Thanks. > On 02 Sep 2015, at 18:20, Ted Yu <[email protected]> wrote: > > Please provide some more information: > > release of hbase / hadoop you're using > were region servers doing compaction ? > have you checked region server logs ? > > Thanks > > On Wed, Sep 2, 2015 at 9:11 AM, Akmal Abbasov <[email protected] > <mailto:[email protected]>> wrote: > Hi, > I’m having strange behaviour in hbase cluster. It is almost idle, only <5 > puts and gets. > But the data in hdfs is increasing, and region servers have very high > iowait(>100, in 2 core CPU). > iotop shows that datanode process is reading and writing all the time. > Any suggestions? > > Thanks. >
