From log below, hbase-rs4 was writing to the datanode. Can you take a look at region server log and see if there is some clue ?
Thanks > On Jul 28, 2015, at 9:41 AM, Akmal Abbasov <akmal.abba...@icloud.com> wrote: > > Hi, I’m observing strange behaviour in HDFS/HBase cluster. > The disk space of one of datanodes is increasing very fast even when there > are no write requests. > It is 8GB per hour in average. Here is the graph which shows it. > <screenshot.png> > I am using hbase-0.98.7-hadoop2 and hadoop-2.5.1. > > And this is logs from the node > 2015-07-28 15:40:38,795 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: > /10.32.1.12:50010, dest: /10.32.0.140:38699, bytes: 1071, op: HDFS_READ, > cliID: DFSClient_NONMAPREDUCE_-689748537_1, offset: 0, srvID: > 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: > BP-439084760-10.32.0.180-1387281790961:blk_1074784244_1045663, duration: > 17759797 > 2015-07-28 15:41:15,111 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311714_1574450 src: > /10.0.0.21:60540 dest: /10.32.1.12:50010 > 2015-07-28 15:41:15,304 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: > /10.0.0.21:59054, dest: /10.32.1.12:50010, bytes: 124121, op: HDFS_WRITE, > cliID: DFSClient_hb_rs_hbase-rs4,60020,1438094355024_530940245_35, offset: 0, > srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: > BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, duration: > 3600203675041 > 2015-07-28 15:41:15,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > PacketResponder: > BP-439084760-10.32.0.180-1387281790961:blk_1075311536_1574238, > type=HAS_DOWNSTREAM_IN_PIPELINE terminating > 2015-07-28 15:50:40,745 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > Slow BlockReceiver write data to disk cost:6099ms (threshold=300ms) > 2015-07-28 15:59:21,130 INFO > org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool > BP-439084760-10.32.0.180-1387281790961 Total blocks: 65856, missing metadata > files:0, missing block files:0, missing blocks in memory:0, mismatched > blocks:0 > 2015-07-28 16:00:16,770 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311715_1574451 src: > /10.32.1.12:36998 dest: /10.32.1.12:50010 > 2015-07-28 16:00:17,469 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: > /10.32.1.12:36150, dest: /10.32.1.12:50010, bytes: 32688, op: HDFS_WRITE, > cliID: DFSClient_hb_rs_hbase-rs5,60020,1438088401479_1146354759_35, offset: > 0, srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: > BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, duration: > 3601152263901 > 2015-07-28 16:00:17,472 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > PacketResponder: > BP-439084760-10.32.0.180-1387281790961:blk_1075311706_1574442, > type=HAS_DOWNSTREAM_IN_PIPELINE terminating > 2015-07-28 16:03:44,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311716_1574452 src: > /10.0.0.19:35851 dest: /10.32.1.12:50010 > 2015-07-28 16:03:44,169 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: > /10.0.0.20:40176, dest: /10.32.1.12:50010, bytes: 316062, op: HDFS_WRITE, > cliID: DFSClient_hb_rs_hbase-rs1,60020,1438092204868_-99326843_35, offset: 0, > srvID: 6c25ffd4-3dc7-4e3a-af56-5cc8aa9220e0, blockid: > BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, duration: > 3600482062810 > 2015-07-28 16:03:44,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > PacketResponder: > BP-439084760-10.32.0.180-1387281790961:blk_1075311707_1574443, > type=LAST_IN_PIPELINE, downstreams=0:[] terminating > 2015-07-28 16:11:10,961 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > Slow BlockReceiver write data to disk cost:7884ms (threshold=300ms) > 2015-07-28 16:11:14,122 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > Slow BlockReceiver write data to disk cost:4362ms (threshold=300ms) > 2015-07-28 16:11:14,123 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > Slow flushOrSync took 3160ms (threshold=300ms), isSync:false, > flushTotalNanos=3160364203ns > 2015-07-28 16:13:29,968 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > Slow BlockReceiver write data to disk cost:659ms (threshold=300ms) > 2015-07-28 16:18:33,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Receiving BP-439084760-10.32.0.180-1387281790961:blk_1075311717_1574453 src: > /10.0.0.20:41527 dest: /10.32.1.12:50010 > 2015-07-28 16:18:38,926 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > Slow BlockReceiver write data to disk cost:1767ms (threshold=300ms) > 2015-07-28 16:28:40,580 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > Slow BlockReceiver write data to disk cost:4099ms (threshold=300ms) > > What could be the cause of this? > Thank you. > >
