Hey all! I am having an issue with hadoop's daily datanode log growing to + 1.8 GB. I have 3 Nodes in my hdfs cluster, all sharing the same configuration (including same log4j.properties). While running operations and jobs equally (automatically) on whole of the nodes, only one of them (data node*03*)is having this issue with the log growing high.
/var/log/hadoop/hadoop-hadoop-datanode-ip-10-10-10-4.log The log does not show any exceptions, just many hdfs operations (read+write). I am currently running *Cloudera* *hadoop-0.20.2-cdh3u1* and this is my architecture: *MasterServer*: NameNode, JobTracker, HBase HMaster (*hbase-0.90.3-cdh3u1*), Zookeeper01 *Node01*: Data Node, TaskTracker, HBase HRegion *Node02*: Data Node, TaskTracker, HBase HRegion *Node03*: Data Node, TaskTracker, HBase HRegion *SecondaryServer*: Secondary NameNode, HBase Backup HMaster, Zookeeper02 *ServerX*: ZooKeeper03 Does any one can think of a good reason, why it happens ? why to a specific node? is is related to hbase operations? hdfs block scanner? Here is a sample of the log file: 2011-10-30 08:52:27,313 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /* 10.10.10.4*:50010, dest: /*10.10.10.4*:43447, bytes: 66564, op: HDFS_READ, cliID: DFSClient_hb_rs_ip-*10-10-10-4*.ec2.internal,60020,1318334166605_1318334167243, offset: 34500096, srvID: DS-75443592-10.93.67.113-50010-1318335522512, blockid: blk_2773771462926694276_25674, duration: 274702 2011-10-30 08:52:27,314 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /* 10.10.10.4*:50010, dest: /*10.10.10.4*:43448, bytes: 66564, op: HDFS_READ, cliID: DFSClient_hb_rs_ip-*10-10-10-4*.ec2.internal,60020,1318334166605_1318334167243, offset: 34631168, srvID: DS-75443592-10.93.67.113-50010-1318335522512, blockid: blk_2773771462926694276_25674, duration: 236691 Thanks, *Ronen.* <http://www.taykey.com/>
