Hi Meena, Hadoop 0.20.205 and Chukwa 0.4 is a mismatch. Hadoop 0.20.20x has been modified to use MetricsSink instead of MentricsContext. Therefore, you will not see Hadoop metrics by using Chukwa 0.4. You will need Chukwa 0.5 to monitor Hadoop 0.20.205+. In addition, you should copy hadoop-metrics2.properties to $HADOOP_CONF_DIR to monitor Hadoop. Hope this helps.
regards, Eric On Tue, Apr 3, 2012 at 3:33 AM, Meena K.S <ksmeena1...@gmail.com> wrote: > Hi, > > > > We want to use chukwa for hadoop log collection. We are running hive > queries on EMR. So for the purpose of experimenting with chukwa we > configured chukwa on a 2 node EMR cluster. The collector and the > agent are running and also the system logs are generated but the > hadoop mapred and rpc /jvm logs are not being generated. Chukwa > version : 0.4 and the hadoop version :0.20.205 > > > > My hadoop-metrics.properties is as below : > > > # Licensed to the Apache Software Foundation (ASF) under one or more > > # contributor license agreements. See the NOTICE file distributed with > > # this work for additional information regarding copyright ownership. > > # The ASF licenses this file to You under the Apache License, Version 2.0 > > # (the "License"); you may not use this file except in compliance with > > # the License. You may obtain a copy of the License at > > # > > # http://www.apache.org/licenses/LICENSE-2.0 > > # > > # Unless required by applicable law or agreed to in writing, software > > # distributed under the License is distributed on an "AS IS" BASIS, > > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > > # See the License for the specific language governing permissions and > > # limitations under the License. > > > > log4j.appender.chukwa.rpc.recordType=HadoopMetricsProcessor > > log4j.appender.chukwa.rpc.chukwaClientHostname=localhost > > log4j.appender.chukwa.rpc.chukwaClientPortNum=9093 > > log4j.appender.chukwa.rpc.DatePattern=.yyyy-MM-dd > > log4j.appender.chukwa.rpc.layout=org.apache.log4j.PatternLayout > > log4j.appender.chukwa.rpc.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n > > log4j.appender.chukwa.rpc.Dir=/tmp/chukwa/log/metrics > > > > log4j.appender.chukwa.dfs.recordType=HadoopMetricsProcessor > > log4j.appender.chukwa.dfs.chukwaClientHostname=localhost > > log4j.appender.chukwa.dfs.chukwaClientPortNum=9093 > > log4j.appender.chukwa.dfs.DatePattern=.yyyy-MM-dd > > log4j.appender.chukwa.dfs.layout=org.apache.log4j.PatternLayout > > log4j.appender.chukwa.dfs.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n > > log4j.appender.chukwa.dfs.Dir=/tmp/chukwa/log/metrics > > > > log4j.appender.chukwa.mapred.recordType=HadoopMetricsProcessor > > log4j.appender.chukwa.mapred.chukwaClientHostname=localhost > > log4j.appender.chukwa.mapred.chukwaClientPortNum=9093 > > log4j.appender.chukwa.mapred.DatePattern=.yyyy-MM-dd > > log4j.appender.chukwa.mapred.layout=org.apache.log4j.PatternLayout > > log4j.appender.chukwa.mapred.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n > > log4j.appender.chukwa.mapred.Dir=/tmp/chukwa/log/metrics > > > > I have also copied chukwa-hadoop-*-client.jar and json.jar to > HADOOP_HOME/lib, and restarted hadoop, but still the logs were not > generated. > > Also when I run telnet localhost 9093, the jvm or other hadoop metrics > are not running, but the other system adapters are running correctly. > > Can someone help me here.. > > Thanks > Meena
