Hello all,
Trying to get a Chukwa 0.3.0 installation up and running for
our Hadoop (Hadoop hadoop-0.20.1+152 from Cloudera) logs here. I've read all
the documentation, and the archives; I can't find anyone who seems to have
encountered my error before. I followed the instructions here for Collector
configuration: http://www.cs.berkeley.edu/~asrabkin/chukwa/admin.html. ant test
runs without any test failures.
The problem is that when I run tools/init.d/chukwa-collector I
don't see the daemon listening on port 8080. Here is the output from the
program launch:
[r...@hadoopm102.sacpa init.d]# ./chukwa-collector start
Starting chukwa-collector: /usr/local/chukwa/bin/chukwa-daemon.sh: line 110:
/usr/local/chukwa/bin/../var/tmp/cron.200912180201: No such file or directory
/usr/local/chukwa/bin/chukwa-daemon.sh: line 114:
/usr/local/chukwa/bin/../var/tmp/cron.200912180201: No such file or directory
/usr/local/chukwa/bin/chukwa-daemon.sh: line 121:
/usr/local/chukwa/bin/../var/tmp/cron.200912180201: No such file or directory
Registering watchdog..done
starting jettyCollector.sh, logging to
/var/chukwa/log/chukwa-chukwizzle-jettyCollector.sh-hadoopm102.sacpa.videoegg.com.out
2009-12-18 02:01:15.677::INFO: Logging to STDERR via org.mortbay.log.StdErrLog
2009-12-18 02:01:15.704::INFO: jetty-6.1.11
I have attached output showing that Chukwa is configured according to the
documentation to file, in order to keep the thread clear of clutter.
Does anyone have troubleshooting suggestions?
Thanks in advance for any help!
Chukwa is running:
[r...@hadoopm102.sacpa init.d]# ps -ef |grep org.apache.hadoop.chukwa.datacollec
tion.collector.CollectorStub
chukwa 11135 1 1 02:01 ? 00:00:01 /usr/java/latest/bin/java -DAPP=
collector -Dlog4j.configuration=chukwa-log4j.properties -DCHUKWA_HOME=/usr/local
/chukwa/bin/.. -DCHUKWA_CONF_DIR=/usr/local/chukwa/conf -DCHUKWA_LOG_DIR=/var/ch
ukwa/log -classpath /usr/local/chukwa/conf:/etc/hadoop-0.20/conf::/usr/local/chu
kwa/bin/../chukwa-core-0.3.0.jar:/usr/local/chukwa/bin/../lib/ant-1.7.1.jar:/usr
/local/chukwa/bin/../lib/ant-launcher-1.7.1.jar:/usr/local/chukwa/bin/../lib/asm
-3.1.jar:/usr/local/chukwa/bin/../lib/commons-beanutils-1.8.0.jar:/usr/local/chu
kwa/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/usr/local/chukwa/bin/../lib/commons
-codec-1.3.jar:/usr/local/chukwa/bin/../lib/commons-collections-3.1.jar:/usr/loc
al/chukwa/bin/../lib/commons-fileupload-1.2.jar:/usr/local/chukwa/bin/../lib/com
mons-httpclient-3.0.1.jar:/usr/local/chukwa/bin/../lib/commons-io-1.4.jar:/usr/l
ocal/chukwa/bin/../lib/commons-lang-2.4.jar:/usr/local/chukwa/bin/../lib/commons
-logging-1.1.1.jar:/usr/local/chukwa/bin/../lib/commons-logging-api-1.0.4.jar:/u
sr/local/chukwa/bin/../lib/commons-net-1.4.1.jar:/usr/local/chukwa/bin/../lib/co
re-3.1.1.jar:/usr/local/chukwa/bin/../lib/ezmorph-1.0.6.jar:/usr/local/chukwa/bi
n/../lib/jchronic-0.2.3.jar:/usr/local/chukwa/bin/../lib/jersey-bundle-1.1.0-ea.
jar:/usr/local/chukwa/bin/../lib/jetty-6.1.11.jar:/usr/local/chukwa/bin/../lib/j
etty-util-6.1.11.jar:/usr/local/chukwa/bin/../lib/json.jar:/usr/local/chukwa/bin
/../lib/json-lib-2.2.3-jdk15.jar:/usr/local/chukwa/bin/../lib/jsp-2.1-6.1.11.jar
:/usr/local/chukwa/bin/../lib/jsp-api-2.1-6.1.11.jar:/usr/local/chukwa/bin/../li
b/jsr311-api-1.0.jar:/usr/local/chukwa/bin/../lib/junit-3.8.1.jar:/usr/local/chu
kwa/bin/../lib/log4j-1.2.13.jar:/usr/local/chukwa/bin/../lib/mysql-connector-jav
a-5.1.10-bin.jar:/usr/local/chukwa/bin/../lib/mysql-connector-java-5.1.6.jar:/us
r/local/chukwa/bin/../lib/NagiosAppender-1.5.0.jar:/usr/local/chukwa/bin/../lib/
prefuse.jar:/usr/local/chukwa/bin/../lib/servlet-api-2.5-6.1.11.jar:/usr/lib/had
oop-0.20//hadoop-0.20.1+152-core.jar org.apache.hadoop.chukwa.datacollection.col
lector.CollectorStub
Here is the out output of the logs:
[r...@hadoopm102.sacpa log]# cat
chukwa-chukwizzle-jettyCollector.sh-hadoopm102.sacpa.videoegg.com.out
2009-12-18 02:01:15.677::INFO: Logging to STDERR via org.mortbay.log.StdErrLog
2009-12-18 02:01:15.704::INFO: jetty-6.1.11
[r...@hadoopm102.sacpa log]# cat collector.log
2009-12-18 02:01:15,240 INFO main root - initing servletCollector
2009-12-18 02:01:15,243 INFO main SeqFileWriter - rotateInterval is 300000
2009-12-18 02:01:15,243 INFO main SeqFileWriter - outputDir is /chukwa/logs/
2009-12-18 02:01:15,243 INFO main SeqFileWriter - fsname is
hdfs://hadoopm101.sacpa.videoegg.com:8020/
2009-12-18 02:01:15,243 INFO main SeqFileWriter - filesystem type from
core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
2009-12-18 02:01:16,242 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2009-12-18 02:02:16,249 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2009-12-18 02:03:16,257 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
The database is configured as the documentation directs:
mysql> show tables;
+---------------------------------+
| Tables_in_chukwizzle |
+---------------------------------+
| ClientTrace_template |
| HodJob_template |
| chukwaagent_template |
| chukwahttpsender_template |
| chunkqueue_template |
| cluster_disk_template |
| cluster_hadoop_rpc_template |
| cluster_system_metrics_template |
| dfs_datanode_template |
| dfs_fsnamesystem_template |
| dfs_namenode_template |
| dfs_throughput_template |
| disk_template |
| filesystem_fsm_template |
| hadoop_jvm_template |
| hadoop_mapred_template |
| hadoop_rpc_calls_template |
| hadoop_rpc_template |
| hdfs_usage_template |
| hod_job_digest_template |
| hod_machine_template |
| mapreduce_fsm_template |
| mr_job_conf_template |
| mr_job_template |
| mr_job_timeline_template |
| mr_task_template |
| node_activity_template |
| switch_data_template |
| system_metrics_template |
| user_job_summary_template |
| user_util_template |
| util_template |
+---------------------------------+
32 rows in set (0.00 sec)
mysql> show grants for 'root'@'localhost';
+---------------------------------------------------------------------+
| Grants for r...@localhost |
+---------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION |
Chukwa-collector.conf.xml:
<configuration>
<!-- LocalWriter parameters
<property>
<name>chukwaCollector.localOutputDir</name>
<value>/tmp/chukwa/dataSink/</value>
<description>Chukwa local data sink directory, see LocalWriter.java</descrip
tion>
</property>
<property>
<name>chukwaCollector.writerClass</name>
<value>org.apache.hadoop.chukwa.datacollection.writer.localfs.LocalWriter</v
alue>
<description>Local chukwa writer, see LocalWriter.java</description>
</property>
-->
<property>
<name>writer.hdfs.filesystem</name>
<value>hdfs://hadoopm101.sacpa.videoegg.com:8020/</value>
<description>HDFS to dump to</description>
</property>
<property>
<name>chukwaCollector.outputDir</name>
<value>/chukwa/logs/</value>
<description>Chukwa data sink directory</description>
</property>
<property>
<name>chukwaCollector.rotateInterval</name>
<value>300000</value>
<description>Chukwa rotate interval (ms)</description>
</property>
<property>
<name>chukwaCollector.http.port</name>
<value>8080</value>
<description>The HTTP port number the collector will listen on</description>
</property>
</configuration>
And finally, the contents of ${CHUKWA_HOME}/conf/collectors (which actually did
not exist until I created it):
[r...@hadoopm102.sacpa conf]# cat collectors
http://hadoopm102.sacpa.videoegg.com:8080/
However nothing is listening on port 8080:
[r...@hadoopm102.sacpa log]#netstat -an|grep 8080
[r...@hadoopm102.sacpa log]#
