Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "Chukwa_Quick_Start" page has been changed by EricYang. http://wiki.apache.org/hadoop/Chukwa_Quick_Start?action=diff&rev1=24&rev2=25 -------------------------------------------------- - =NOTE: This document is potentially quite stale; you should probably look at the [[http://hadoop.apache.org/chukwa/docs/current/admin.html|Administration Guide]] instead. + =NOTE: This document is updated for Chukwa trunk development instruction; you should probably look at the [[http://incubator.apache.org/chukwa/docs/current/admin.html|Administration Guide]] for stable release instructions instead. + + + == Purpose == + Chukwa is a system for large-scale reliable log collection and processing with Hadoop. The Chukwa design overview discusses the overall architecture of Chukwa. You should read that document before this one. The purpose of this document is to help you install and configure Chukwa. + + == Pre-requisites == + Chukwa should work on any POSIX platform, but GNU/Linux is the only production platform that has been tested extensively. Chukwa has also been used successfully on Mac OS X, which several members of the Chukwa team use for development. + + The only absolute software requirements are Java 1.6 or better and Hadoop 0.20+. HICC, the Chukwa visualization interface, requires HBase 0.20.6. + + The Chukwa cluster management scripts rely on ssh; these scripts, however, are not required if you have some alternate mechanism for starting and stopping daemons. + + == Installing Chukwa == + A minimal Chukwa deployment has three components: + + A Hadoop and HBase cluster on which Chukwa will process data (referred to as the Chukwa cluster). + A collector process, that writes collected data to HBase. + One or more agent processes, that send monitoring data to the collector. The nodes with active agent processes are referred to as the monitored source nodes. + In addition, you may wish to run the Chukwa Demux jobs, which parse collected data, or HICC, the Chukwa visualization tool. == Introduction == The Chukwa monitoring system has a number of components. This section gives guidance on starting each of them on your local machine. You should start the collector first, then the agent, and finally any adaptors. == Compiling and installing Chukwa == + 1. Copy hbase-0.20.6.jar, hbase-0.20.6-test.jar and HBASE_HOME/lib/zookeeper-*.jar to CHUKWA_HOME/lib - 1. To compile Chukwa, just say ''ant'' in the project root directory. + 1. To compile Chukwa, just say ''ant tar'' in the project root directory. - 1. Move the compiled jars from build to the Chukwa root directory. + 1. Extract the compiled tar file from build/chukwa-0.x.y.tar.gz to the Chukwa root directory. + + == Setup Chukwa Cluster == + General Hadoop configuration is available at: [[http://hadoop.apache.org/common/docs/current/cluster_setup.html][Hadoop Configuration]] + General HBASE configuration is available at: [[http://hbase.apache.org/docs/current/api/overview-summary.html#overview_description][HBase Configuration]] + + After Hadoop and HBase has been configured properly, run: bin/hbase shell < /path/to/CHUKWA_HOME/conf/hbase.schema + This procedure initializes the default Chukwa HBase schema. == Configuring and starting the Collector == 1. Copy conf/chukwa-collector-conf.xml.template to conf/chukwa-collector-conf.xml 1. Copy conf/chukwa-env.sh-template to conf/chukwa-env.sh. - 1. Edit chukwa-env.sh. You almost certainly need to set JAVA_HOME, HADOOP_HOME, and HADOOP_CONF_DIR, at least. + 1. Edit chukwa-env.sh. You almost certainly need to set JAVA_HOME, HADOOP_HOME, HADOOP_CONF_DIR, HBASE_HOME, and HBASE_CONF_DIR at least. - 1. In the chukwa root directory, say bash bin/jettyCollector.sh'' '' + 1. In the chukwa root directory, say bash bin/chukwa collector'' '' - == Configuring and starting the Local Agent == + == Configuring and starting the local agent == 1. ''Copy conf/chukwa-agent-conf.xml.template to conf/chukwa-agent-conf.xml '' 1. ''Copy conf/collectors.template to conf/collectors '' - 1. ''In the chukwa root directory, say bash bin/agent.sh'' + 1. ''In the chukwa root directory, say bash bin/chukwa agent'' - 1. Without other options, that will have the agent push data across to the collector; if you instead say bin/agent.sh local, that will cause the agent to just print to standard out. == Starting Adaptors == - The local agent speaks a simple text-based protocol, by default over port 9093. Suppose you want Chukwa to start tailing a file /path/to/file of type MyFileType on localhost: + The local agent speaks a simple text-based protocol, by default over port 9093. Suppose you want Chukwa to monitor system metrics on the localhost: 1. Telnet to localhost 9093 - 1. Type [without quotation marks] "ADD filetailer.CharFileTailingAdaptorUTF8 aDataType /path/to/file 0" + 1. Type [without quotation marks] "add org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics SystemMetrics 60 0" 1. Type "list" -- you should see the adaptor you just started, listed as running. 1. Type "close" to break the connection. 1. If you don't have telnet, you can get the same effect using the netcat (''nc'') command line tool. - == Setting up data processing == - . See the Chukwa administration guide for instructions in setting up data processing. + == Set Up HICC == + The Hadoop Infrastructure Care Center (HICC) is the Chukwa web user interface. To set up HICC, do the following: + 1. bin/chukwa hicc + + == Data visualization == + + 1. Point web browser to http://localhost:4080/hicc/jsp/graph_explorer.jsp + 1. The default user name and password is "demo" without quotes. + 1. System Metrics collected by Chukwa collector will be browsable through graph_explorer.jsp file. +
