Hi bill, I have edited my collectors file..it contains following two lines now -
localhost http://localhost:9999/ Regards, Mohammad Tariq On Fri, Nov 11, 2011 at 3:38 AM, Bill Graham <[email protected]> wrote: > Unfortunately, the conf/collectors config file is used in different ways > (with different expected formats) by the agent and the collector. See this > discussion: > http://lucene.472066.n3.nabble.com/Chukwa-setup-issues-tt2764763.html#a2765143 > If you try to run both processes frmo the same configs, you'll run into > issues. > > On Thu, Nov 10, 2011 at 1:51 PM, TARIQ <[email protected]> wrote: >> >> Here is the collector.log file - >> >> 2011-11-11 03:20:37,394 INFO main ChukwaConfiguration - chukwaConf is >> /home/prashant/chukwa-0.4.0/bin/../conf >> 2011-11-11 03:20:38,960 INFO main root - initing servletCollector >> 2011-11-11 03:20:38,967 INFO main PipelineStageWriter - using >> pipelined writers, pipe length is 2 >> 2011-11-11 03:20:38,972 INFO Thread-6 SocketTeeWriter - listen thread >> started >> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - rotateInterval is 300000 >> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - outputDir is >> /chukwa/logs/ >> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - fsname is >> hdfs://localhost:9000/ >> 2011-11-11 03:20:38,979 INFO main SeqFileWriter - filesystem type from >> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem >> 2011-11-11 03:20:39,205 ERROR main SeqFileWriter - can't connect to >> HDFS, trying default file system instead (likely to be local) >> java.lang.NoClassDefFoundError: >> org/apache/commons/configuration/Configuration >> at >> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37) >> at >> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34) >> at >> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51) >> at >> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196) >> at >> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159) >> at >> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216) >> at >> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83) >> at >> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189) >> at >> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159) >> at >> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216) >> at >> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409) >> at >> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395) >> at >> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418) >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226) >> at >> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123) >> at >> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88) >> at >> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112) >> at >> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433) >> at >> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256) >> at >> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39) >> at >> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616) >> at >> org.mortbay.jetty.servlet.Context.startContext(Context.java:140) >> at >> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513) >> at >> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39) >> at >> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) >> at org.mortbay.jetty.Server.doStart(Server.java:222) >> at >> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39) >> at >> org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.commons.configuration.Configuration >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> ... 29 more >> >> Regards, >> Mohammad Tariq >> >> >> >> On Fri, Nov 11, 2011 at 3:09 AM, AD [via Apache Chukwa] >> <[hidden email]> wrote: >> > can you share the collector.log file? >> > Also are you setting CHUKWA_HOME variable properly ? >> > the iostat error is a bit concerning. >> > >> > On Thu, Nov 10, 2011 at 2:53 PM, Mohammad Tariq <[hidden email]> wrote: >> >> >> >> Can you share 'configured' Chukwa configuration files that are >> >> necessary in order to run Chukwa agent and collector, if you have any >> >> and if possible for you so that I can observe them and figure out what >> >> I am missing??? >> >> >> >> Regards, >> >> Mohammad Tariq >> >> >> >> >> >> >> >> On Fri, Nov 11, 2011 at 1:16 AM, Mohammad Tariq <[hidden email]> wrote: >> >> > This is the output on my terminal when i issue >> >> > "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa agent" - >> >> > prashant@ubuntu:~/chukwa-0.4.0$ java.io.IOException: Cannot run >> >> > program "/usr/bin/iostat": java.io.IOException: error=2, No such file >> >> > or directory >> >> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) >> >> > at java.lang.Runtime.exec(Runtime.java:593) >> >> > at java.lang.Runtime.exec(Runtime.java:431) >> >> > at java.lang.Runtime.exec(Runtime.java:328) >> >> > at >> >> > >> >> > org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66) >> >> > at >> >> > >> >> > org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68) >> >> > at java.util.TimerThread.mainLoop(Timer.java:512) >> >> > at java.util.TimerThread.run(Timer.java:462) >> >> > Caused by: java.io.IOException: java.io.IOException: error=2, No such >> >> > file or directory >> >> > at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) >> >> > at java.lang.ProcessImpl.start(ProcessImpl.java:65) >> >> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) >> >> > ... 7 more >> >> > java.io.IOException: Cannot run program "/usr/bin/sar": >> >> > java.io.IOException: error=2, No such file or directory >> >> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) >> >> > at java.lang.Runtime.exec(Runtime.java:593) >> >> > at java.lang.Runtime.exec(Runtime.java:431) >> >> > at java.lang.Runtime.exec(Runtime.java:328) >> >> > at >> >> > >> >> > org.apache.hadoop.chukwa.inputtools.plugin.ExecPlugin.execute(ExecPlugin.java:66) >> >> > at >> >> > >> >> > org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor$RunToolTask.run(ExecAdaptor.java:68) >> >> > at java.util.TimerThread.mainLoop(Timer.java:512) >> >> > at java.util.TimerThread.run(Timer.java:462) >> >> > Caused by: java.io.IOException: java.io.IOException: error=2, No such >> >> > file or directory >> >> > at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) >> >> > at java.lang.ProcessImpl.start(ProcessImpl.java:65) >> >> > at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) >> >> > ... 7 more >> >> > >> >> > and when i issue "prashant@ubuntu:~/chukwa-0.4.0$ bin/chukwa >> >> > collector" i get this - >> >> > prashant@ubuntu:~/chukwa-0.4.0$ 2011-11-11 01:04:25.851::INFO: >> >> > Logging to STDERR via org.mortbay.log.StdErrLog >> >> > 2011-11-11 01:04:25.894::INFO: jetty-6.1.11 >> >> > >> >> > Regards, >> >> > Mohammad Tariq >> >> > >> >> > >> >> > >> >> > On Fri, Nov 11, 2011 at 1:12 AM, Mohammad Tariq <[hidden email]> >> >> > wrote: >> >> >> Hello AD, >> >> >> this my initial_adaptors file - >> >> >> >> >> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor >> >> >> Iostat >> >> >> 60 /usr/bin/iostat -x -k 55 2 0 >> >> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df >> >> >> 60 >> >> >> /bin/df -l 0 >> >> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar >> >> >> 60 >> >> >> /usr/bin/sar -q -r -n ALL 55 0 >> >> >> add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top >> >> >> 60 >> >> >> /usr/bin/top -b -n 1 -c 0 >> >> >> >> >> >> and this is the collectors file - >> >> >> localhost >> >> >> >> >> >> I am trying to run both agent and collector on the same machine. >> >> >> >> >> >> The collector.log file looks like this - >> >> >> 2011-11-11 01:04:25,180 INFO main ChukwaConfiguration - chukwaConf >> >> >> is >> >> >> /home/prashant/chukwa-0.4.0/bin/../conf >> >> >> 2011-11-11 01:04:25,434 INFO main root - initing servletCollector >> >> >> 2011-11-11 01:04:25,438 INFO main PipelineStageWriter - using >> >> >> pipelined writers, pipe length is 2 >> >> >> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - rotateInterval is >> >> >> 300000 >> >> >> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - outputDir is >> >> >> /chukwa/logs/ >> >> >> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - fsname is >> >> >> hdfs://localhost:9000/ >> >> >> 2011-11-11 01:04:25,448 INFO main SeqFileWriter - filesystem type >> >> >> from >> >> >> core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem >> >> >> 2011-11-11 01:04:25,455 INFO Thread-6 SocketTeeWriter - listen >> >> >> thread >> >> >> started >> >> >> 2011-11-11 01:04:25,593 ERROR main SeqFileWriter - can't connect to >> >> >> HDFS, trying default file system instead (likely to be local) >> >> >> java.lang.NoClassDefFoundError: >> >> >> org/apache/commons/configuration/Configuration >> >> >> at >> >> >> >> >> >> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:196) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:189) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:159) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:216) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:409) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:395) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1418) >> >> >> at >> >> >> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1319) >> >> >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFileWriter.java:123) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112) >> >> >> at >> >> >> >> >> >> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433) >> >> >> at >> >> >> >> >> >> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256) >> >> >> at >> >> >> >> >> >> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39) >> >> >> at >> >> >> >> >> >> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616) >> >> >> at >> >> >> org.mortbay.jetty.servlet.Context.startContext(Context.java:140) >> >> >> at >> >> >> >> >> >> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513) >> >> >> at >> >> >> >> >> >> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39) >> >> >> at >> >> >> >> >> >> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) >> >> >> at org.mortbay.jetty.Server.doStart(Server.java:222) >> >> >> at >> >> >> >> >> >> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39) >> >> >> at >> >> >> >> >> >> org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:121) >> >> >> Caused by: java.lang.ClassNotFoundException: >> >> >> org.apache.commons.configuration.Configuration >> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202) >> >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190) >> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> >> >> at >> >> >> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >> >> >> ... 29 more >> >> >> >> >> >> Regards, >> >> >> Mohammad Tariq >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Nov 10, 2011 at 8:30 PM, AD <[hidden email]> wrote: >> >> >>> what happens when you tail the collector.log file? >> >> >>> can you share your initial_adaptors and collectors file in conf ? >> >> >>> >> >> >>> On Thu, Nov 10, 2011 at 8:13 AM, Mohammad Tariq <[hidden email]> >> >> >>> wrote: >> >> >>>> >> >> >>>> Hello AD, >> >> >>>> I am sorry for leaving the discussion in between, I was busy >> >> >>>> with >> >> >>>> my exams.Now I am back and need your help.I started with what you >> >> >>>> have >> >> >>>> suggested and read about the concepts (agents and collectors etc). >> >> >>>> Now >> >> >>>> I am trying to collect the data from a file on the local file >> >> >>>> system >> >> >>>> and trying to dump it into the HDFS and I am running the agent and >> >> >>>> collector on the same machine.I was able to start the agent but >> >> >>>> when >> >> >>>> I >> >> >>>> tried to start the collector I got the following line at the >> >> >>>> terminal >> >> >>>> and the terminal got stuck there itself - >> >> >>>> tariq@ubuntu:~/chukwa-0.4.0$ 2011-11-10 18:42:09.521::INFO: >> >> >>>> Logging >> >> >>>> to STDERR via org.mortbay.log.StdErrLog >> >> >>>> 2011-11-10 18:42:09.544::INFO: jetty-6.1.11 >> >> >>>> >> >> >>>> When I issued jps, I could see only the agent. >> >> >>>> >> >> >>>> Regards, >> >> >>>> Mohammad Tariq >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> On Sat, Nov 5, 2011 at 5:38 AM, Bill Graham <[hidden email]> >> >> >>>> wrote: >> >> >>>> > One difference between Chukwa and Flume is that in Chukwa you >> >> >>>> > configure >> >> >>>> > each >> >> >>>> > component individually to know how to handle different >> >> >>>> > datatypes. >> >> >>>> > This >> >> >>>> > makes >> >> >>>> > it fairly straight-forward to set up, but can be painful to >> >> >>>> > manage >> >> >>>> > for >> >> >>>> > very >> >> >>>> > large installs (so I've heard). Flume centralized the configs of >> >> >>>> > "flows" >> >> >>>> > which helps when managing larger installs, at the cost of >> >> >>>> > additional >> >> >>>> > complexity. >> >> >>>> > >> >> >>>> > On Fri, Nov 4, 2011 at 7:02 AM, AD <[hidden email]> wrote: >> >> >>>> >> >> >> >>>> >> Are you running the agent and collector on the same machine? >> >> >>>> >> Off >> >> >>>> >> the >> >> >>>> >> cuff >> >> >>>> >> its something like >> >> >>>> >> 1 - set your CHUKWA_HOME env variable >> >> >>>> >> 2 - edit conf/chukwa-env.sh and make sure JAVA_HOME and all the >> >> >>>> >> vars >> >> >>>> >> look >> >> >>>> >> good >> >> >>>> >> 3 - edit conf/initial_adaptors with the adaptors you need. >> >> >>>> >> note >> >> >>>> >> here >> >> >>>> >> that >> >> >>>> >> the dataType >> >> >>>> >> 4 - edit conf/collectors and add the hostname/ip of the >> >> >>>> >> collectors >> >> >>>> >> 5 - edit conf/agents and add ip/hostname of the agents >> >> >>>> >> 6 - edit conf/chukwa-demux.conf and make sure you have a demux >> >> >>>> >> processor >> >> >>>> >> for your dataType (name from initial_adaptors) >> >> >>>> >> On agent node run bin/chukwa agent , tail >> >> >>>> >> ${CHUKWA_LOGS}/agent.log >> >> >>>> >> On collector node run bin/chukwa collector , tail >> >> >>>> >> ${CHUKWA_LOGS]/collector.log >> >> >>>> >> On collector node run bin/chukwa Demux , tail >> >> >>>> >> ${CHUKWA_LOGS}/Demux.log >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> On Fri, Nov 4, 2011 at 9:48 AM, Mohammad Tariq <[hidden email]> >> >> >>>> >> wrote: >> >> >>>> >>> >> >> >>>> >>> Hello AD, >> >> >>>> >>> Yes, I have read the wiki and right now I am trying to set >> >> >>>> >>> up a >> >> >>>> >>> Chukwa cluster using the wiki itself. Are there any other >> >> >>>> >>> links >> >> >>>> >>> you >> >> >>>> >>> are aware of that can help me learn and use Chukwa in a better >> >> >>>> >>> fashion?? I am trying to use Chukwa for data in "The Wellsite >> >> >>>> >>> Information Transfer Standard Markup Language (WITSML)" >> >> >>>> >>> format. >> >> >>>> >>> >> >> >>>> >>> Regards, >> >> >>>> >>> Mohammad Tariq >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> On Fri, Nov 4, 2011 at 6:49 PM, AD <[hidden email]> wrote: >> >> >>>> >>> > >> >> >>>> >>> > I think it helps to read he wiki and understand the basic >> >> >>>> >>> > concepts, >> >> >>>> >>> > like agents and collectors. What kind of data do you want >> >> >>>> >>> > to >> >> >>>> >>> > parse? >> >> >>>> >>> > I have >> >> >>>> >>> > used both Flume and Chukwa myself and can say for sure >> >> >>>> >>> > Chukwa >> >> >>>> >>> > is >> >> >>>> >>> > miles >> >> >>>> >>> > easier to work with and configure. Flume may have some more >> >> >>>> >>> > advanced >> >> >>>> >>> > features, but so far i found Chukwa both easy to extend from >> >> >>>> >>> > a >> >> >>>> >>> > development >> >> >>>> >>> > standpoint, and much easier to configure. >> >> >>>> >>> > Have you read the wiki pages ? >> >> >>>> >>> > >> >> >>>> >>> > On Fri, Nov 4, 2011 at 8:39 AM, Mohammad Tariq <[hidden >> >> >>>> >>> > email]> >> >> >>>> >>> > wrote: >> >> >>>> >>> >> >> >> >>>> >>> >> Hello list, >> >> >>>> >>> >> I am totally new to Chukwa and I have to learn and start >> >> >>>> >>> >> using it >> >> >>>> >>> >> as >> >> >>>> >>> >> soon soon as possible as a part of my new project. Could >> >> >>>> >>> >> any >> >> >>>> >>> >> one >> >> >>>> >>> >> let me know >> >> >>>> >>> >> how to begin with?? It would be a great favor to me. I know >> >> >>>> >>> >> Flume >> >> >>>> >>> >> and have >> >> >>>> >>> >> used it already.Also I would like to know how different is >> >> >>>> >>> >> Chukwa >> >> >>>> >>> >> from >> >> >>>> >>> >> Flume?? >> >> >>>> >>> >> Many thanks in advance. >> >> >>>> >>> >> >> >> >>>> >>> >> Regards, >> >> >>>> >>> >> Mohammad Tariq >> >> >>>> >>> > >> >> >>>> >> >> >> >>>> > >> >> >>>> > >> >> >>> >> >> >>> >> >> >> >> >> > >> > >> > >> > >> > ________________________________ >> > If you reply to this email, your message will be added to the discussion >> > below: >> > >> > http://apache-chukwa.679492.n3.nabble.com/How-to-begin-with-Chukwa-tp3479982p3498042.html >> > To unsubscribe from Apache Chukwa, click here. >> > See how NAML generates this email >> >> ________________________________ >> View this message in context: Re: How to begin with Chukwa >> Sent from the Chukwa - Users mailing list archive at Nabble.com. > >
