A little SUCCESS !!! I can see that logs are read from Hadoop Logs and data written in hdfs..evident from the following command hadoop fs -ls /chukwa/logs
-rw-r--r-- 1 ec2-user supergroup 657818 2013-01-08 11:09 /chukwa/logs/201308110441175_domU12313907193B_216836a313c19d565ee8000.done -rw-r--r-- 1 ec2-user supergroup 483520 2013-01-08 11:14 /chukwa/logs/201308110941296_domU12313907193B_216836a313c19d565ee7ffe.done -rw-r--r-- 1 ec2-user supergroup 483155 2013-01-08 11:19 /chukwa/logs/201308111441296_domU12313907193B_216836a313c19d565ee7ffc.done -rw-r--r-- 1 ec2-user supergroup 463708 2013-01-08 11:24 /chukwa/logs/201308111941296_domU12313907193B_216836a313c19d565ee7ffa.done -rw-r--r-- 1 ec2-user supergroup 386337 2013-01-08 11:36 /chukwa/logs/201308113134948_domU12313907193B_488afeb413c19ee05bf8000.done -rw-r--r-- 1 ec2-user supergroup 479656 2013-01-08 11:41 /chukwa/logs/201308113635034_domU12313907193B_488afeb413c19ee05bf7ffe.done -rw-r--r-- 1 ec2-user supergroup 487748 2013-01-08 11:46 /chukwa/logs/201308114135034_domU12313907193B_488afeb413c19ee05bf7ffb.done -rw-r--r-- 1 ec2-user supergroup 479696 2013-01-08 11:51 /chukwa/logs/201308114635034_domU12313907193B_488afeb413c19ee05bf7ff8.done -rw-r--r-- 1 ec2-user supergroup 488030 2013-01-08 11:56 /chukwa/logs/201308115135034_domU12313907193B_488afeb413c19ee05bf7ff5.done -rw-r--r-- 1 ec2-user supergroup 479754 2013-01-08 12:01 /chukwa/logs/201308115635034_domU12313907193B_488afeb413c19ee05bf7ff2.done -rw-r--r-- 1 ec2-user supergroup 488067 2013-01-08 12:06 /chukwa/logs/201308120135034_domU12313907193B_488afeb413c19ee05bf7fef.done -rw-r--r-- 1 ec2-user supergroup 479961 2013-01-08 12:11 /chukwa/logs/201308120635034_domU12313907193B_488afeb413c19ee05bf7fec.done -rw-r--r-- 1 ec2-user supergroup 487976 2013-01-08 12:16 /chukwa/logs/201308121135034_domU12313907193B_488afeb413c19ee05bf7fe9.done -rw-r--r-- 1 ec2-user supergroup 479863 2013-01-08 12:21 /chukwa/logs/201308121635034_domU12313907193B_488afeb413c19ee05bf7fe6.done -rw-r--r-- 1 ec2-user supergroup 649168 2013-01-08 12:42 /chukwa/logs/201308123755390_domU12313907193B_5ab1511313c1a2ac24f8000.done -rw-r--r-- 1 ec2-user supergroup 483972 2013-01-08 12:47 /chukwa/logs/201308124255466_domU12313907193B_5ab1511313c1a2ac24f7ffc.done -rw-r--r-- 1 ec2-user supergroup 483417 2013-01-08 12:52 /chukwa/logs/201308124755466_domU12313907193B_5ab1511313c1a2ac24f7ff8.done -rw-r--r-- 1 ec2-user supergroup 576711 2013-01-08 13:06 /chukwa/logs/201308130134839_domU12313907193B_2abeddbf13c1a406b0b8000.done -rw-r--r-- 1 ec2-user supergroup 483877 2013-01-08 13:11 /chukwa/logs/201308130634912_domU12313907193B_2abeddbf13c1a406b0b7ffc.done -rw-r--r-- 1 ec2-user supergroup 484352 2013-01-08 13:16 /chukwa/logs/201308131134912_domU12313907193B_2abeddbf13c1a406b0b7ff7.done -rw-r--r-- 1 ec2-user supergroup 486158 2013-01-08 13:21 /chukwa/logs/201308131634912_domU12313907193B_2abeddbf13c1a406b0b7ff2.done -rw-r--r-- 1 ec2-user supergroup 483925 2013-01-08 13:26 /chukwa/logs/201308132134913_domU12313907193B_2abeddbf13c1a406b0b7fed.done -rw-r--r-- 1 ec2-user supergroup 629788 2013-01-08 13:37 /chukwa/logs/201308133253308_domU12313907193B_7382697b13c1a5d14db8000.done -rw-r--r-- 1 ec2-user supergroup 485716 2013-01-08 13:42 /chukwa/logs/201308133753394_domU12313907193B_7382697b13c1a5d14db7ffb.done -rw-r--r-- 1 ec2-user supergroup 486475 2013-01-08 13:47 /chukwa/logs/201308134253394_domU12313907193B_7382697b13c1a5d14db7ff5.done .done tells us that On Tue, Jan 8, 2013 at 6:34 PM, Shouvanik Haldar <shouvanik.hal...@gmail.com > wrote: > I am not able to see any thing on hdfs: > when I browse using the url -> > http://ec2-54-234-57-44.compute-1.amazonaws.com:50070/nn_browsedfscontent.jsp > I see nothing..but > > " Network Error (tcp_error) > > A communication error occurred: "Operation timed out" The Web Server > may be down, too busy, or experiencing other problems preventing it from > responding to requests. You may wish to try again at a later time. > For assistance, contact your network support team. " > > > > > On Tue, Jan 8, 2013 at 6:22 PM, Shouvanik Haldar < > shouvanik.hal...@gmail.com> wrote: > >> How I was sure data getting collected at collector, is by seeing the >> below statistics >> >> >> Date:1357649516006 >> Now:1357649518803 >> numberHTTPConnection in time window:2 >> numberchunks in time window:7 >> lifetimechunks:2612 >> >> >> >> >> On Tue, Jan 8, 2013 at 6:20 PM, Shouvanik Haldar < >> shouvanik.hal...@gmail.com> wrote: >> >>> Hi, >>> >>> I was following the post -> http://www.undercloud.org/?p=429. I did the >>> steps as was mentioned here. But, when I was going to check in HDFS whether >>> data has come from chukwa, I cannot see the page. I get the exception as >>> >>> >>> >>> Also, I want to know what is the default location of chukwa logs. I >>> found out a location -> >>> "/home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/logs". Is this the >>> location? Previously, I had some logs inside this directory. I have deleted >>> these logs. Now, no logs are getting created. >>> >>> Previously, when logs were existing, I checked one log entry. It says >>> "nice: /home/ec2-user/chukwa/chukwa-incubating-0.5.0/sbin/bin/chukwa: No >>> such file or directory". >>> >>> >>> I have edited bash_profile to give the path of CHUKWA_HOME. Is this >>> required? >>> >>> I am using "chukwa-incubating-0.5.0" and "hadoop-1.0.4". I can see data >>> getting collected at the collector by browsing the url => >>> http://ec2-54-234-57-44.compute-1.amazonaws.com:8080/chukwa?ping=true >>> >>> >>> >>> >>> >>> But somehow, data not getting inside hdfs. Please help. >>> >>> >>> Regards, >>> Shouvanik >>> >>> >>> On Tue, Jan 8, 2013 at 10:22 AM, Eric Yang <eric...@gmail.com> wrote: >>> >>>> The quick start guide may help: >>>> >>>> http://incubator.apache.org/chukwa/docs/r0.5.0/Quick_Start_Guide.html >>>> >>>> regards, >>>> Eric >>>> >>>> >>>> On Mon, Jan 7, 2013 at 1:17 AM, Shouvanik Haldar < >>>> shouvanik.hal...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> I want to install chukwa on a single node and test it. Where can I >>>>> find a suitable document? >>>>> >>>>> >>>>> On Mon, Jan 7, 2013 at 2:47 AM, Eric Yang <eric...@gmail.com> wrote: >>>>> >>>>>> There are a few things that you need to do: >>>>>> >>>>>> 1. Configure the collector to use HBaseWriter. >>>>>> >>>>>> http://incubator.apache.org/chukwa/docs/r0.5.0/collector.html >>>>>> >>>>>> Make sure that hadoop and hbase configuration are in the classpath of >>>>>> the collector. >>>>>> >>>>>> 2. Write a demux parser for your log file to store on HBase, and this >>>>>> runs in chukwa collector. >>>>>> >>>>>> http://incubator.apache.org/chukwa/docs/r0.5.0/programming.html >>>>>> >>>>>> 3. What log4j adaptors and appenders are you using to write the log >>>>>> files? >>>>>> I would recommend to use socket appender from Log4j and use socket >>>>>> adaptor to listen for the incoming data stream. This will ensure the >>>>>> records are delimited properly. >>>>>> >>>>>> Hope this helps. >>>>>> >>>>>> regards, >>>>>> Eric >>>>>> >>>>>> On Sun, Jan 6, 2013 at 6:00 AM, NehaS Singh < >>>>>> nehas.si...@lntinfotech.com> wrote: >>>>>> >>>>>>> I am using chukwa 0.5. My probleam is want to store the logs from >>>>>>> log4j or other party which is not hadoop.Also i want to write the logs >>>>>>> to >>>>>>> hbase. >>>>>>> But i am not ale to do so. I have configuerd log4j.properties file >>>>>>> so i am able to see th logs in hdfs. >>>>>>> what are the configuration or steps using which i can write the logs >>>>>>> to hbase. >>>>>>> Please help. >>>>>>> ________________________________________ >>>>>>> From: Eric Yang [eric...@gmail.com] >>>>>>> Sent: Saturday, January 05, 2013 11:55 PM >>>>>>> To: chukwa-user@incubator.apache.org >>>>>>> Subject: Re: Writing logs collected by chukwa to hbase >>>>>>> >>>>>>> What version of Chukwa are you running? >>>>>>> >>>>>>> regards, >>>>>>> Eric >>>>>>> >>>>>>> On Fri, Dec 28, 2012 at 2:56 AM, NehaS Singh < >>>>>>> nehas.si...@lntinfotech.com<mailto:nehas.si...@lntinfotech.com>> >>>>>>> wrote: >>>>>>> Hi, >>>>>>> I am using Chukwa to collect log4j logs which I can >>>>>>> see in HDFS. But I am not able to write it to HBASE. Can anyone help ? >>>>>>> >>>>>>> Regards, >>>>>>> Neha Singh >>>>>>> >>>>>>> >>>>>>> ________________________________ >>>>>>> The contents of this e-mail and any attachment(s) may contain >>>>>>> confidential or privileged information for the intended recipient(s). >>>>>>> Unintended recipients are prohibited from taking action on the basis of >>>>>>> information in this e-mail and using or disseminating the information, >>>>>>> and >>>>>>> must notify the sender and delete it from their system. L&T Infotech >>>>>>> will >>>>>>> not accept responsibility or liability for the accuracy or completeness >>>>>>> of, >>>>>>> or the presence of any virus or disabling code in this e-mail" >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> *Shouvanik* >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> *Shouvanik* >>> >> >> >> >> -- >> Thanks, >> *Shouvanik* >> > > > > -- > Thanks, > *Shouvanik* > -- Thanks, *Shouvanik*
