Yeah your absolutely right Eric this is Dev now. I'll sign up to dev right now and participate over there. Thanks Lewis
On Saturday, February 21, 2015, Eric Yang <eric...@gmail.com> wrote: > Hi Lewis, > > Thank you for the patch. This looks interesting. I think this thread has > turned into a development thread. I replied your questions via Chukwa > JIRA, and our discussions will automatically post to dev mailing list for > people who are interested to join the development discussion. Thanks > > regards, > Eric > > On Fri, Feb 20, 2015 at 6:43 AM, Lewis John Mcgibbney < > lewis.mcgibb...@gmail.com > <javascript:_e(%7B%7D,'cvml','lewis.mcgibb...@gmail.com');>> wrote: > >> Hi Eric, >> I submitted an initial patch for CHUKWA-734 with a bunch of issues in my >> comment as well. >> I've also taken time to read through as much documentation as possible >> and have a much better idea of data adaptors which essentially provide data >> to the agent(s). >> The Nagios work is very nice and I'll look at implementing this with my >> monitoring of Nutch logs. >> Thanks >> Lewis >> >> >> On Sun, Feb 15, 2015 at 10:03 AM, Lewis John Mcgibbney < >> lewis.mcgibb...@gmail.com >> <javascript:_e(%7B%7D,'cvml','lewis.mcgibb...@gmail.com');>> wrote: >> >>> This is dynamite I checked out the source from svn and I have undertaken >>> initial analysis. Your hBase package looks like it inherits feom a nice >>> interface face plus your documentation is very helpful. >>> Thanks >>> >>> >>> On Sunday, February 15, 2015, Eric Yang <eric...@gmail.com >>> <javascript:_e(%7B%7D,'cvml','eric...@gmail.com');>> wrote: >>> >>>> Hi Lewis, >>>> >>>> There is an example in Chukwa to write errors out map reduce error to >>>> InError file, and send out Nagios alert. >>>> The example is in: >>>> src/main/java/org/apache/hadoop/chukwa/extraction/demux/DemuxManager.java >>>> >>>> Nagios support is updated in CHUKWA-735 to support better encryption >>>> for Nagios communication. >>>> >>>> Log4j Nagios appender maybe useful in CHUKWA-734. >>>> >>>> regards, >>>> Eric >>>> >>>> On Sat, Feb 14, 2015 at 1:31 PM, Lewis John Mcgibbney < >>>> lewis.mcgibb...@gmail.com> wrote: >>>> >>>>> https://issues.apache.org/jira/browse/CHUKWA-734 >>>>> >>>>> On Sat, Feb 14, 2015 at 12:13 PM, Lewis John Mcgibbney < >>>>> lewis.mcgibb...@gmail.com> wrote: >>>>> >>>>>> Hi Eric, >>>>>> Thank you for the feedback. >>>>>> This is more than helpful. >>>>>> I am going to write a Gora module for Chuckwa. >>>>>> I am going to progress on basis of implementing log monitor for >>>>>> Nutch. >>>>>> Can Chuckwa currently write to file and email response? >>>>>> Thanks >>>>>> Lewis >>>>>> >>>>>> [0] http://gora.apache.org >>>>>> >>>>>> On Sat, Feb 14, 2015 at 9:30 AM, Eric Yang <eric...@gmail.com> wrote: >>>>>> >>>>>>> Hi Lewis, >>>>>>> >>>>>>> Parse error can be captured and store errors to another HDFS >>>>>>> location. In Chukwa 0.4 and earlier, we have demux map reduce job which >>>>>>> does the extraction and store structured data in HDFS, and errors are >>>>>>> channel to another HDFS folder called InError, with the cause of the >>>>>>> parsing error. This is still a batch oriented operation. In Chukwa >>>>>>> 0.6, >>>>>>> we can setup multiple pipeline writer. The pipeline writers can be >>>>>>> configured to provide parsing and channel error to somewhere else, if >>>>>>> data >>>>>>> parse properly, then write it to HBase or HDFS. However, you will need >>>>>>> to >>>>>>> write the pipeline writer class to extend this functionality. We >>>>>>> currently >>>>>>> only have a couple pipeline writers, LocalWriter, HBaseWriter, and >>>>>>> SeqFileWriter. SeqFileWriter needs to be the last one in the pipeline, >>>>>>> if >>>>>>> you choose to write data to HDFS. See this page for how to configure >>>>>>> pipeline writer to achieve partially of what you are looking for: >>>>>>> >>>>>>> http://chukwa.apache.org/docs/r0.6.0/pipeline.html >>>>>>> >>>>>>> Hope this helps. >>>>>>> >>>>>>> regards, >>>>>>> Eric >>>>>>> >>>>>>> On Thu, Feb 12, 2015 at 11:12 PM, Lewis John Mcgibbney < >>>>>>> lewis.mcgibb...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Folks, >>>>>>>> For some time I have been meaning to get in touch to get advice on >>>>>>>> developing a tool for log analysis of Apache Nutch [0] logs. >>>>>>>> What I am referring to particularly is monitoring of logs in a bid >>>>>>>> to identify particular errors which we may anticipate. >>>>>>>> Nutch jobs are batch oriented in architecture which are inherited >>>>>>>> from Hadoop, we typically see errors in the parse phase of a crawl so >>>>>>>> it is >>>>>>>> events like this that I would like to anticipate, monitor and report >>>>>>>> on, >>>>>>>> possibly through email. >>>>>>>> So I am therefore thinking about building a Chuckwa-powered tool >>>>>>>> for Nutch which would become part of our codebase. >>>>>>>> Is Chukwa the right tool for this? Any information about similar >>>>>>>> efforts would be very much appreciated. >>>>>>>> best >>>>>>>> Lewis >>>>>>>> >>>>>>>> [0] http://nutch.apache.org >>>>>>>> >>>>>>>> -- >>>>>>>> *Lewis* >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Lewis* >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Lewis* >>>>> >>>> >>>> >>> >>> -- >>> *Lewis* >>> >>> >> >> >> -- >> *Lewis* >> > > -- *Lewis*
