The easiest method to get going is to use trunk version of Chukwa, and setup
your log4j configuration like:
log4j.appender.DRFA=org.apache.log4j.net.SocketAppender
log4j.appender.DRFA.RemoteHost=localhost
log4j.appender.DRFA.Port=9000
log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
This will setup log4j to stream data to localhost:9096
Then setup init_adaptors with:
add SocketAdaptor DataType 9000 0
This will start streaming data to HDFS. You will need to build your own
indexing of your log data on HDFS to provide search function. Hadoop has a
tutorial for running "grep", it may be a starting point to build a search
service. Hope this helps.
regards,
Eric
On Jul 18, 2011, at 10:26 PM, Ariel Rabkin wrote:
> Yes, this is a reasonable use for Chukwa. You'll have an adaptor per
> log file. Depending on your particular needs, you can either have a
> custom log4j appender to set up the Chukwa side or else explicitly
> configure adaptors on the log files you care about.
>
> You will have to build the search part yourself. Chukwa designed to
> get stuff into HDFS; the rest is up to you. Of course, if you build
> something generally usable and contribute it to the project, I'm sure
> you'd get help with development and maintenance.
>
> --Ari
>
> On Mon, Jul 18, 2011 at 10:21 PM, sangeetha <[email protected]> wrote:
>> Hi,
>>
>> I wanted to aggregate log4j logs files from different systems and perform
>> search on the logs.
>> Is it possible in chukwa? How can I use chukwa for this?
>>
>> Let me know the configuration details for doing the same.
>>
>> Thanks,
>> Sangeetha
>>
>> --
>> View this message in context:
>> http://apache-chukwa.679492.n3.nabble.com/chukwa-for-processing-log4j-logs-tp3181483p3181483.html
>> Sent from the Chukwa - Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Ari Rabkin [email protected]
> UC Berkeley Computer Science Department
