Flume is one of the options to do this, yes. There are a (growing) number of
tools you could use to get logs in to Kafka, depending on your environment,
needs, level of skill, deployment size, applications, log volume:
(in no particular order)
Kafkacat : https://github.com/edenhill/kafkacat
Logstash: https://www.elastic.co/products/logstash
Klogger (from my team): https://github.com/blackberry/Klogger
Rsyslog:
http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html
Log4j:
http://stackoverflow.com/questions/22034895/how-to-use-kafka-0-8-log4j-appender
Depending on your app, you may need to read the logs in from the filesystem to
send to kafka. Better though, would be not bothering with the filesystem
read/write, and using something that will listen to syslog/use log4j, and write
the log data directly to Kafka. If you have control over the application code,
you could also look at tacking in a producer library (kafkas provided ones, or
others (https://github.com/blackberry/Krackle)) and writing directly to kafka
from the app - no need for log4j, syslog, filesystem logs.
If you're starting off, testing is nice and simple using logstash or klogger -
you can take existing logs and just grab them from where they are, and put them
in to kafka without changing any of your existing workflow/code/configuration.
There are lots of solutions, you need to experiment to find the one that fits
your environment best.
Cheers,
T.
-----Original Message-----
From: Snehalata Nagaje [mailto:[email protected]]
Sent: Friday, July 31, 2015 3:32
To: [email protected]
Subject: How to use kafka for log aggregation
Hi All,
I want to use kafka for log aggregation, how can we use it?
Do we need to manually read log files and post this logs into kafka using
producer API?
Can Apache Flume help in it?
Thanks,
Snehalata
