Take a look at the real-time aggregation engine that we offer at Stratio. Fully open - source, no coding, kafka integration out of the box, based on spark...
http://docs.stratio.com/modules/sparkta/development/ Regards El 31/07/2015 16:57, "Todd Snyder" <[email protected]> escribió: > Flume is one of the options to do this, yes. There are a (growing) number > of tools you could use to get logs in to Kafka, depending on your > environment, needs, level of skill, deployment size, applications, log > volume: > (in no particular order) > > Kafkacat : https://github.com/edenhill/kafkacat > Logstash: https://www.elastic.co/products/logstash > Klogger (from my team): https://github.com/blackberry/Klogger > Rsyslog: > http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html > Log4j: > http://stackoverflow.com/questions/22034895/how-to-use-kafka-0-8-log4j-appender > > Depending on your app, you may need to read the logs in from the > filesystem to send to kafka. Better though, would be not bothering with > the filesystem read/write, and using something that will listen to > syslog/use log4j, and write the log data directly to Kafka. If you have > control over the application code, you could also look at tacking in a > producer library (kafkas provided ones, or others ( > https://github.com/blackberry/Krackle)) and writing directly to kafka > from the app - no need for log4j, syslog, filesystem logs. > > If you're starting off, testing is nice and simple using logstash or > klogger - you can take existing logs and just grab them from where they > are, and put them in to kafka without changing any of your existing > workflow/code/configuration. > > There are lots of solutions, you need to experiment to find the one that > fits your environment best. > > Cheers, > > T. > > -----Original Message----- > From: Snehalata Nagaje [mailto:[email protected]] > Sent: Friday, July 31, 2015 3:32 > To: [email protected] > Subject: How to use kafka for log aggregation > > > > Hi All, > > I want to use kafka for log aggregation, how can we use it? > > Do we need to manually read log files and post this logs into kafka using > producer API? > > Can Apache Flume help in it? > > Thanks, > Snehalata >
