We built a special purpose daemon for tailing files like syslog and pumping them into Kafka. The reason is that reliable tailing is not trivial. You have to handle the rotation of syslog and other files, and be able to pick up where you left off in case of restarts, even if the file moved -- if you want guaranteed delivery that is.
Flume might be able to do something like this, but there are lots of threads about how unreliable the tailing there is, and it is kind of heavyweight, so we decided to do it ourselves. On Tue, May 15, 2012 at 8:22 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > Hi Navneet, > > I don't have such an example, it would be great if somebody tried > this. I don't know whether or not this would be a good solution for > syslog, or whether it would be good to have a special-purpose daemon. > I would be happy to modify that tool if there are changes that would > be needed. > > To get files instead of the console you would presumably pipe them in. > > -Jay > > On Tue, May 15, 2012 at 6:07 AM, navneet sharma > <navneetsharma0...@gmail.com> wrote: > > Jay, > > > > is there any example or documentation to understand how to use it. If i > > type something on console, it will be captured by producer, but how to > send > > messages from a syslog to this console-producer? > > > > Thanks, > > Navneet Sharma > > > > On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > >> Yes, in the bin/ directory you will see kafka-console-producer.sh which > >> runs it. > >> > >> -Jay > >> > >> On Thu, May 10, 2012 at 11:34 PM, navneet sharma > >> <navneetsharma0...@gmail.com> wrote: > >> > Hi, > >> > > >> > I was going through archive. And came across this jira ticket: > >> > https://issues.apache.org/jira/browse/KAFKA-130 > >> > > >> > My question, is the tailing utility available now with 0.7.0 version? > >> > > >> > > >> > Thanks, > >> > Navneet Sharma > >> > -- -- *Evan Chan* Senior Software Engineer | e...@ooyala.com | (650) 996-4600 www.ooyala.com | blog <http://www.ooyala.com/blog> | @ooyala<http://www.twitter.com/ooyala>
