Yeah, one of these days we hope to open source it. We have to first extract a bunch of internal dependencies out, and have cycles to do it...
Basically, the idea is to monitor the file/directories where the file rotation happens (using, say, nio2 or inotify/jnotify) and periodically persist state for recovery. -Evan On Tue, May 15, 2012 at 1:18 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > Any interest in sharing what you did? Either as an example or as a > general tool... > > -Jay > > On Tue, May 15, 2012 at 10:07 AM, Evan Chan <e...@ooyala.com> wrote: > > We built a special purpose daemon for tailing files like syslog and > pumping > > them into Kafka. The reason is that reliable tailing is not trivial. > You > > have to handle the rotation of syslog and other files, and be able to > pick > > up where you left off in case of restarts, even if the file moved -- if > you > > want guaranteed delivery that is. > > > > Flume might be able to do something like this, but there are lots of > > threads about how unreliable the tailing there is, and it is kind of > > heavyweight, so we decided to do it ourselves. > > > > On Tue, May 15, 2012 at 8:22 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > >> Hi Navneet, > >> > >> I don't have such an example, it would be great if somebody tried > >> this. I don't know whether or not this would be a good solution for > >> syslog, or whether it would be good to have a special-purpose daemon. > >> I would be happy to modify that tool if there are changes that would > >> be needed. > >> > >> To get files instead of the console you would presumably pipe them in. > >> > >> -Jay > >> > >> On Tue, May 15, 2012 at 6:07 AM, navneet sharma > >> <navneetsharma0...@gmail.com> wrote: > >> > Jay, > >> > > >> > is there any example or documentation to understand how to use it. If > i > >> > type something on console, it will be captured by producer, but how to > >> send > >> > messages from a syslog to this console-producer? > >> > > >> > Thanks, > >> > Navneet Sharma > >> > > >> > On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <jay.kr...@gmail.com> > wrote: > >> > > >> >> Yes, in the bin/ directory you will see kafka-console-producer.sh > which > >> >> runs it. > >> >> > >> >> -Jay > >> >> > >> >> On Thu, May 10, 2012 at 11:34 PM, navneet sharma > >> >> <navneetsharma0...@gmail.com> wrote: > >> >> > Hi, > >> >> > > >> >> > I was going through archive. And came across this jira ticket: > >> >> > https://issues.apache.org/jira/browse/KAFKA-130 > >> >> > > >> >> > My question, is the tailing utility available now with 0.7.0 > version? > >> >> > > >> >> > > >> >> > Thanks, > >> >> > Navneet Sharma > >> >> > >> > > > > > > > > -- > > -- > > *Evan Chan* > > Senior Software Engineer | > > e...@ooyala.com | (650) 996-4600 > > www.ooyala.com | blog <http://www.ooyala.com/blog> | > > @ooyala<http://www.twitter.com/ooyala> > -- -- *Evan Chan* Senior Software Engineer | e...@ooyala.com | (650) 996-4600 www.ooyala.com | blog <http://www.ooyala.com/blog> | @ooyala<http://www.twitter.com/ooyala>
