Yeah, that would be a great tool to have out there if you guys ever get around to it...
-Jay On Tue, May 15, 2012 at 1:23 PM, Evan Chan <e...@ooyala.com> wrote: > Yeah, one of these days we hope to open source it. We have to first > extract a bunch of internal dependencies out, and have cycles to do it... > > Basically, the idea is to monitor the file/directories where the file > rotation happens (using, say, nio2 or inotify/jnotify) and periodically > persist state for recovery. > > -Evan > > On Tue, May 15, 2012 at 1:18 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> Any interest in sharing what you did? Either as an example or as a >> general tool... >> >> -Jay >> >> On Tue, May 15, 2012 at 10:07 AM, Evan Chan <e...@ooyala.com> wrote: >> > We built a special purpose daemon for tailing files like syslog and >> pumping >> > them into Kafka. The reason is that reliable tailing is not trivial. >> You >> > have to handle the rotation of syslog and other files, and be able to >> pick >> > up where you left off in case of restarts, even if the file moved -- if >> you >> > want guaranteed delivery that is. >> > >> > Flume might be able to do something like this, but there are lots of >> > threads about how unreliable the tailing there is, and it is kind of >> > heavyweight, so we decided to do it ourselves. >> > >> > On Tue, May 15, 2012 at 8:22 AM, Jay Kreps <jay.kr...@gmail.com> wrote: >> > >> >> Hi Navneet, >> >> >> >> I don't have such an example, it would be great if somebody tried >> >> this. I don't know whether or not this would be a good solution for >> >> syslog, or whether it would be good to have a special-purpose daemon. >> >> I would be happy to modify that tool if there are changes that would >> >> be needed. >> >> >> >> To get files instead of the console you would presumably pipe them in. >> >> >> >> -Jay >> >> >> >> On Tue, May 15, 2012 at 6:07 AM, navneet sharma >> >> <navneetsharma0...@gmail.com> wrote: >> >> > Jay, >> >> > >> >> > is there any example or documentation to understand how to use it. If >> i >> >> > type something on console, it will be captured by producer, but how to >> >> send >> >> > messages from a syslog to this console-producer? >> >> > >> >> > Thanks, >> >> > Navneet Sharma >> >> > >> >> > On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <jay.kr...@gmail.com> >> wrote: >> >> > >> >> >> Yes, in the bin/ directory you will see kafka-console-producer.sh >> which >> >> >> runs it. >> >> >> >> >> >> -Jay >> >> >> >> >> >> On Thu, May 10, 2012 at 11:34 PM, navneet sharma >> >> >> <navneetsharma0...@gmail.com> wrote: >> >> >> > Hi, >> >> >> > >> >> >> > I was going through archive. And came across this jira ticket: >> >> >> > https://issues.apache.org/jira/browse/KAFKA-130 >> >> >> > >> >> >> > My question, is the tailing utility available now with 0.7.0 >> version? >> >> >> > >> >> >> > >> >> >> > Thanks, >> >> >> > Navneet Sharma >> >> >> >> >> >> > >> > >> > >> > -- >> > -- >> > *Evan Chan* >> > Senior Software Engineer | >> > e...@ooyala.com | (650) 996-4600 >> > www.ooyala.com | blog <http://www.ooyala.com/blog> | >> > @ooyala<http://www.twitter.com/ooyala> >> > > > > -- > -- > *Evan Chan* > Senior Software Engineer | > e...@ooyala.com | (650) 996-4600 > www.ooyala.com | blog <http://www.ooyala.com/blog> | > @ooyala<http://www.twitter.com/ooyala>
