Any interest in sharing what you did? Either as an example or as a general tool...
-Jay On Tue, May 15, 2012 at 10:07 AM, Evan Chan <e...@ooyala.com> wrote: > We built a special purpose daemon for tailing files like syslog and pumping > them into Kafka. The reason is that reliable tailing is not trivial. You > have to handle the rotation of syslog and other files, and be able to pick > up where you left off in case of restarts, even if the file moved -- if you > want guaranteed delivery that is. > > Flume might be able to do something like this, but there are lots of > threads about how unreliable the tailing there is, and it is kind of > heavyweight, so we decided to do it ourselves. > > On Tue, May 15, 2012 at 8:22 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> Hi Navneet, >> >> I don't have such an example, it would be great if somebody tried >> this. I don't know whether or not this would be a good solution for >> syslog, or whether it would be good to have a special-purpose daemon. >> I would be happy to modify that tool if there are changes that would >> be needed. >> >> To get files instead of the console you would presumably pipe them in. >> >> -Jay >> >> On Tue, May 15, 2012 at 6:07 AM, navneet sharma >> <navneetsharma0...@gmail.com> wrote: >> > Jay, >> > >> > is there any example or documentation to understand how to use it. If i >> > type something on console, it will be captured by producer, but how to >> send >> > messages from a syslog to this console-producer? >> > >> > Thanks, >> > Navneet Sharma >> > >> > On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <jay.kr...@gmail.com> wrote: >> > >> >> Yes, in the bin/ directory you will see kafka-console-producer.sh which >> >> runs it. >> >> >> >> -Jay >> >> >> >> On Thu, May 10, 2012 at 11:34 PM, navneet sharma >> >> <navneetsharma0...@gmail.com> wrote: >> >> > Hi, >> >> > >> >> > I was going through archive. And came across this jira ticket: >> >> > https://issues.apache.org/jira/browse/KAFKA-130 >> >> > >> >> > My question, is the tailing utility available now with 0.7.0 version? >> >> > >> >> > >> >> > Thanks, >> >> > Navneet Sharma >> >> >> > > > > -- > -- > *Evan Chan* > Senior Software Engineer | > e...@ooyala.com | (650) 996-4600 > www.ooyala.com | blog <http://www.ooyala.com/blog> | > @ooyala<http://www.twitter.com/ooyala>
