Juhani, Thanks for the response :) I'll take a look at the jira. I like your solution. I might end up going a route similar.
Chris On Wed, Aug 29, 2012 at 8:51 PM, Juhani Connolly < [email protected]> wrote: > Hi Chris, > > A few months back I actually ported the original flumes tail source, but > it was decided(and I agree with the reasoning) not to include it for a > number of reasons, which can be seen on the original ticket at > https://issues.apache.org/jira/browse/FLUME-931 . One of the big ones is > the fact that java cannot access inode information. > > What we do is have a python program that tracks the files in a directory > and then sends the data using the scribe format to the ScribeSource(we were > using scribe until switching to flume, so are just using our ingest system > from then). This allows for the freedom to customize the ingest to our own > expectations, and we write checkpoints of how far we have tailed. You could > write this in whatever language you're comfortable with and pass the data > via avro or thrift. > > > On 08/30/2012 01:18 AM, Chris Neal wrote: > > Hey guys, > > I'm sure this is not a new question, but I haven't found an answer in my > searches. I'm curious why there is as of yet no Tail Source with NG? It > seems one of the most common use cases for Flume is to tail a log file and > dump it "somewhere". Given that, it sure would seem that a Tail Source > would be one of the first sources that gets written with a new version. > > I know about all the other ways to implement something *like* a Tail > Source: Exec Source, AVRO, Log4Jappender... and unfortunately they all > have limitations with regards to either functionality or > reliability/recoverability. > > What am I missing here? > > Is there any work being done on a Tail Source for NG? > > I promise I'm not complaining, just trying to understand the logic. :) > > Much appreciated. > Chris > > >
