Hey Chris, Mark has got it - for the behavior you are looking, you probably want a tier of flume agents running where you currently have avro clients, using an exec source with "tail -F" as the command.
Keep in minding the red warning box in the user guide related to sources like this. If a flume agent restarts, it may re-send or miss certain log entries. - Patrick On Mon, Aug 13, 2012 at 1:03 PM, Stern, Mark <[email protected]> wrote: > Use an exec source running 'tail -F'. > ________________________________________ > From: Chris Neal [[email protected]] > Sent: Monday, August 13, 2012 8:43 PM > To: [email protected] > Subject: Avro client in NG as replacement for tail source in OG? > > Hi all. > > I have a very typical configuration: > > Application logs to log4J file. > FlumeNG avro-client watches the file and sends events into FlumeNG Agent Tier > 1 > FlumeNG Agent Tier 1: AvroSource to FileChannel to AvroSink > FlumeNG Agent Tier 2: AvroSource to FileChannel to HDFSSink > > I was noticing that after some time, the Tier 1 Agent would disconnect from > the avro-client. What is happening is that the avro-client sends events to > the Tier 1 Agent as fast as it can, and when it reaches the end of the file, > it exits. The problem is, the application is still logging to the log4J > file, but now all future events are lost because the avro-client has exited. > > I thought the "-F" option to avro-client was like the "-F" option to tail, > but after looking at the code, it is not. There seems to be no "follow" mode > for the avro client that I can see. I then stumbled across this key sentence > from > here<https://cwiki.apache.org/confluence/display/FLUME/Getting+Started#GettingStarted-flumengavroclientoptions>: > > Think of the avro-client command as cat for Flume > > So, it's a "cat", not a "tail". > > So I'm wondering, what's the right/best/current way to emulate OG's > tailSource? > > Much appreciated. > Chris
