Ah yes. Good point. I'll look at that. Also, I noticed that the Log4JAppender doesn't do anything with a Pattern Layout if it's supplied, so I added that capability. I'll work on getting that submitted back.
Chris On Tue, Aug 14, 2012 at 3:15 PM, Patrick Wendell <[email protected]> wrote: > Just a note on reliability with this approach, > > I'm not sure how the log4j appender handles the case where it can't > connect to the downstream avro source (i.e. if there is a failure at > the flume agent tier). I just looked through the code and it looks > like it throws a FlumeException (a RuntimeException) if an append > fails. I'm not sure if log4j eats RunitmeExceptions in appenders or > not. If it does, this risks data loss, and if it doesn't you'll need > to handle this possibility of an exception in your own code. > > - Patrick > > On Tue, Aug 14, 2012 at 8:36 AM, Chris Neal <[email protected]> wrote: > > Ahh...and the best solution yet...Use > > > .//flume-ng-clients/flume-ng-log4jappender/target/flume-ng-log4jappender-1.3.0-SNAPSHOT.jar > > and use the already built appender for log4j. :) I knew there was a > better > > way! > > > > chris > > > > > > On Tue, Aug 14, 2012 at 10:00 AM, Chris Neal <[email protected]> wrote: > >> > >> I could get around the tail -F problem of Solaris with gtail, but that > >> doesn't address the reliability problem. > >> > >> I also found this: > >> > http://logging.apache.org/log4j/2.x/manual/appenders.html#FlumeAvroAppender > >> which has potential, but it's in an alpha release of log4J, which won't > fly > >> in my production environment. > >> > >> Can anyone comment on the priority/timeframe of a tail source in NG? :) > >> > >> Thanks! > >> > >> > >> On Tue, Aug 14, 2012 at 9:19 AM, Chris Neal <[email protected]> wrote: > >>> > >>> Thanks Mark and Patrick, > >>> > >>> I had seen the ExecSource, and it's associated caveats :) That has two > >>> problems for me: > >>> > >>> 1) It's aforementioned unreliability should the channel have errors, > and > >>> 2) All my applications run on Solaris, whose tail command does not > have > >>> a -F equivalent from the Linux world. :( > >>> > >>> I noticed that the wiki here does mention a feature for NG for > supporting > >>> a tail source, but no JIRA is assigned to it yet. If that page is > still up > >>> to date, I'm glad to see that it's at least on the roadmap! > >>> > >>> If anyone can think of any other creative ways to do this, please > share. > >>> > >>> Much appreciated! > >>> Chris > >>> > >>> > >>> On Mon, Aug 13, 2012 at 6:28 PM, Patrick Wendell <[email protected]> > >>> wrote: > >>>> > >>>> Hey Chris, > >>>> > >>>> Mark has got it - for the behavior you are looking, you probably want > >>>> a tier of flume agents running where you currently have avro clients, > >>>> using an exec source with "tail -F" as the command. > >>>> > >>>> Keep in minding the red warning box in the user guide related to > >>>> sources like this. If a flume agent restarts, it may re-send or miss > >>>> certain log entries. > >>>> > >>>> - Patrick > >>>> > >>>> On Mon, Aug 13, 2012 at 1:03 PM, Stern, Mark <[email protected]> > wrote: > >>>> > Use an exec source running 'tail -F'. > >>>> > ________________________________________ > >>>> > From: Chris Neal [[email protected]] > >>>> > Sent: Monday, August 13, 2012 8:43 PM > >>>> > To: [email protected] > >>>> > Subject: Avro client in NG as replacement for tail source in OG? > >>>> > > >>>> > Hi all. > >>>> > > >>>> > I have a very typical configuration: > >>>> > > >>>> > Application logs to log4J file. > >>>> > FlumeNG avro-client watches the file and sends events into FlumeNG > >>>> > Agent Tier 1 > >>>> > FlumeNG Agent Tier 1: AvroSource to FileChannel to AvroSink > >>>> > FlumeNG Agent Tier 2: AvroSource to FileChannel to HDFSSink > >>>> > > >>>> > I was noticing that after some time, the Tier 1 Agent would > disconnect > >>>> > from the avro-client. What is happening is that the avro-client > sends > >>>> > events to the Tier 1 Agent as fast as it can, and when it reaches > the end of > >>>> > the file, it exits. The problem is, the application is still > logging to the > >>>> > log4J file, but now all future events are lost because the > avro-client has > >>>> > exited. > >>>> > > >>>> > I thought the "-F" option to avro-client was like the "-F" option to > >>>> > tail, but after looking at the code, it is not. There seems to be > no > >>>> > "follow" mode for the avro client that I can see. I then stumbled > across > >>>> > this key sentence from > >>>> > here< > https://cwiki.apache.org/confluence/display/FLUME/Getting+Started#GettingStarted-flumengavroclientoptions > >: > >>>> > > >>>> > Think of the avro-client command as cat for Flume > >>>> > > >>>> > So, it's a "cat", not a "tail". > >>>> > > >>>> > So I'm wondering, what's the right/best/current way to emulate OG's > >>>> > tailSource? > >>>> > > >>>> > Much appreciated. > >>>> > Chris > >>> > >>> > >> > > >
