Hi Eric
Thanks for that. I will look at integrating Log4J appender for flume for sure.
Couple of additional questions.
1. From a performance standpoint, does Log4J appender have any significant
advantages over tailing the log file?2. It would be ideal if the Log4J appender
also allows to put in some meta data that I need to use for output bucketing.
Any ideas how that can be achieved?
RegardsSrini
Date: Wed, 30 Nov 2011 10:28:41 -0800
Subject: Re: Log4J appender
From: esam...@cloudera.com
To: flume-user@incubator.apache.org
Srini:
On Wed, Nov 30, 2011 at 12:23 AM, Srinivasan Subramanian
<ssrini_va...@hotmail.com> wrote:
I was evaluating the log4j appender provided with Flume. But there is one
aspect I dont understand:
The log4j appender makes a connection to teh flume-agent and retries a maximum
of 10 times (default - configurable) if the connection is not made
successfully.
Questions:
1. When will the connection fail? If the agent is not running on the node? In
that case given that the default implementation waits for 1 second before each
retry for a total of 10 retries, would this mean that each logging call from
the application would be delayed by 10 seconds? That would affect performance
right?
Almost certainly, yes, assuming log4j is synchronous (I'm 99.9% sure it is). Of
course, synchronous logging is the only way to guarantee event delivery in this
context; if the application were to log the event and move on without waiting
for a response an event could get dropped and no one would be responsible for
retrying the send.
2. What happens to the log message when the agent is not available? Is it lost?
If the log4j appender runs out of retries I believe I wrote it to throw an
exception. This would be the equivalent of using a standard file appender and
running out of disk space. In other words, the log call failed and should be
handled by the application.
Let me know if you have any other questions!
I am a little confused with the implementation and any help in explaining this
is appreciated.
RegardsSrini
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
