Re: Flume Appender

[Ralph Goers]

As Christian pointed out, Log4j 2 is still considered beta quality software. While it works fine in the environments I have tested in others are starting to use it in other environments and as they encounter problems and fixes are implemented the quality will improve. That said, I am using the Flume Appender and we recently did some performance testing.  The chart below shows the result of a single application writing the specified number of events.  The application and remote agents are running in VMs, which is not going to yield the best performance Flume has to offer. However, our application doesn't really reach a very large scale until we start aggregating events from a number of applications into Flume Agents acting as collectors. We don't expect a single application to be writing more than about 10 events per second so the numbers below are just fine even in the non-embedded case.  The agent collectors can be run on bare metal and achieve a fairly high level of performance.  In addition, you can run a cluster of agents in your collector pool and point the applications to different agents to scale up to the numbers you are talking about. Of course, your ultimate destination also has to support that kind of rate. I am not familiar with Elasticsearch.  For more discussion on how to get Flume to scale I would suggest asking on the Flume mailing lists. As you can see, the embedded agent is significantly faster than the remote agent. The downside though, is that when you want to update Flume your application has to be modified. I had a discussion with the Flume developers just last week about the embedded appender. The current release of Flume doesn't support embedded out of the box so I had to do a bit of work.  In one of the next releases they plan on improving the support. That really should impact the visible portion of the Appender but should make it easier to implement and maintain. HTH Ralph On Oct 25, 2012, at 6:34 AM, SimonMo wrote: Hi, does anybody know how stable is the flume appender embedded? I have a network with nodes logging 20.000 logs/sec and i want to use Log4j 2 -> Flume -> Elasticsearch. Is this a good approach? I could use additionally a FailoverAppender to get my logs on a second route to my server... hmm, i'm not sure. Has anybody experience with so much logs in an network? Regards, Simon -- View this message in context: http://old.nabble.com/Flume-Appender-tp34601088p34601088.html Sent from the Log4j - Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org For additional commands, e-mail: log4j-user-h...@logging.apache.org