[
https://issues.apache.org/jira/browse/FLUME-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neerja Khattar updated FLUME-2669:
----------------------------------
Summary: Flume Appender for log4j 1.x - Support for event batching (was:
ability to batch a number of logging events together in order to send them over
the wire in one go in log4j1.x)
> Flume Appender for log4j 1.x - Support for event batching
> ---------------------------------------------------------
>
> Key: FLUME-2669
> URL: https://issues.apache.org/jira/browse/FLUME-2669
> Project: Flume
> Issue Type: New Feature
> Reporter: Neerja Khattar
>
> A better support for Flume in the log4j 1.x appender may be beneficial.
> There is support for load-balancing with back-off and transparent failover
> in it already. What it needs is the ability to batch a number of logging
> events together in order to send them over the wire in one go.
> With a blocking queue in Java 5, it will be "fairly easy" to implement the
> following logic:
> 1) the appender is initialized with a blocking queue on
> AppenderSkeleton.activateOptions() and will be the producer for it - on every
> call to AppenderSkeleton.append() it will add a logging event to the queue
> then return.
> 2) a consumer thread is also brought up on AppenderSkeleton.activateOptions()
> and the same queue is passed to it. The thread will continuously peek an
> object from the queue and store it in a buffer. If no objects on the queue,
> the thread will block. Once the buffer is full, it will call
> RpcClient.appendBatch() then free up the buffer.
> 3) in case an error occurs while sending the batch to a remote Flume agent
> (and the RPC client fails to deliver it having tried all the agents), either
> an exception is thrown (safe mode) or the events are silently dropped (unsafe
> mode).
> 4) size of the in-memory buffer / size of the batch as well as size of the
> blocking queue should be made configurable for greater flexibility
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
