[
https://issues.apache.org/jira/browse/CHUKWA-389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756174#action_12756174
]
Jerome Boulon commented on CHUKWA-389:
--------------------------------------
My proposal is to use Thrift for the communication layer between Log4J and the
collector.
I want to use Thrift for 2 reasons, it's available and well debugged with good
performance and using Thrift will open chukwa to a large number of programming
language.
The new Log4j Appender will store all logs to an in-memory buffering queue. One
or more sender will read from the in-memory queue and send the data to the
collector using Thrift for data serialization and transport. A sender will
collect up to N messages at regular interval. If the queue's size reach a
threshold then the sender will be wakeup and the data will be send
automatically.
A sender will stay with the same collector unless the connection is dropped. In
that case the sender will reconnect to another collector and send the same data.
Data from the same log4j appender could be sent to different collectors for
load-balancing and fail-over.
The new ThriftCollector class will act at the same level as the current
JettyCollector.
The writer interface and all current writers would be reusable without any
modification.
> Send chunks directly from a log4j appender to the collector without writing
> to local drive
> ------------------------------------------------------------------------------------------
>
> Key: CHUKWA-389
> URL: https://issues.apache.org/jira/browse/CHUKWA-389
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
>
> Currently Chukwa required to first write the data to the local drive and run
> an agent on every single box in order to be able to collect logs.
> This is a good solution if you cannot afford to loose any data but sometime
> you don't want to run an agent on very single box.
> In that case, a new log4J appender could be used to send data directly from
> the application to a collector.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
