shreyas subramanya created CHUKWA-678:
-----------------------------------------
Summary: Make use of ChukwaWriter in agent
Key: CHUKWA-678
URL: https://issues.apache.org/jira/browse/CHUKWA-678
Project: Chukwa
Issue Type: Task
Components: Data Collection
Environment: MacOSX, Java 6
Reporter: shreyas subramanya
The chukwa agent sends out data chunks to various destinations through the
combination of Connector and ChukwaSender interfaces. For sending chunks to
collector, we have http implementation of these interfaces. The collector
writes out the received chunks to various destinations through classes
implementing ChukwaWriter interface. Optionally, multiple destinations can be
chosen by specifying PipelineStageWriter.
The proposal is to:
1. Use ChukwaWriter to send out data chunks to multiple destinations from the
agent. Further, PipelinestageWriter can be made default and pipeline
configuration specified in the agent config file
2. Implement (or modify) Pipelineable writers for HBase, Http, Hdfs and WebHdfs
3. Do away with the Connector interface and have a single non configurable
connector object as part of the agent. This class initiates the configured
writer, waits for data chunks and passes the chunks to Writer.add()/send().
Connection protocol for each destination is handled by the init() of the
individual writers.
Considerations:
1. In case of Pipelineable writers, we need a way to merge the results of each
pipeline stage before the agent commits the chunk.
2. Handling pipeline failure
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
