[
https://issues.apache.org/jira/browse/CHUKWA-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040299#comment-14040299
]
Eric Yang edited comment on CHUKWA-674 at 6/22/14 11:24 PM:
------------------------------------------------------------
Redesign the system to work like this:
Adaptor -> ChukwaAgent -> Connector (Configurable)
The proposed implementation support this:
Adaptor -> ChukwaAgent -> PipelineConnector -> HBaseWriter
There is a new property called chukwa.pipeline in Agent configuration which
defines a list of writer to send data. User can also mirror data to both HDFS
and HBase by define both HBaseWriter and SeqFileWriter.
was (Author: eyang):
Redesign the system to work like this:
Adaptor -> ChukwaAgent -> Connector (Configurable)
The proposed implementation support this:
Adaptor -> ChukwaAgent -> PipelineConnector -> HBaseWriter
There is a new property called chukwa.pipeline in Agent configuration which
defines a list of writer to send data. User can also mirror data to both HDFS
and HBase by define both HBaseWriter or SeqFileWriter.
> Integrate Chukwa collector feature to Chukwa agent
> --------------------------------------------------
>
> Key: CHUKWA-674
> URL: https://issues.apache.org/jira/browse/CHUKWA-674
> Project: Chukwa
> Issue Type: Improvement
> Components: Data Collection
> Environment: MacOSX, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
>
> Feature offered in Chukwa collector can be integrated into Chukwa agent, and
> use multi-tier Chukwa agent to collect data for large scale cluster. For
> small cluster, agents can talk directly to HDFS cluster to reduce the
> complexity of deployment. The required features to reduce the need of Chukwa
> collectors are:
> - Enhance agent rest api to receive chunk data.
> - Pipeline writer to channel data to storage destinations (HDFS, HBASE).
> - Improve connector interface and replace http connector with collector
> connector for bandwidth balance.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
