[
https://issues.apache.org/jira/browse/CHUKWA-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739128#action_12739128
]
Jerome Boulon commented on CHUKWA-102:
--------------------------------------
The initial idea when Mac creates this Jira was to stream data from an
application to the local agent but my proposal is to send directly to the
collector and scale horizontally if possible then if you can no longer scale
horizontally (too many files on HDFS for example) then scale by creating a DAG
of collector (similar to what scribe is doing).
So in my proposal there's no need for an Agent, the agent could be a simple as
a cronjob that read a file and stream it over to a remote collector.
I may put some logic inside the log4j Appender to fall back to disk in case of
network issue or to have the same kind of checkpoint directly at the log4j
appender. But I'm still working on the design.
> Provide streaming adapter
> -------------------------
>
> Key: CHUKWA-102
> URL: https://issues.apache.org/jira/browse/CHUKWA-102
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: input tools
> Reporter: Mac Yang
>
> Enable applications to send data directly to the agent without writing to
> disk first.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
