[
https://issues.apache.org/jira/browse/CHUKWA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744738#action_12744738
]
Eric Yang commented on CHUKWA-369:
----------------------------------
Let's say if the scan frequency is set at every minute because we are set to
close file per minute and demux at the same frequency, and we have 100
collectors, (typical yahoo chukwa cluster) this means the name node needs to
handled 1.4 ls command per second. In my opinion, this design is too costly.
The robustness of HDFS writer should be handled in the hadoop hdfs client or
collector. Name node should not bare this cost.
> proposed reliability mechanism
> ------------------------------
>
> Key: CHUKWA-369
> URL: https://issues.apache.org/jira/browse/CHUKWA-369
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: delayedAcks.patch
>
>
> We like to say that Chukwa is a system for reliable log collection. It isn't,
> quite, since we don't handle collector crashes. Here's a proposed
> reliability mechanism.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
