[
https://issues.apache.org/jira/browse/CHUKWA-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743808#action_12743808
]
Ari Rabkin commented on CHUKWA-369:
-----------------------------------
Patch has basically five pieces; I'm happy to split them up and commit
separately if some are uncontroversial.
1) Sender and Connector are refactored to allow the HttpClient to be reused,
and used more generally.
2) Writers now return an instance of ChukwaWriter.CommitStatus. This is either
OK, Failure, or Pending. The first two are singletons, the latter includes a
list of strings.
3) SeqFileWriter returns a CommitPending on writes.
4) A new servlet, CommitCheckServlet for periodically scanning HDFS.
5) A new Sender, the AsyncAckSender, that doesn't automatically commit, but
only does so when it either receives an OK, or else after a pending commit
becomes stable. The Sender periodically asks a CommitCheckServlet what's been
committed.
I think (1), and possibly (2+3) may make sense even without 4 and 5, which are
the bits that I think need serious testing before we should even discuss
committing them.
> proposed reliability mechanism
> ------------------------------
>
> Key: CHUKWA-369
> URL: https://issues.apache.org/jira/browse/CHUKWA-369
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Affects Versions: 0.3.0
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: delayedAcks.patch
>
>
> We like to say that Chukwa is a system for reliable log collection. It isn't,
> quite, since we don't handle collector crashes. Here's a proposed
> reliability mechanism.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
