[
https://issues.apache.org/jira/browse/STORM-39?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rick Kellogg updated STORM-39:
------------------------------
Component/s: storm-core
> Expose number of attempts for a batch within Trident API
> --------------------------------------------------------
>
> Key: STORM-39
> URL: https://issues.apache.org/jira/browse/STORM-39
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: James Xu
>
> https://github.com/nathanmarz/storm/issues/352
> This would let someone implement policies like "retry a batch up to 4 times,
> then skip it".
> -----------------------------------------------------------------------------------------------------
> nathanmarz:
> I think the best strategy is to just expose this in the coordinator. isReady
> can be changed to take in a TransactionAttempt and return an enum:
> enum BatchAction {
> PROCESS,
> WAIT,
> SKIP
> }
> Perhaps isReady can be renamed to "getBatchAction".
> -----------------------------------------------------------------------------------------------------
> nathanmarz:
> Actually the semantics aren't that clear. What does skip mean for an opaque
> spout, for example? In an opaque spout, not every batch is the same.
> Additionally, multiple spouts can be participating in the same batch. You'd
> want SKIP to cause only that one particular spout to skip the batch, while
> the other one emits.
> -----------------------------------------------------------------------------------------------------
> mrclip:
> It might make sense pull the failed records into a separate stream -- one
> that could be routed into a separate kafka topic for instance, or counted and
> then dropped. If nothing is listening on that stream, the records are
> effectively skipped. The spout just emits transparently, and the skipped
> records can still be transactionally handled on the Lethe stream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
