Adrian Seungjin Lee created STORM-618:
-----------------------------------------
Summary: should provide optional way to implement at-most once
semantic
Key: STORM-618
URL: https://issues.apache.org/jira/browse/STORM-618
Project: Apache Storm
Issue Type: Bug
Components: storm-kafka
Affects Versions: 0.9.3
Reporter: Adrian Seungjin Lee
While it's nice for kafka spout to push failed tuple back into a sorted set and
try to process it again, this way of guaranteed message processing sometimes
makes situation pretty bad when a failed tuple repeatedly fails in downstream
bolts since PartitionManager#fill method tries to fetch from that offset
repeatedly.
This is a corresponding code snippet.
private void fill() {
...
if (had_failed) {
offset = failed.first();
} else {
offset = _emittedToOffset;
}
...
msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer,
_partition, offset);
...
So there should be an option for a developer to decide if he wants to process
failed tuple again or just skip failed tuple. One of the best thing of Storm is
that spout together with trident can be implemented to guarantee
at-least-once,exactly-once and at-most-once message processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
