Prasanna Ranganathan created STORM-2546:
-------------------------------------------
Summary: Kafka spout can stall / get stuck due to edge case with
failing tuples
Key: STORM-2546
URL: https://issues.apache.org/jira/browse/STORM-2546
Project: Apache Storm
Issue Type: Bug
Components: storm-kafka-client
Affects Versions: 2.0.0, 1.x
Reporter: Prasanna Ranganathan
The mechanism for replaying a failed tuple involves seeking the kafka consumer
to the failing offset and then re-emitting it into the topology. A tuple, when
emitted the first time, will have an entry created in OffsetManager. This entry
will be removed only after the tuple is successfully acknowledged and its
offset successfully committed. Till then, commits for offsets beyond the
failing offset for that TopicPartition will be blocked.
It is possible that when the spout seeks the consumer to the failing offset,
the corresponding kafka message is not returned in the poll response. This can
happen due to that offset being deleted or compacted away. In this scenario
that partition will be blocked from committing and progressing.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
