Kevin Conaway created STORM-2608:
------------------------------------
Summary: Out Of Range Offsets Should Be Removed From Pending Queue
Key: STORM-2608
URL: https://issues.apache.org/jira/browse/STORM-2608
Project: Apache Storm
Issue Type: Bug
Components: storm-kafka
Affects Versions: 0.10.0, 1.0.4, 1.1.1
Reporter: Kevin Conaway
There is a bug that occurs when failed tuples are invalidated due to Kafka
throwing a {{TopicOffsetOutOfRangeException}}.
Below is what happens:
- Spout emits tuples
- Offsets are added to the _pending_ tree
- Some tuples fail and are added to the _failedMsgRetryManager_
- On the next fetch request, a {{TopicOffsetOutOfRangeException}} is thrown and
the new offset is _after_ the offset that are currently sitting in both the
_pending_ tree and the _failedMsgRetryManager_
- All offsets smaller than the the new offset are removed from the
_failedMsgRetryManager_ but *not* the _pending_ tree.
- Since those offsets were removed from the _failedMsgRetryManager_ they will
never be retried and thus never get removed from _pending_
- {{lastCommittedOffset()}} will always return the same value which means that
offset in zookeeper for that partition will never get updated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
