[
https://issues.apache.org/jira/browse/STORM-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280384#comment-16280384
]
Stig Rohde Døssing commented on STORM-2844:
-------------------------------------------
[~hmclouro] I took another look at the STORM-2666 fix, and I think there is
another case where that fix is incomplete.
STORM-2666 was supposed to ensure that once offset _n_ is committed in the
OffsetManager, any offset _k_ < _n_ won't be emitted again. STORM-2666 adjusts
the KafkaConsumer position when tuples are committed to prevent this.
I missed that waitingToEmit might contain records that are behind the committed
offset. We need to make sure that once offsets get committed, any records
behind the committed offset are removed from waitingToEmit as well. I'd raise a
new issue for this and fix it, but I'm wondering if you want to handle it in
the PR for this as well, since it's likely going to involve some of the same
code?
> KafkaSpout Throws IllegalStateException After Committing to Kafka When First
> Poll Strategy Set to EARLIEST
> ----------------------------------------------------------------------------------------------------------
>
> Key: STORM-2844
> URL: https://issues.apache.org/jira/browse/STORM-2844
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
> Affects Versions: 2.0.0, 1.2.0
> Reporter: Hugo Louro
> Assignee: Hugo Louro
> Priority: Critical
>
> This
> [code|https://github.com/apache/storm/blob/1.x-branch/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L407-L409],
> which was committed to fix
> [STORM-2666|https://issues.apache.org/jira/browse/STORM-2666] throws
> IllegalStateException when the KafkaSpout commits to Kafka and is restarted
> with the same consumer group id and first poll strategy set to EARLIEST.
> For example consider the following sequence:
> # KafkaSpout with consumer_group_id=TEST polls and commits offsets 1-5
> # KafkaSpout with consumer_group_id=TEST is restarted with first poll
> strategy set to EARLIEST
> ==> IllegalStateException will be thrown
> This bug could be a blocker. I am setting it to Critical because assigning a
> different consumer id serves as a workaround to the problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
