[
https://issues.apache.org/jira/browse/STORM-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370499#comment-16370499
]
Ishwara Varnasi commented on STORM-2957:
----------------------------------------
Sure, for now we'd to go with storm-core 1.0.5 and storm-kafka-client 1.0.1.
Will update here if we upgrade to storm 1.0.6. Thanks.
> Offsets are not committed when no ackers set (ackers=0) and KafkaSpout stops
> polling messages once it reaches the max uncommitted offsets
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: STORM-2957
> URL: https://issues.apache.org/jira/browse/STORM-2957
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
> Affects Versions: 1.0.5
> Reporter: Ishwara Varnasi
> Priority: Blocker
>
> A KafkaSpout stops polling messages from topic when topology has no ackers
> set ("topology.acker.executors"= 0). This happens after KafkaSpout polls (and
> emits) messages equal to the max uncommitted offsets
> (KafkaspoutConfig.maxUncommittedOffsets).
> This happens because the KafkaSpout.ack() is called even before the emit
> method adds the message to "emitted" collection (Set). In such cases, the
> message is not added "acked" collection (Map), and hence never committed.
> This seem to be bug introduced in a version later than storm-kafka-client
> 1.0.1 and current code (1.0.5) has a check in ack() that checks if the acked
> message is available in "emitted" collection and if not, ignores it (never
> adds it to "acked" collection)
> Steps to reproduce issue:
> 1. Need a topology with KafkaSpout from storm-kafka-client
> 2. Set number of ackers ("topology.acker.executors") to o
> (org.apache.storm.Config.setNumAckers(0)
> 3. While creating KafkaSpout instance, set
> SpoutConfig.setMaxUncommittedOffsets to small number (50 or 100)
> 4. Start topology, see to it that KafkaSpout polls more messages from Kafka
> that setMaxUncommittedOffsets
> 5. KafkaSpout stops polling messages from topic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)