Ishwara Varnasi created STORM-2957:
--------------------------------------

             Summary: Offsets are not committed when no ackers set (ackers=0) 
and KafkaSpout stops polling messages once it reaches the max uncommitted 
offsets 
                 Key: STORM-2957
                 URL: https://issues.apache.org/jira/browse/STORM-2957
             Project: Apache Storm
          Issue Type: Bug
          Components: storm-kafka-client
    Affects Versions: 1.0.5
            Reporter: Ishwara Varnasi


A KafkaSpout stops polling messages from topic when topology has no ackers set 
("topology.acker.executors"= 0). This happens after KafkaSpout polls (and 
emits) messages equal to the max uncommitted offsets 
(KafkaspoutConfig.maxUncommittedOffsets).

This happens because the KafkaSpout.ack() is called even before the emit method 
adds the message to "emitted" collection (Set). In such cases, the message is 
not added "acked" collection (Map), and hence never committed. This seem to be 
bug introduced in a version later than storm-kafka-client 1.0.1 and current 
code (1.0.5) has a check in ack() that checks if the acked message is available 
in "emitted" collection and if not, ignores it (never adds it to "acked" 
collection)

Steps to reproduce issue:
1. Need a topology with KafkaSpout from storm-kafka-client
2. Set number of ackers ("topology.acker.executors") to o 
(org.apache.storm.Config.setNumAckers(0)
3. While creating KafkaSpout instance, set SpoutConfig.setMaxUncommittedOffsets 
to small number (50 or 100)
4. Start topology, see to it that KafkaSpout polls more messages from Kafka 
that setMaxUncommittedOffsets
5. KafkaSpout stops polling messages from topic



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to