Github user srdo commented on the issue:
https://github.com/apache/storm/pull/1487
You probably shouldn't enable auto commit if you can't accept some message
loss. As I understand it, it makes the Kafka consumer commit the messages it
has received from poll() every so often. That means that messages can be lost
if they've been pulled out of the consumer, but the topology hasn't acked them
yet. If you disable auto commit, the messages will instead be committed when
the topology acks them.
I can't reproduce this issue. Here are the steps I took:
1) Do a fresh install of Kafka 0.9.0.1
2) Do a fresh install of Storm 1.0.1, start Nimbus and 1 supervisor on
localhost
3) Create a 1 partition topic on a newly installed 0.9.0.1 Kafka broker
4) Deploy a topology containing only the Kafka spout and a bolt printing
the messages it receives to system.out.
5) Send 5 messages to Kafka using the kafka-console-producer.sh script in
kafka/bin
6) Verify that the topology prints those messages
7) Kill the topology and wait for 30 seconds for it to be completely shut
down
8) Send 5 more messages
9) Redeploy
10) Confirm that all 5 new messages are received
Here's the code I used https://github.com/srdo/TestKafkaSpout
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
