Hi everybody,
I just tried to upgrade from the "traditional" KafkaSpout to the new
storm-kafka-client version. Initially everything was working fine, yet I soon
discovered that the new Spout stops emitting tuples after some time.
The happy path (just ack, no errors) seems to work fine, yet when tuples are
failed or delayed, it seems that the (spout-internal) counters are incorrect.
After some debugging sessions, I found the reason here:
private boolean poll() {
return !waitingToEmit() && numUncommittedOffsets <
kafkaSpoutConfig.getMaxUncommittedOffsets();
}
This method always returned false since numUncommittedOffsets was much higher
than kafkaSpoutConfig.getMaxUncommittedOffsets(). Apparently I am not the only
one facing this problem because similar issues are reported here:
https://github.com/apache/storm/pull/1131#issuecomment-217776015
I could not find any commits related to these issues.
In the latest 1.0.2 release I saw that there are quite a few fixes for the
"old" KafkaSpout, yet almost none for the "new" one. Will both versions be
supported in future? Which one is recommended version to use?
Thanks,
Tom
