Jeff Fenchel created STORM-2107: ----------------------------------- Summary: Storm-kafka-client: Slow topology start breaks the Kafka Client with session timeouts Key: STORM-2107 URL: https://issues.apache.org/jira/browse/STORM-2107 Project: Apache Storm Issue Type: Bug Reporter: Jeff Fenchel
Specifically CommitFailedExceptions are being thrown when an attempt to commit the offset is made. The problem is detailed here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records On topology start, the spout will start emitting tuples before all bolts are prepared. This means that tuples will hang around in the topology until everything is ready to start processing. This caused the max spout pending limit to be hit, and the spout to stop polling kafka for the duration of the timeout. It seems like there could be other use cases that cause this too. I propose we start polling on a timer kafka to avoid this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)