Jeff Fenchel commented on STORM-2106:

[~Srdo] I don't think it quite works that way. In the case you provided, I 
agree with you that x+1..y will be skipped while x is processing. However in 
doSeekRetriableTopicPartitions we seek to the next offset that may be committed 
based on acked messages. So in this case, since x is still processing it seeks 
back to x and filters out all of x-y again because x is processing and the rest 
are acked. The result is nothing gets emitted until x finishes processing. 

For me the batch size kafka provide (~300) was << maxUncommittedOffsets which 
severely crippled the throughput of my topology. 

I agree that we should poll kafka again at y and that is why I made this 

Code snippet: 
    private void doSeekRetriableTopicPartitions() {
        final Set<TopicPartition> retriableTopicPartitions = 

        for (TopicPartition rtp : retriableTopicPartitions) {
            final OffsetAndMetadata offsetAndMeta = 
            if (offsetAndMeta != null) {
                kafkaConsumer.seek(rtp, offsetAndMeta.offset() + 1);  // seek 
to the next offset that is ready to commit in next commit cycle
            } else {
                kafkaConsumer.seek(rtp, acked.get(rtp).committedOffset + 1);    
// Seek to last committed offset

> Storm Kafka Client is paused while failed tuples are replayed
> -------------------------------------------------------------
>                 Key: STORM-2106
>                 URL: https://issues.apache.org/jira/browse/STORM-2106
>             Project: Apache Storm
>          Issue Type: Bug
>            Reporter: Jeff Fenchel
> With the changes in STORM-2087, the kafka 10 spout will limited to emitting 
> tuples that are within the poll() size for kafka. This means that if the 
> first tuple in a batch from kafka is failed, the spout will not emit more 
> than the size of the batch from kafka until the tuple is either processed 
> successfully or given up on. This behavior is exacerbated by the exponential 
> backoff retry policy.  
> There probably needs to be bookkeeping for the next emittable offset.

This message was sent by Atlassian JIRA

Reply via email to