thanks for the response Jason, i've already experimented with a similar solution myself, lowering max.partition.fetch.bytes to barely fit the largest message (2k at the moment)
still, i've observed similar problems, which is caused by really long processing times, e.g. downloading a large video via a link received in the message it's not very feasible to increase the heartbeat timeout too much, as session timeout is recommened to be at least 3 times that of heartbeat timeout. and that is bounded by broker's group.max.session.timeout.ms, which i would not want to increase as it would affect all other topics/consumers could there be an api for triggering the heartbeat manually maybe? it can be argued that that would beat the purpose of a heartbeat though, it might be used improperly, i.e. in my case rather than sending heartbeats inside the download/save loop but in an empty loop waiting for the download to complete, which might never happen. again, sending heartbeats in application code might be considered tight coupling as well other than that, i will experiment with the pause() api, separate thread for the actual message processing and poll()'ing with all partitions paused guven > On 25 Feb 2016, at 20:19, Jason Gustafson <[email protected]> wrote: > > Hey Guven, > > This problem is what KIP-41 was created for: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records > . > > The patch for this was committed yesterday and will be included in 0.10. If > you need something in the shorter term, you could probably use the client > from trunk (no changes to the server are needed). > > If this is still not sufficient, I recommend looking into the pause() API, > which can facilitate asynchronous message processing in another thread. > > -Jason > > On Thu, Feb 25, 2016 at 8:53 AM, Guven Demir <[email protected]> > wrote: > >> hi all, >> >> i'm having trouble processing a topic which includes paths to images which >> need to be downloaded and saved to disk (each takes ~3-5 seconds) and >> several are received on each poll >> >> within this scenario, i'm receiving the following error: >> >> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot >> be completed due to group rebalance >> >> which i assume is due to heartbeat failure and broker re-assigning the >> consumer's partition to another consumer >> >> are there any recommendations for processing long to process messages? >> >> thanks in advance, >> guven >> >> >>
