This topic is currently being discussed at https://issues.apache.org/jira/browse/KAFKA-2986 and https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records
On 3/2/16, 8:11 AM, "Vanessa Gligor" <vanessagli...@gmail.com> wrote: >Hello, > >I am using Kafka higher consumer 0.9.0. I am not using the auto commit for >the offsets, so after I consume the messaged (poll from Kafka) I will have >to commit the offsets manually. > >The issue that I have is actually that the processing of the messages takes >longer than 30s (and I cannot call poll again, before these messages are >processed) and when I try to commit the offset a exception is thrown: >ERROR o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred >while committing offsets for group MetadataConsumerSpout. >(I have found on stackoverflow this explanation: so if you wait for longer >that the timeout request then the coordinator for the topic will kickout >the consumer because it will think is dead and it will rebalance the group) > >In order to get rid of this I have thought about a couple of solutions: > >1. The configuration session.timeout.ms has a maximum value, so if I try to >set it to 60 seconds, also I get an exception, because this value is not in >the valid interval. > >2. I have tried to find a solution to get a paginated request when the >polling method is called - no success. > >3. I have tried to send a heart beat from the outside of the poll (because >this method sends the heartbeats) - no success. > > >Thank you. CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
