Thanks for your answer. Maybe we need to ensure broker-side that size won't overflow and not send all topics, if it does. Let's talk about this on https://issues.apache.org/jira/browse/KAFKA-436
-- Grégoire -----Original Message----- From: Jun Rao [mailto:jun...@gmail.com] Sent: jeudi 2 août 2012 19:39 To: kafka-users@incubator.apache.org Subject: Re: multifetch overflow In some sense, fetchSize is a per partition thing since it has to be larger than the largest single message. Otherwise, the consumer will get stuck. I agree that this will make the multi-fetch size unpredictable, which is inconvenient. We probably need to think more about a solution. Could you open a jira for this? Thanks, Jun On Thu, Aug 2, 2012 at 2:54 AM, Grégoire Seux <g.s...@criteo.com> wrote: > Hello, > > I am running kafka 0.7.1 and try to make mirroring works accross DC. > It usually works well when mirroring a single topic. > However I have noticed that when I am not limiting the number of > topics, the mirrormaker hangs with an InvalidRequestException : > -xxxx is not a valid request size > > This is thrown by the BoundedByteBufferReceive class ( > https://github.com/apache/kafka/blob/0.7.1/core/src/main/scala/kafka/n > etwork/BoundedByteBufferReceive.scala#L60) > when the announced size of the MultiFetchRequest response has an > negative size. > > This is due to an overflow on server side : > > https://github.com/apache/kafka/blob/0.7.1/core/src/main/scala/kafka/s > erver/MultiMessageSetSend.scala#L30 > > Indeed while server respect the fetchsize (in my case 100mb), it send > fetchsize per topic per partition. Since this is unbounded, this might > (and > does) overflow. > > This bug seems to be present also in 0.8 : > https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/api/FetchResponse.scala#L171(even > if I have not tested it yet). > > > A quick work-around for me would be to decrease fetchsize (using the > number of topics and the number of partition I expect). However this > does not scale up because the number of partitions and topics are not > known when configuring the mirrormaker. > > Can we think of a solution to fix this ? Either by taking the > fetchsize as a hint (maybe dropping some data if the data size sum > overflows) and to take the fetch size as a global limit (not per topic per > partition). > > -- > Grégoire > > > >
