In some sense, fetchSize is a per partition thing since it has to be larger than the largest single message. Otherwise, the consumer will get stuck. I agree that this will make the multi-fetch size unpredictable, which is inconvenient. We probably need to think more about a solution. Could you open a jira for this?
Thanks, Jun On Thu, Aug 2, 2012 at 2:54 AM, Grégoire Seux <g.s...@criteo.com> wrote: > Hello, > > I am running kafka 0.7.1 and try to make mirroring works accross DC. > It usually works well when mirroring a single topic. > However I have noticed that when I am not limiting the number of topics, > the mirrormaker hangs with an InvalidRequestException : > -xxxx is not a valid request size > > This is thrown by the BoundedByteBufferReceive class ( > https://github.com/apache/kafka/blob/0.7.1/core/src/main/scala/kafka/network/BoundedByteBufferReceive.scala#L60) > when the announced size of the MultiFetchRequest response has an negative > size. > > This is due to an overflow on server side : > > https://github.com/apache/kafka/blob/0.7.1/core/src/main/scala/kafka/server/MultiMessageSetSend.scala#L30 > > Indeed while server respect the fetchsize (in my case 100mb), it send > fetchsize per topic per partition. Since this is unbounded, this might (and > does) overflow. > > This bug seems to be present also in 0.8 : > https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/api/FetchResponse.scala#L171(even > if I have not tested it yet). > > > A quick work-around for me would be to decrease fetchsize (using the > number of topics and the number of partition I expect). However this does > not scale up because the number of partitions and topics are not known when > configuring the mirrormaker. > > Can we think of a solution to fix this ? Either by taking the fetchsize as > a hint (maybe dropping some data if the data size sum overflows) and to > take the fetch size as a global limit (not per topic per partition). > > -- > Grégoire > > > >
