Hello, I am running kafka 0.7.1 and try to make mirroring works accross DC. It usually works well when mirroring a single topic. However I have noticed that when I am not limiting the number of topics, the mirrormaker hangs with an InvalidRequestException : -xxxx is not a valid request size
This is thrown by the BoundedByteBufferReceive class (https://github.com/apache/kafka/blob/0.7.1/core/src/main/scala/kafka/network/BoundedByteBufferReceive.scala#L60) when the announced size of the MultiFetchRequest response has an negative size. This is due to an overflow on server side : https://github.com/apache/kafka/blob/0.7.1/core/src/main/scala/kafka/server/MultiMessageSetSend.scala#L30 Indeed while server respect the fetchsize (in my case 100mb), it send fetchsize per topic per partition. Since this is unbounded, this might (and does) overflow. This bug seems to be present also in 0.8 : https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/api/FetchResponse.scala#L171 (even if I have not tested it yet). A quick work-around for me would be to decrease fetchsize (using the number of topics and the number of partition I expect). However this does not scale up because the number of partitions and topics are not known when configuring the mirrormaker. Can we think of a solution to fix this ? Either by taking the fetchsize as a hint (maybe dropping some data if the data size sum overflows) and to take the fetch size as a global limit (not per topic per partition). -- Grégoire
