My observations so far: If I start with 1 broker, 1 consumer (code is pretty much the same as in the simple-consumer-demo) and 1 producer I can do arbitrary many runs and everything works fine. Adding a broker after the producer stopped and restarting the producer (consumer still running) throws on the consumer side the InvalidMessageSize exception - which continues to happen on subsequent produce/consume cycles as long as the second broker is running.
I looked around Zookeeper - there are 8 partitions when one broker is active and 16 with two... i see all the topics are registered in zookeeper and overall zookeeper logs are clean. After this fact I tried to see if two brokers and one partition each would work so I shutdown the brokers, changed the global partition size to 1 in each of the configs and restarted them. The result is the same, I still get InvalidMessageSize exceptions on the consumer size. On Fri, Dec 9, 2011 at 11:10 AM, Florian Leibert <f...@leibert.de> wrote: > On that note - another error I see right around the same time is below - i > see it on the consumer end. Note that in this case I have one consumer with > a single iterator - and on each broker 8 partitions. When printing > messages, I see an empty message on occasion... > > Exception in thread "main" kafka.common.InvalidMessageSizeException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at java.lang.Class.newInstance0(Class.java:355) > at java.lang.Class.newInstance(Class.java:308) > at kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) > at > kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:60) > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:67) > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:30) > at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:58) > at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:50) > at com.airbnb.logging.util.ConsumerTest.main(ConsumerTest.java:50) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120) > > -- Best regards, Florian http://twitter.com/flo <http://twitter.com/floleibert> http://flori.posterous.com/
