I've copied the relevant configuration options below. Doesn't this exception normally print out a useful error instead of just "null"?
producer config: max.message.size: 1000000 consumer config: fetch.size: 2072000 On Tue, Jul 10, 2012 at 10:57 AM, Jun Rao <jun...@gmail.com> wrote: > David, > > The fetch request gets an InvalidMessageSizeException. This means either > log corruption or the fetch size is smaller than the largest message. Could > you check your fetch size? > > Thanks, > > Jun > > On Mon, Jul 9, 2012 at 9:02 PM, David Siegel <dsie...@knewton.com> wrote: > > > I've just gotten the following error while running the zookeeper > consumer. > > > > I made a backup of the kafka log directory and wiped the logs. I > > restarting kafka and the consumer. After processing a few hundred > > messages successfully I got the same error again. I restarted the > consumer > > again and got the same error immediately. > > > > I'm running Kafka 0.7.1 > > > > I've included a sample of DumpLogSegments. The rest of the dumps looked > > the same. > > > > Thanks for your help. > > > > -David Siegel > > > > 2012-07-10 02:31:21,998 ERROR [Consumer1] > > c.k.h.c.k.KafkaConsumerServiceWorker: Failed to get next student event > > kafka.common.InvalidMessageSizeException: null > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > Method) ~[na:1.6.0_30] > > at > > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > > ~[na:1.6.0_30] > > at > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > > ~[na:1.6.0_30] > > at > java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > ~[na:1.6.0_30] > > at java.lang.Class.newInstance0(Class.java:355) ~[na:1.6.0_30] > > at java.lang.Class.newInstance(Class.java:308) ~[na:1.6.0_30] > > at > > kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53) > > ~[KPIP-0.4.birdy.jar:na] > > at > > > > > kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99) > > ~[KPIP-0.4.birdy.jar:na] > > at > > > kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82) > > ~[KPIP-0.4.birdy.jar:na] > > at > > kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:81) > > ~[KPIP-0.4.birdy.jar:na] > > at > > kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32) > > ~[KPIP-0.4.birdy.jar:na] > > at > > kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59) > > ~[KPIP-0.4.birdy.jar:na] > > at > kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51) > > ~[KPIP-0.4.birdy.jar:na] > > at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) > > ~[KPIP-0.4.birdy.jar:na] > > at > kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) > > ~[KPIP-0.4.birdy.jar:na] > > at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30] > > 2012-07-10 02:31:21,998 ERROR [Consumer1] > > c.k.h.c.k.KafkaConsumerServiceWorker: Iterator got into bad state. > Thread > > exiting > > java.lang.IllegalStateException: Iterator is in failed state > > at > kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:47) > > ~[KPIP-0.4.birdy.jar:na] > > at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:36) > > ~[KPIP-0.4.birdy.jar:na] > > at > kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:43) > > ~[KPIP-0.4.birdy.jar:na] > > at java.lang.Thread.run(Thread.java:662) [na:1.6.0_30] > > > > Dumping /mnt/spool/kafka/tmp-0/00000000000000000000.kafka > > Starting offset: 0 > > offset: 0 isvalid: true payloadsize: 1233 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 1243 isvalid: true payloadsize: 1232 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 2485 isvalid: true payloadsize: 1713 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 4208 isvalid: true payloadsize: 1181 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 5399 isvalid: true payloadsize: 1601 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 7010 isvalid: true payloadsize: 125 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 7145 isvalid: true payloadsize: 244 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 7399 isvalid: true payloadsize: 125 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 7534 isvalid: true payloadsize: 244 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 7788 isvalid: true payloadsize: 125 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 7923 isvalid: true payloadsize: 244 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 8177 isvalid: true payloadsize: 125 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 8312 isvalid: true payloadsize: 244 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 8566 isvalid: true payloadsize: 125 magic: 1 compresscodec: > > NoCompressionCodec > > offset: 8701 isvalid: true payloadsize: 244 magic: 1 compresscodec: > > NoCompressionCodec > > tail of the log is at offset: 8955 > > >
