I'm not explicitly using an encoder. All of my messages are encoded using Avro and I send the resulting binary is passed to Kafka. That binary array is definitely less than the size of the buffer (About 37MB). Checking the size of that was one of the first things I did.
This begs the question: is there a default encoder that's used? If so, could that be making the size larger than the buffer size? Casey ________________________________________ From: Jun Rao [jun...@gmail.com] Sent: Thursday, February 16, 2012 2:02 PM To: kafka-users@incubator.apache.org Subject: Re: Large message causing a MessageSizeTooLargeException What encoder are you using? Is it possible that after encoding, the message size is larger than the threshold? Thanks, Jun On Thu, Feb 16, 2012 at 10:18 AM, Sybrandy, Casey < casey.sybra...@six3systems.com> wrote: > Yes, I do still see the exception. Is there a server setting that needs > to be updated? > > Has anyone else tried to push multi-megabyte files through Kafka? > > Casey > ________________________________________ > From: Jun Rao [jun...@gmail.com] > Sent: Thursday, February 16, 2012 1:13 PM > To: kafka-users@incubator.apache.org > Subject: Re: Large message causing a MessageSizeTooLargeException > > There could be some transient failures in our unit tests especially on a > less powerful machine. Do you still see the exception? > > Thanks, > > Jun > > On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey < > casey.sybra...@six3systems.com> wrote: > > > Just tried it. No change. However, when I tried to create a release for > > it, it failed a number of unit tests. I resorted to just packaging it > and > > starting zookeeper and kafka in that directory and appeared to work fine, > > though I'm unsure if the failed tests have anything to do with it. I > don't > > suppose there's a pre-compiled release somewhere that I missed. > > ________________________________________ > > From: Jun Rao [jun...@gmail.com] > > Sent: Thursday, February 16, 2012 11:35 AM > > To: kafka-users@incubator.apache.org > > Subject: Re: Large message causing a MessageSizeTooLargeException > > > > Could you try the 0.7 release, which is the first Apache release that we > > officially support? > > > > Thanks, > > > > Jun > > > > On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey < > > casey.sybra...@six3systems.com> wrote: > > > > > No, I do not have compression enabled. > > > > > > Casey > > > > > > ________________________________________ > > > From: Jun Rao [jun...@gmail.com] > > > Sent: Wednesday, February 15, 2012 6:31 PM > > > To: kafka-users@incubator.apache.org > > > Subject: Re: Large message causing a MessageSizeTooLargeException > > > > > > Did you enable compression? If so, try disable compression. > > > > > > Thanks, > > > > > > Jun > > > > > > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey < > > > casey.sybra...@six3systems.com> wrote: > > > > > > > Hello, > > > > > > > > I have a weird bug that I'm trying to figure out. I'm trying to > send a > > > > 30+ MB message through Kafka, but it keeps giving me a > > > > MessageSizeTooLargeException. I did change the max.message.size > > property > > > > on the producer to be 50MB (50 * 1024 * 1024). Is there something > I'm > > > > missing? The consumer is also set to accept messages up to the same > > > size. > > > > > > > > Kafka 0.6.0 > > > > > > > > Thanks. > > > > > > > > Casey > > > > > >
