Casey, This could be a bug. Do you mind filing a JIRA with a test case that we can run to reproduce this problem ?
Thanks, Neha On Thu, Feb 16, 2012 at 12:24 PM, Sybrandy, Casey <casey.sybra...@six3systems.com> wrote: > I'm not explicitly using an encoder. All of my messages are encoded using > Avro and I send the resulting binary is passed to Kafka. That binary array > is definitely less than the size of the buffer (About 37MB). Checking the > size of that was one of the first things I did. > > This begs the question: is there a default encoder that's used? If so, could > that be making the size larger than the buffer size? > > Casey > ________________________________________ > From: Jun Rao [jun...@gmail.com] > Sent: Thursday, February 16, 2012 2:02 PM > To: kafka-users@incubator.apache.org > Subject: Re: Large message causing a MessageSizeTooLargeException > > What encoder are you using? Is it possible that after encoding, the message > size is larger than the threshold? > > Thanks, > > Jun > > On Thu, Feb 16, 2012 at 10:18 AM, Sybrandy, Casey < > casey.sybra...@six3systems.com> wrote: > >> Yes, I do still see the exception. Is there a server setting that needs >> to be updated? >> >> Has anyone else tried to push multi-megabyte files through Kafka? >> >> Casey >> ________________________________________ >> From: Jun Rao [jun...@gmail.com] >> Sent: Thursday, February 16, 2012 1:13 PM >> To: kafka-users@incubator.apache.org >> Subject: Re: Large message causing a MessageSizeTooLargeException >> >> There could be some transient failures in our unit tests especially on a >> less powerful machine. Do you still see the exception? >> >> Thanks, >> >> Jun >> >> On Thu, Feb 16, 2012 at 9:22 AM, Sybrandy, Casey < >> casey.sybra...@six3systems.com> wrote: >> >> > Just tried it. No change. However, when I tried to create a release for >> > it, it failed a number of unit tests. I resorted to just packaging it >> and >> > starting zookeeper and kafka in that directory and appeared to work fine, >> > though I'm unsure if the failed tests have anything to do with it. I >> don't >> > suppose there's a pre-compiled release somewhere that I missed. >> > ________________________________________ >> > From: Jun Rao [jun...@gmail.com] >> > Sent: Thursday, February 16, 2012 11:35 AM >> > To: kafka-users@incubator.apache.org >> > Subject: Re: Large message causing a MessageSizeTooLargeException >> > >> > Could you try the 0.7 release, which is the first Apache release that we >> > officially support? >> > >> > Thanks, >> > >> > Jun >> > >> > On Thu, Feb 16, 2012 at 7:58 AM, Sybrandy, Casey < >> > casey.sybra...@six3systems.com> wrote: >> > >> > > No, I do not have compression enabled. >> > > >> > > Casey >> > > >> > > ________________________________________ >> > > From: Jun Rao [jun...@gmail.com] >> > > Sent: Wednesday, February 15, 2012 6:31 PM >> > > To: kafka-users@incubator.apache.org >> > > Subject: Re: Large message causing a MessageSizeTooLargeException >> > > >> > > Did you enable compression? If so, try disable compression. >> > > >> > > Thanks, >> > > >> > > Jun >> > > >> > > On Wed, Feb 15, 2012 at 2:37 PM, Sybrandy, Casey < >> > > casey.sybra...@six3systems.com> wrote: >> > > >> > > > Hello, >> > > > >> > > > I have a weird bug that I'm trying to figure out. I'm trying to >> send a >> > > > 30+ MB message through Kafka, but it keeps giving me a >> > > > MessageSizeTooLargeException. I did change the max.message.size >> > property >> > > > on the producer to be 50MB (50 * 1024 * 1024). Is there something >> I'm >> > > > missing? The consumer is also set to accept messages up to the same >> > > size. >> > > > >> > > > Kafka 0.6.0 >> > > > >> > > > Thanks. >> > > > >> > > > Casey >> > > >> > >>
