Can you try following? 1) reduce buffer size to exactly 2x of your batch size ?
2) back up and save you old properties file. 3) create a dummy topic 4) produce and consume messages using the new config. Let us know. On Wed, 23 May 2018 at 07:02, Omer Litov <omer.li...@imperva.com> wrote: > Hi, > Thanks for the quick response. > > What I meant is, that had those exceptions on both versions (0.10.1 and > 1.0.1), so upgrading didn't solve the problem. And the same for increasing > the cluster size. > > Here are the properties we set: > > "key.serializer" = "org.apache.kafka.common.serialization.StringSerializer" > "value.serializer" = > "org.apache.kafka.common.serialization.ByteArraySerializer" > "acks" = "all" > "retries" = 5 > "buffer.memory" = 33554432 > "batch.size" = 16384 > "request.timeout.ms" = 50000 > > > thanks, > Omer > > On 22/05/2018, 8:14 PM, "M. Manna" <manme...@gmail.com> wrote: > > Hi, > > You said "On all of those setting there was no change in the described > behavior" - this is slightly confusing. Could you please clarify this? > If > there is no change, that means everything is working :) ? > > From the provided exception stack, it seems as if you are waiting to > batch > a lot of request but they are getting timed out for some reasons. > How have you upgraded the cluster to 1.0.1, and have you changed any > config? > > Regards, > > > On 22 May 2018 at 16:32, Omer Litov <omer.li...@imperva.com> wrote: > > > Hi, > > I have 4 producers, writing to a kafka cluster with 12 partitions. > > Periodically (about once per hour or two), we get exceptions while > trying > > to send messages to kafka. > > The exceptions are “Expiring 12 record(s) for > <topic_name>-<partition>: > > 49193 ms has passed since batch creation plus linger time”. > > We get that for all partitions (some more than others). > > When those exceptions arrive, they usually come a few hundred > together in > > under a second, and then stops. > > We don’t see any indications for retries (even though they are > configured). > > > > We first used a cluster of 3 brokers with version 0.10.1. > > Then upgraded to version 1.0.1 (with 3 brokers), and eventually added > > additional 2 brokers. > > On all of those setting there was no change in the described > behavior. > > > > > > Would appreciate any lead on investigation the issue. > > > > > > Thanks, > > Omer > > ------------------------------------------- > > NOTICE: > > This email and all attachments are confidential, may be proprietary, > and > > may be privileged or otherwise protected from disclosure. They are > intended > > solely for the individual or entity to whom the email is addressed. > > However, mistakes sometimes happen in addressing emails. If you > believe > > that you are not an intended recipient, please stop reading > immediately. Do > > not copy, forward, or rely on the contents in any way. Notify the > sender > > and/or Imperva, Inc. by telephone at +1 (650) 832-6006 and then > delete or > > destroy any copy of this email and its attachments. The sender > reserves and > > asserts all rights to confidentiality, as well as any privileges > that may > > apply. Any disclosure, copying, distribution or action taken or > omitted to > > be taken by an unintended recipient in reliance on this message is > > prohibited and may be unlawful. > > Please consider the environment before printing this email. > > > > >
