Re: Kafka Mirrormaker issue

For whatever is worth and from memory In previous client versions (may have been fixed in 0.11), we had 3 consumers in the same consumer group, when a topic partition reassignment happened and 2 consumers had partitions and but the other one did not get any. So you could be in that scenario but

Re: Kafka Mirrormaker issue

We had trouble with batch expired produce errors for high (not really that high, maybe 400 msgs/sec) volume topic partitions. We solved these by increasing `request.timeout.ms` and all increasing `batch.size` (which reduced the total number of waiting batches in MirrorMaker). More context here:

Re: Kafka Mirrormaker issue

We've been stable all weekend with the following settings: ExecStart=/usr/bin/kafka-mirror-maker --abort.on.send.failure true --new.consumer --num.streams 6 --offset.commit.interval.ms 6 --consumer.config /etc/kafka/mirrormaker/telem_mm/consumer.properties --producer.config

Re: Kafka Mirrormaker issue

I'm hitting the same problem, even with the new consumer, on MirrorMaker 0.9 reading from a 0.9 Kafka cluster and producing to a 0.11 Kafka cluster. On 3/30/18, 3:56 PM, "Andrew Otto" wrote: I’m currently stuck on MirrorMaker version 0.9, and I’m not sure when the

Re: Kafka Mirrormaker issue

I’m currently stuck on MirrorMaker version 0.9, and I’m not sure when the new consumer client became the default. Does your 0.10 version have a —new.consumer option listed in the help message? If so, then the new consumer client is not the default. I haven’t seen the problem you are describing

Re: Kafka Mirrormaker issue

Any other update on this? On Mon, Mar 26, 2018, 7:42 PM Andrew Otto wrote: > I’ve had similar problems, but I don’t have an explanation for ya :/ > > On Sun, Mar 25, 2018 at 12:19 PM, Siva A wrote: > > > Hi, > > > > We have 3 nodes Kafka

Re: Kafka Mirrormaker issue

I’ve had similar problems, but I don’t have an explanation for ya :/ On Sun, Mar 25, 2018 at 12:19 PM, Siva A wrote: > Hi, > > We have 3 nodes Kafka cluster(0.10.0.1) and its mirroring the data from > another 3 node cluster of same Kafka version. > Both the clusters

Kafka Mirrormaker issue

Hi, We have 3 nodes Kafka cluster(0.10.0.1) and its mirroring the data from another 3 node cluster of same Kafka version. Both the clusters are Kerberized and we are running the Mirrormaker on the target cluster using the single principal/keytab with the one way trust on the KDC. At times, the