Hey Ryan, I ran into a similar issue and it was how the RoundRobinAssignor/Partitioner was hashing the keys in my messages. You may want to look at how thats implemented and see if its causing all of your messages to end up in the same partition.
For what its worth, this ticket has the implementation I put together to deal with it: https://issues.apache.org/jira/browse/KAFKA-3333 Stephen On Wed, May 30, 2018 at 5:28 AM, Ryan En <ryan_...@hotmail.com> wrote: > Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the > messages from one Kafka cluster to another. > > > The source and target Kafka cluster are pretty much set up the same... > replication factor is 3, number of partitions is 3, > auto.create.topics.enable is true. > > > I am finding that in the target Kafka cluster, all the messages produced > by Mirrormaker are going into one partition only. > > > Here are my configs: > > > mirrormaker producer > > > security.protocol=SASL_SSL > sasl.mechanism=PLAIN > sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > required username="xxxxxxxx" password="xxxxxxxxxx"; > ssl.truststore.location=/path/to/truststore.jks > ssl.truststore.password=xxxxxxxxxxx > ssl.keystore.location=/path/to/keystore.jks > ssl.keystore.password=xxxxxxxxx > ssl.key.password=xxxxxxxx > retries=5000000 > max.in.flight.requests.per.connection=1 > client.id=xxxxxxxx > compression.type=lz4 > bootstrap.servers=<aggregate-broker>:9292 > acks=all > > > mirrormaker consumer > > > security.protocol=SASL_SSL > sasl.mechanism=PLAIN > sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule > required username="xxxxxxxx" password="xxxxxxxxxx"; > ssl.truststore.location=/path/to/truststore.jks > ssl.truststore.password=xxxxxxxxxx > ssl.keystore.location=/path/to/keystore.jks > ssl.keystore.password=xxxxxxxxxx > ssl.key.password=xxxxxxxxx > client.id=xxxxxxxxxxxx > partition.assignment.strategy=org.apache.kafka.clients. > consumer.RoundRobinAssignor > auto.commit.interval.ms=1000 > auto.offset.reset=earliest > bootstrap.servers=<local-broker>:9192 > session.timeout.ms=30000 > group.id=xxxxxxxxxxxxxx > enable.auto.commit=false > > > Any ideas why this may happen? > > > Thanks in advance >
