Vinayagam Rajanbabu created KAFKA-20818:
-------------------------------------------
Summary: Kafka Mirror Maker latency increase intermittently
Key: KAFKA-20818
URL: https://issues.apache.org/jira/browse/KAFKA-20818
Project: Kafka
Issue Type: Task
Components: connect, mirrormaker
Affects Versions: 3.9.2
Reporter: Vinayagam Rajanbabu
h3. Summary
Kafka MirrorMaker replication latency exhibits intermittent increases between
Confluent Cloud Kafka and a standalone Kafka cluster hosted in GCP.
A Kafka Connect MirrorMaker deployment is operating in distributed mode.
Replication latency increases occur intermittently.
The following checks were performed and did not reveal a definitive root cause
in the monitored metrics. Please review and indicate any additional metrics or
diagnostics to pursue in order to advance the analysis of this issue.
* Confluent Kafka broker CPU and memory utilization follow expected patterns.
* Standalone Kafka broker CPU and memory utilization follow expected patterns.
* Standalone Kafka broker socket connections, cache utilization, and disk
usage remained within normal parameters.
* MirrorMaker (MM2) record counts, source record poll rate, and sink record
rate were consistent with baseline behavior prior to the incidents.
* Other application services communicating with the standalone Kafka cluster
were able to connect to Confluent and process requests according to normal
patterns.
* No new or recurring errors were observed in Kafka Connect services.
* Replication latency affects the majority of topics; however, topics with
relatively high throughput experience a greater and more prolonged latency
impact.
* No consistent pattern has been identified with respect to impacted topics or
brokers.
*Versions:*
* Standalone Kafka Server Version: v3.9.2
* Kafka Connect Version: v3.9.2
h4. connect-distributed.properties
bootstrap.servers=my-standalone-kafka-server
group.id=kafka-connect
consumer.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
connector.client.config.override.policy=All
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
offset.storage.topic=kafka-connect-offsets
config.storage.topic=kafka-connect-configs
status.storage.topic=kafka-connect-status
listeners=http://:8083
rest.advertised.port=8083
{{}}
Task Request Submission Json:
{
"name": "mirror-connector",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"topics": "my-confluent-topic1, myconfluent-topic2",
"source.cluster.alias": "confluent",
"target.cluster.alias": "standalone",
"source.cluster.bootstrap.servers": "my-confluent-kafka-server",
"target.cluster.bootstrap.servers": "my-standalone-kafka-server",
"refresh.topics.enabled": true,
"refresh.groups.enabled": false,
"sync.topic.configs.enabled": true,
"sync.topic.acls.enabled": false,
"groups.exclude": "my-consumer-group1",
"topics.exclude": "my-confluent-topic4, myconfluent-topic5",
"replication.policy.class":
"org.apache.kafka.connect.mirror.IdentityReplicationPolicy",
"offset-syncs.topic.replication.factor": 3,
"tasks.max": 3,
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"header.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"source.consumer.group.id": "mirror-maker",
"source.consumer.auto.offset.reset": "latest",
"source.consumer.partition.assignment.strategy":
"org.apache.kafka.clients.consumer.RoundRobinAssignor",
"source.consumer.ssl.endpoint.identification.algorithm": "https",
"source.consumer.security.protocol": "SASL_SSL",
"source.consumer.sasl.mechanism": "PLAIN",
"source.consumer.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"mirror-user\" password=\"mirror-password\";",
"source.consumer.max.partition.fetch.bytes": "41943040",
"source.consumer.fetch.max.bytes": "41943040",
"source.admin.ssl.endpoint.identification.algorithm": "https",
"source.admin.security.protocol": "SASL_SSL",
"source.admin.sasl.mechanism": "PLAIN",
"source.admin.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"mirror-user\" password=\"mirror-password\";",
"source.producer.max.request.size": "41943040",
"producer.override.max.request.size": "41943040",
"source.producer.ssl.endpoint.identification.algorithm": "https",
"source.producer.security.protocol": "SASL_SSL",
"source.producer.sasl.mechanism": "PLAIN",
"source.producer.sasl.jaas.config":
"org.apache.kafka.common.security.plain.PlainLoginModule required
username=\"mirror-user\" password=\"mirror-password\";"
}
*Mitigation Steps:*
The issue has been mitigated using the following interventions.
# Increased the number of connector tasks.
# Increased the partition count for topics with high throughput requirements.
# Scaled the broker cluster by adding broker instances.
# Performed data rebalancing to mitigate concentrated load on specific brokers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
