Hello All,
Anyone has installed KafkaMirrorMaker2 on GKE ?
Need some help to debug/resolve issues that I've been having.
I've 2 clusters on GKE with strimzi kafka installed.
cluster1(nossl-e) on use-east1, cluster2(nossl-e) on us-west1.
Both the clusters are Auto-pilot cluster (mentioning this, thought it might
not be relevant for the issues being faced)
source cluster -> nossl-w
target cluster -> nossl-e
Currently(to get MM2 working), SSL & authorization are not enabled.
KafkaMirrorMaker2 is installed on source cluster i.e. nossl-w
Here is the yaml :
```
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
name: my-mirror-maker-2
spec:
version: 3.0.0
replicas: 1
connectCluster: "my-target-cluster-east"
clusters:
- alias: "my-source-cluster-west"
bootstrapServers: nossl-w-kafka-bootstrap:9092
- alias: "my-target-cluster-east"
bootstrapServers: xx.xx.xx.xx:9094
config:
# -1 means it will use the default replication factor configured in
the broker
config.storage.replication.factor: -1
offset.storage.replication.factor: -1
status.storage.replication.factor: -1
mirrors:
- sourceCluster: "my-source-cluster-west"
targetCluster: "my-target-cluster-east"
sourceConnector:
config:
replication.factor: 1
offset-syncs.topic.replication.factor: 1
sync.topic.acls.enabled: "false"
heartbeatConnector:
config:
heartbeats.topic.replication.factor: 1
checkpointConnector:
config:
checkpoints.topic.replication.factor: 1
topicsPattern: ".*"
groupsPattern: ".*"
```
While the pods & the svc come up, MirrorMaker is not transferring data from
source to target topics. (mmtest - in my case, that is the only topic
created on both the clusters)
Here are the logs from the mirrormaker pod :
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Requesting metadata update for partition
mirrormaker2-cluster-offsets-7 due to error LEADER_NOT_AVAILABLE
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition
mirrormaker2-cluster-offsets-21 from 226 to epoch 226 from new metadata
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition
mirrormaker2-cluster-status-0 from 225 to epoch 225 from new metadata
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Requesting metadata update for partition
mirrormaker2-cluster-status-0 due to error LEADER_NOT_AVAILABLE
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition
mirrormaker2-cluster-status-1 from 226 to epoch 226 from new metadata
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition
mirrormaker2-cluster-status-4 from 226 to epoch 226 from new metadata
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition
mirrormaker2-cluster-status-2 from 226 to epoch 226 from new metadata
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition
mirrormaker2-cluster-status-3 from 225 to epoch 225 from new metadata
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Requesting metadata update for partition
mirrormaker2-cluster-status-3 due to error LEADER_NOT_AVAILABLE
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition mmtest-0 from 460 to
epoch 460 from new metadata (org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition mmtest-2 from 416 to
epoch 416 from new metadata (org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition mmtest-1 from 433 to
epoch 433 from new metadata (org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updating last seen epoch for partition
mirrormaker2-cluster-configs-0 from 226 to epoch 226 from new metadata
(org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Updated cluster metadata updateVersion 1597 to
MetadataCache{clusterId='yoD3YJaKTz6dvaVT2MfiOw',
nodes={1=nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1 rack:
null), 2=nossl-w-kafka-2.nossl-w-kafka-brokers.kafka.svc:9092 (id: 2 rack:
null)}, partitions=[PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-offsets-0, leader=Optional[1],
leaderEpoch=Optional[226], replicas=1, isr=1, offlineReplicas=),
PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-status-3, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-2,
leader=Optional[2], leaderEpoch=Optional[226], replicas=2, isr=2,
offlineReplicas=), PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-status-1, leader=Optional[2],
leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-offsets-4, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mmtest-0, leader=Optional[2],
leaderEpoch=Optional[460], replicas=2,1,0, isr=1,2, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-6,
leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
offlineReplicas=), PartitionMetadata(error=NONE, partition=mmtest-2,
leader=Optional[2], leaderEpoch=Optional[416], replicas=0,2,1, isr=1,2,
offlineReplicas=0), PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-offsets-8, leader=Optional[2],
leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-offsets-10, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-12,
leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
offlineReplicas=), PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-offsets-14, leader=Optional[2],
leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-15,
leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
offlineReplicas=), PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-offsets-17, leader=Optional[2],
leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-offsets-19, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-21,
leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
offlineReplicas=), PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-offsets-23, leader=Optional[2],
leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-offsets-1, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-status-4,
leader=Optional[2], leaderEpoch=Optional[226], replicas=2, isr=2,
offlineReplicas=), PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-offsets-3, leader=Optional[1],
leaderEpoch=Optional[226], replicas=1, isr=1, offlineReplicas=),
PartitionMetadata(error=NONE, partition=mmtest-1, leader=Optional[1],
leaderEpoch=Optional[433], replicas=1,0,2, isr=1,2, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-status-2,
leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
offlineReplicas=), PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-offsets-5, leader=Optional[2],
leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-status-0, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-offsets-7, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-9,
leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
offlineReplicas=), PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-offsets-11, leader=Optional[2],
leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-offsets-13, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-offsets-16, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-18,
leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
offlineReplicas=), PartitionMetadata(error=NONE,
partition=mirrormaker2-cluster-configs-0, leader=Optional[2],
leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-20,
leader=Optional[2], leaderEpoch=Optional[226], replicas=2, isr=2,
offlineReplicas=), PartitionMetadata(error=LEADER_NOT_AVAILABLE,
partition=mirrormaker2-cluster-offsets-22, leader=Optional.empty,
leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-24,
leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
offlineReplicas=)],
controller=nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1
rack: null)} (org.apache.kafka.clients.Metadata)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,666 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Received FETCH response from node 1 for request with header
RequestHeader(apiKey=FETCH, apiVersion=12,
clientId=consumer-mirrormaker2-cluster-1, correlationId=12072):
FetchResponseData(throttleTimeMs=0, errorCode=0, sessionId=1476981493,
responses=[]) (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work
Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,666 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Node 1 sent an incremental fetch response with throttleTimeMs = 0 for
session 1476981493 with 0 response partition(s), 8 implied partition(s)
(org.apache.kafka.clients.FetchSessionHandler) [KafkaBasedLog Work Thread -
mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-2 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
(id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-8 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
(id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-17 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
(id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-23 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
(id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-5 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
(id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-11 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
(id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-20 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
(id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-14 at position FetchPosition{offset=2,
offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
(id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Built incremental fetch (sessionId=1476981493, epoch=4013) for node 1.
Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of
8 partition(s) (org.apache.kafka.clients.FetchSessionHandler)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(),
implied=(mirrormaker2-cluster-offsets-2, mirrormaker2-cluster-offsets-8,
mirrormaker2-cluster-offsets-14, mirrormaker2-cluster-offsets-17,
mirrormaker2-cluster-offsets-23, mirrormaker2-cluster-offsets-5,
mirrormaker2-cluster-offsets-11, mirrormaker2-cluster-offsets-20)) to
broker 34.75.2.112:9094 (id: 1 rack: null)
(org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,667 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Sending FETCH request with header RequestHeader(apiKey=FETCH,
apiVersion=12, clientId=consumer-mirrormaker2-cluster-1,
correlationId=12075) and timeout 30000 to node 1:
FetchRequestData(clusterId=null, replicaId=-1, maxWaitMs=500, minBytes=1,
maxBytes=52428800, isolationLevel=0, sessionId=1476981493,
sessionEpoch=4013, topics=[], forgottenTopicsData=[], rackId='')
(org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread -
mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Received FETCH response from node 0 for request with header
RequestHeader(apiKey=FETCH, apiVersion=12,
clientId=consumer-mirrormaker2-cluster-1, correlationId=12073):
FetchResponseData(throttleTimeMs=0, errorCode=0, sessionId=1395933211,
responses=[]) (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work
Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Node 0 sent an incremental fetch response with throttleTimeMs = 0 for
session 1395933211 with 0 response partition(s), 9 implied partition(s)
(org.apache.kafka.clients.FetchSessionHandler) [KafkaBasedLog Work Thread -
mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-0 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
(id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-12 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
(id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-15 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
(id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-21 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
(id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-18 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
(id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-24 at position FetchPosition{offset=0,
offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
(id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-6 at position FetchPosition{offset=1,
offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
(id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-3 at position FetchPosition{offset=2,
offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
(id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Added READ_UNCOMMITTED fetch request for partition
mirrormaker2-cluster-offsets-9 at position FetchPosition{offset=1,
offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
(id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Built incremental fetch (sessionId=1395933211, epoch=4010) for node 0.
Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of
9 partition(s) (org.apache.kafka.clients.FetchSessionHandler)
[KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(),
implied=(mirrormaker2-cluster-offsets-0, mirrormaker2-cluster-offsets-6,
mirrormaker2-cluster-offsets-12, mirrormaker2-cluster-offsets-15,
mirrormaker2-cluster-offsets-21, mirrormaker2-cluster-offsets-3,
mirrormaker2-cluster-offsets-9, mirrormaker2-cluster-offsets-18,
mirrormaker2-cluster-offsets-24)) to broker 35.231.229.1:9094 (id: 0 rack:
null) (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog
Work Thread - mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,692 DEBUG [Consumer
clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
Sending FETCH request with header RequestHeader(apiKey=FETCH,
apiVersion=12, clientId=consumer-mirrormaker2-cluster-1,
correlationId=12076) and timeout 30000 to node 0:
FetchRequestData(clusterId=null, replicaId=-1, maxWaitMs=500, minBytes=1,
maxBytes=52428800, isolationLevel=0, sessionId=1395933211,
sessionEpoch=4010, topics=[], forgottenTopicsData=[], rackId='')
(org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread -
mirrormaker2-cluster-offsets]
2023-02-23 22:05:54,699 DEBUG [Worker clientId=connect-1,
groupId=mirrormaker2-cluster] Received HEARTBEAT response from node
2147483645 for request with header RequestHeader(apiKey=HEARTBEAT,
apiVersion=4, clientId=connect-1, correlationId=780):
HeartbeatResponseData(throttleTimeMs=0, errorCode=0)
(org.apache.kafka.clients.NetworkClient) [DistributedHerder-connect-1-1]
2023-02-23 22:05:54,699 DEBUG [Worker clientId=connect-1,
groupId=mirrormaker2-cluster] Received successful Heartbeat response
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
[DistributedHerder-connect-1-1]
2023-02-23 22:05:54,732 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Sending metadata request
MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
name='mirrormaker2-cluster-offsets'),
MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
name='mirrormaker2-cluster-status'),
MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='mmtest'),
MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
name='mirrormaker2-cluster-configs')], allowAutoTopicCreation=true,
includeClusterAuthorizedOperations=false,
includeTopicAuthorizedOperations=false) to node
nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1 rack: null)
(org.apache.kafka.clients.NetworkClient)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:05:54,732 DEBUG [Consumer clientId=consumer-null-5,
groupId=null] Sending METADATA request with header
RequestHeader(apiKey=METADATA, apiVersion=11, clientId=consumer-null-5,
correlationId=17039) and timeout 30000 to node 1:
MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
name='mirrormaker2-cluster-offsets'),
MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
name='mirrormaker2-cluster-status'),
MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='mmtest'),
MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
name='mirrormaker2-cluster-configs')], allowAutoTopicCreation=true,
includeClusterAuthorizedOperations=false,
includeTopicAuthorizedOperations=false)
(org.apache.kafka.clients.NetworkClient)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
```
Intermittently, I'm seeing -
org.apache.kafka.common.errors.DisconnectException
when MirrorMaker tries to connect to the broker on the source cluster, even
though MM2 is installed on the src connection.
Data in the src topic (mmtest on nossl-w, region - us-west1) is not moving
to target cluster/topic (mmtest on nossl-e, region - us-east1)
Pls note:
If i logon to the MirrorMaker pod(in cluster - nossl-w), as expected - I'm
able to access the kafka pods on the nossl-w.
Any ideas on what needs to be done to debug/resolve this issue & enable
KafkaMirrorMaker2 working ?
tia!
