Hi Denny,
If you use mirrormaker to replicate data between clusters then it
needs access to all brokers.
There is also a clear reason for this, Mirrormaker acts as a normal
Kafka client when connecting to Kafka clusters.
Mirrormaker uses the normal produce and consume protocol, which means
that a producer needs to be able to contact the broker that is the
leader of the partition.
The consume functionality needs to connect to the leader, or an in
sync replica of the partition to retrieve the records for replication.
The basic rule for network access is that a Kafka client needs network
access to all advertised ports belonging to the same Kafka broker
listener (SASL, SSL, EXTERNAL-XX).
Kind regards,
Richard Bosch
Developer Advocate
Axual BV
E : richard.bo...@axual.com
M : +31 6 11 850 846
W : www.axual.com
On Mon, Oct 2, 2023 at 2:25 PM Denny Fuchs
<denny.fu...@payabl.com.invalid> wrote:
>
> hi,
>
> I try to use mirror maker to replicate the data, and I have one trivial
> question:
>
> Do I need access to / add **all** Kafka Brokers ?
>
> I have a 3.1 Cluster with 7 brokers and a new 3.5 with 5 brokers and I
> want to migrate from the 3.1 (Zoo) to 3.5 (KRaft). Both clusters are in
> different networks with a lot of firewalls and we allowed only access to
> one of the old clusters:
>
> My mirror-mager is on the end.
>
> The problem is .. that if I start the sync .. it seems to be stuck
> somewhere:
>
>
> ======================================
>
> (org.apache.kafka.connect.mirror.MirrorHeartbeatConfig:369)
> [2023-10-02 14:07:50,091] INFO [MirrorHeartbeatConnector|task-0]
> WorkerSourceTask{id=MirrorHeartbeatConnector-0} Source task finished
> initialization and start
> (org.apache.kafka.connect.runtime.AbstractWorkerSource
> Task:275)
> [2023-10-02 14:07:50,113] INFO [MirrorHeartbeatConnector|task-0]
> [Producer clientId=connector-producer-MirrorHeartbeatConnector-0]
> Cluster ID: Rnpnd4EcRBeWo8vUrWlOIQ (org.apache.kafka.clients.Metadata:287)
> [2023-10-02 14:07:50,238] INFO [MirrorSourceConnector|worker]
> [AdminClient
> clientId=FC-R02->FC-R01|MirrorSourceConnector|replication-source-admin]
> Disconnecting from node 7 due to socket connection setup timeout.
> The timeout value is 11336 ms. (org.apache.kafka.clients.NetworkClient:858)
> [2023-10-02 14:07:50,828] INFO [MirrorCheckpointConnector|worker]
> [AdminClient
> clientId=FC-R02->FC-R01|MirrorCheckpointConnector|checkpoint-source-admin]
> Disconnecting from node 7 due to socket connection setup ti
> meout. The timeout value is 11943 ms.
> (org.apache.kafka.clients.NetworkClient:858)
> [2023-10-02 14:07:58,045] INFO [MirrorCheckpointConnector|worker]
> [AdminClient
> clientId=FC-R02->FC-R01|MirrorCheckpointConnector|checkpoint-source-admin]
> Disconnecting from node 6 due to socket connection setup ti
> meout. The timeout value is 8916 ms.
> (org.apache.kafka.clients.NetworkClient:858)
> [2023-10-02 14:08:00,271] INFO [MirrorSourceConnector|worker]
> [AdminClient
> clientId=FC-R02->FC-R01|MirrorSourceConnector|replication-source-admin]
> Disconnecting from node 6 due to socket connection setup timeout.
> The timeout value is 10018 ms. (org.apache.kafka.clients.NetworkClient:858)
> [2023-10-02 14:08:00,359] INFO [MirrorSourceConnector|worker]
> [AdminClient
> clientId=FC-R02->FC-R01|MirrorSourceConnector|replication-source-admin]
> Disconnecting from node 7 due to socket connection setup timeout.
> The timeout value is 11266 ms. (org.apache.kafka.clients.NetworkClient:858)
> [2023-10-02 14:08:00,440] INFO [MirrorSourceConnector|worker] loading
> initial set of topic-partitions took 11359 ms
> (org.apache.kafka.connect.mirror.Scheduler:95)
> [2023-10-02 14:08:00,448] INFO [MirrorSourceConnector|worker] creating
> downstream topic-partitions took 7 ms
> (org.apache.kafka.connect.mirror.Scheduler:95)
> [2023-10-02 14:08:00,454] INFO [MirrorSourceConnector|worker] refreshing
> known target topics took 5 ms
> (org.apache.kafka.connect.mirror.Scheduler:95)
> [2023-10-02 14:08:00,457] INFO [MirrorSourceConnector|worker] Started
> MirrorSourceConnector with 0 topic-partitions.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:172)
> [2023-10-02 14:08:00,458] INFO [MirrorSourceConnector|worker] Starting
> MirrorSourceConnector took 11510 ms.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector:173)
> [2023-10-02 14:08:00,461] INFO SourceConnectorConfig values:
> config.action.reload = restart
> connector.class =
> org.apache.kafka.connect.mirror.MirrorSourceConnector
>
> ======================================
>
> So, If I see it correct, it tries to connect to the other nodes .. which
> is not allowed yet .. right ?
>
> cu denny
>
>
> #######################
>
> clusters = FC-R02, FC-R01
>
> FC-R02.bootstrap.servers = fc-r02-kafka-01.example.com:9093
> FC-R01.bootstrap.servers = fc-r01-kafka-01.example.com:9092
>
> FC-R02->FC-R01.enabled = true
> FC-R01->FC-R02.enabled = false
>
> FC-R02->FC-R01.topics =
> "(blacklist_transaction.*|foo_transaction.*|chargebacks.*)"
> replication.factor=4
> sync.topic.acls.enabled=false
>
> checkpoints.topic.replication.factor=3
> heartbeats.topic.replication.factor=3
> offset-syncs.topic.replication.factor=3
>
> offset.storage.replication.factor=3
> status.storage.replication.factor=3
> config.storage.replication.factor=3
>
> security.protocol: SASL_SSL
> ssl.truststore.type = PEM
> ssl.truststore.location=/etc/ssl/certs/ca-certificates.crt
> ssl.enabled.protocols=TLSv1.2
> security.protocol = SASL_SSL
>
> FC-R01.sasl.mechanism=PLAIN
> FC-R01.ssl.endpoint.identification.algorithm=https
> FC-R01.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
> required \
> serviceName='kafka' \
> username='kafka' \
> password='secret';
>
> FC-R02.security.protocol: SSL
> FC-R02.sasl.mechanism=PLAIN
> FC-R02.ssl.endpoint.identification.algorithm=https
> FC-R02.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
> required \
> serviceName='kafka' \
> username='kafka' \
> password='secret';
>
> ==========================================
>
>
>
>
