[
https://issues.apache.org/jira/browse/KAFKA-17784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889213#comment-17889213
]
George Yang commented on KAFKA-17784:
-------------------------------------
Thank you [~gharris1727].
The error indicates that the topic used for storing offset information
(mm2-offsets.idcb.internal) does not have the correct cleanup policy. I'm not
entirely sure if this theory is accurate, but it seems that Kafka MirrorMaker 2
requires the topic to have a cleanup.policy=compact setting to ensure that old
offset information isn't deleted and data consistency is maintained. If so, I
believe I should set the cleanup policy for the topic separately, as follows:
{code:java}
kafka-configs.sh --bootstrap-server 192.168.2.147:13399 --entity-type topics
--entity-name mm2-offsets.idcb.internal --alter --add-config
cleanup.policy=compact{code}
However, I have set the broker configuration log.cleanup.policy=delete. Will
this conflict with the above topic-specific setting?
Additionally, is there another way to set cleanup.policy=compact without using
the kafka-configs.sh command, such as configuring it directly in the
configuration file?
> Mirror Maker2 Pod CrashLoopbackOff
> ----------------------------------
>
> Key: KAFKA-17784
> URL: https://issues.apache.org/jira/browse/KAFKA-17784
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.7.1
> Reporter: George Yang
> Priority: Blocker
> Attachments: connect-1.log
>
>
> When I use k8s deployment with mirrormaker v3.7.1, and deploy one kafka node
> in each data center, I always got the crashloopbackoff error, please see the
> attachment.
>
> The configuration of mirrormaker is:
> ```configuration
> clusters = idca, idcb
> idca.bootstrap.servers = 192.168.2.146:13399
> idcb.bootstrap.servers = 192.168.2.147:13399
> idca->idcb.enabled = true
> idca->idcb.topics = .*
> idcb->idca.enabled = true
> idcb->idca.topics = .*
> replication.factor=1
> tasks.max=6
> emit.checkpoints.interval.seconds=5
> idca.producer.acks=all
> idca.producer.batch.size=50000
> idca.consumer.auto.offset.reset=latest
> idcb.consumer.auto.offset.reset=latest
> idca.consumer.max.poll.interval.ms=20000
> idcb.consumer.max.poll.interval.ms=20000
> refresh.topics.enabled=true
> refresh.topics.interval.seconds=5
> refresh.groups.enabled=true
> refresh.groups.interval.seconds=5
> dedicated.mode.enable.internal.rest = true
> idca.scheduled.rebalance.max.delay.ms=20000
> idcb.scheduled.rebalance.max.delay.ms=20000
> checkpoints.topic.replication.factor=3
> heartbeats.topic.replication.factor=3
> offset-syncs.topic.replication.factor=3
> offset.storage.replication.factor=3
> status.storage.replication.factor=3
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
