Summary
I'm migrating from an on-prem Apache Kafka cluster (“onpre-kafka”) to an
AWS MSK cluster using MirrorMaker 2 (MM2).
After enabling one-way replication (onpre-kafka → MSK), I plan to cut
consumers over to MSK while producers still write to onpre-kafka.
I’d like to confirm the exact offset semantics.
Topology
- Source: onpre-kafka
- Target: MSK
- Replication: MM2 (MirrorSourceConnector + MirrorHeartbeatConnector +
MirrorCheckpointConnector)
- Direction: source → target (one-way)
- Producers: continue writing to onpre-kafka
- Consumers: switch from onpre-kafka to MSK after offset translation
Scenario
1. MM2 replicates topics from onpre-kafka to MSK.
2. I start consuming from MSK (consumers move), while producers still
write to onpre-kafka.
3. MSK consumer offsets advance (e.g., 100 → 102). onpre-kafka consumer
group offsets remain at 100.
Questions
- Are there any circumstances where the target (MSK) offsets could be
overwritten by smaller source offsets while consumers are already running
on MSK?
- I assume that this kind of behavior is not intended by design.
- Will new messages written to onpre-kafka continue to replicate to MSK
via MM2, and remain readable by consumers from MSK?
- I assume that consumers can keep consuming new messages from MSK
that are replicated via MM2.
Thanks in advance!
