Hi Ryanne, thank you for your response.
1) Yes, right - I've missed the condition that is part of the PartitionState class. Thanks for pointing me. :) 2) Ok, how should I go on there? Shall I create a ticket or a KIP or even both? From my point of view this is not a major change, but for people relying on the fact that those topics are always created this may look different. Kind regards Georg Friedrich -----Original Message----- From: Ryanne Dolan <ryannedo...@gmail.com> Sent: Wednesday, March 17, 2021 4:57 AM To: dev <dev@kafka.apache.org> Subject: Re: MirrorMaker 2.0 - Offset Sync - Questions/Improvements Georg, sorry for the delay, but hopefully I can still help. 1) I think the detail you're missing is that the offset syncs are very sparse. Normally, you only get a new sync when the Connector first starts running. You are right that it is possible for a consumer to lag behind the most recent offset sync, but that will be a rare, transient condition, e.g. when the Connector first starts running. 2) I think you are right -- disabling checkpoints probably should also prevent those topics from being created. I'd support that change. Ryanne On Fri, Feb 26, 2021, 4:24 PM Georg Friedrich <georg.friedr...@webfleet.com> wrote: > Hi, > > recently I've started to look deeper into the code of MirrorMaker 2.0 > and was faced with some confusing details. Maybe you can point me into > a right direction here. > > > * The line at > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith > ub.com%2Fapache%2Fkafka%2Fblob%2F02226fa090513882b9229ac834fd493d71ae6 > d96%2Fconnect%2Fmirror%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkafka%2Fco > nnect%2Fmirror%2FOffsetSyncStore.java%23L52&data=04%7C01%7Cgeorg.f > riedrich%40webfleet.com%7Cf53651fc33834e4a793d08d8e8f8c563%7Ce648a6341 > 151497c97970f975bddecc0%7C0%7C0%7C637515502445685286%7CUnknown%7CTWFpb > GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 > %3D%7C1000&sdata=D3%2BdZuT3d7wvqLN%2F3q5zIi07iwI7ZO1doxVq2NSHjWU%3 > D&reserved=0 checks whether the offsets that get translated are > smaller than the last offset sync. > If this is the case, no translation happens. But I'm confused here: > Isn't this a potential issue? What if some consumers are slow in > regards to processing messages from Kafka and fall back behand the > offset sync process of the MirrorMaker. > In this case the MirrorMaker would stop to translate any offsets. Do I > miss something here or is this really broken? > * I'm wondering: One is able to deactivate emitting checkpoints to the > target cluster. But when this happens, the offset sync topic is still > written to the source cluster. Why is that? As far as I can see the > only consumer of the offset sync topic is the checkpoint connector. So > one could also deactivate the whole offset sync production entirely > when disabling emitting checkpoints. Or is there again something that > I miss? If not, is this worth a KIP? > > Thanks in advance for your answers and help. > > Kind regards > Georg Friedrich >
