Hi Daniel, I've not had time to take a close look at the KIP but my initial feedback would be to adjust the name to make it clear it's about MirrorMaker. The word "checkpoint" has several meanings in Kafka and from the current KIP name it's not clear if it's about KRaft, Streams or Connect.
Thanks, Mickael On Mon, Oct 21, 2024 at 2:55 PM Dániel Urbán <urb.dani...@gmail.com> wrote: > > Hi Viktor, > > Thank you for the comments! > > SVV1: I think the feature has some performance implications. If the reverse > checkpointing is enabled, task startup will be possibly slower, since it > will need to consume from a second offset-syncs topic; and it will also use > more memory, to keep the second offset-sync history. Additionally, it is > also possible to have an offset-syncs topic present without an actual, > opposite flow being active - I think only users can tell if the reverse > checkpointing should be active, and they should be the one opting in for > the higher resource usage. > > SVV2: I mention the DefaultReplicationPolicy to provide examples. I don't > think it is required. The actual requirement related to the > ReplicationPolicy is that the policy should be able to correctly tell which > topic was replicated from which cluster. Because of this, > IdentityReplicationPolicy would not work, but DefaultReplicationPolicy, or > any other ReplicationPolicy implementations with a correctly implemented > "topicSource" method should work. I will make an explicit note of this in > the KIP. > > Thank you, > Daniel > > Viktor Somogyi-Vass <viktor.somo...@cloudera.com.invalid> ezt írta > (időpont: 2024. okt. 18., Pén 17:28): > > > Hey Dan, > > > > I think this is a very useful idea. Two questions: > > > > SVV1: Do you think we need the feature flag at all? I know that not having > > this flag may technically render the KIP unnecessary (however it may still > > be useful to discuss this topic and create a concensus). As you wrote in > > the KIP, we may be able to look up the target and source topics and if we > > can do this, we can probably detect if the replication is one-way or > > prefixless (identity). So that means we don't need this flag to control > > when we want to use this. Then it is really just there to act as something > > that can turn the feature on and off if needed, but I'm not really sure if > > there is a great risk in just enabling this by default. If we really just > > turn back the B -> A checkpoints and save them in the A -> B, then maybe > > it's not too risky and users would get this immediately by just upgrading. > > > > SVV2: You write that we need DefaultReplicationPolicy to use this feature, > > but most of the functionality is there on interface level in > > ReplicationPolicy. Is there anything that is missing from there and if so, > > what do you think about pulling it into the interface? If this improvement > > only works with the default replication policy, then it's somewhat limiting > > as users may have their own policy for various reasons, but if we can make > > it work on the interface level, then we could provide this feature to > > everyone. Of course there can be replication policies like the identity one > > that by design disallows this feature, but for that, see my previous point. > > > > Best, > > Viktor > > > > On Fri, Oct 18, 2024 at 3:30 PM Dániel Urbán <urb.dani...@gmail.com> > > wrote: > > > > > Hi everyone, > > > > > > I'd like to start the discussion on KIP-1098: Reverse Checkpointing ( > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1098%3A+Reverse+Checkpointing > > > ) > > > which aims to minimize message reprocessing for consumers in failbacks. > > > > > > TIA, > > > Daniel > > > > >
