Hi Mickael, Good point, I renamed the KIP and this thread: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1098%3A+Reverse+Checkpointing+in+MirrorMaker Thank you, Daniel
Mickael Maison <mickael.mai...@gmail.com> ezt írta (időpont: 2024. okt. 21., H, 15:22): > Hi Daniel, > > I've not had time to take a close look at the KIP but my initial > feedback would be to adjust the name to make it clear it's about > MirrorMaker. > The word "checkpoint" has several meanings in Kafka and from the > current KIP name it's not clear if it's about KRaft, Streams or > Connect. > > Thanks, > Mickael > > On Mon, Oct 21, 2024 at 2:55 PM Dániel Urbán <urb.dani...@gmail.com> > wrote: > > > > Hi Viktor, > > > > Thank you for the comments! > > > > SVV1: I think the feature has some performance implications. If the > reverse > > checkpointing is enabled, task startup will be possibly slower, since it > > will need to consume from a second offset-syncs topic; and it will also > use > > more memory, to keep the second offset-sync history. Additionally, it is > > also possible to have an offset-syncs topic present without an actual, > > opposite flow being active - I think only users can tell if the reverse > > checkpointing should be active, and they should be the one opting in for > > the higher resource usage. > > > > SVV2: I mention the DefaultReplicationPolicy to provide examples. I don't > > think it is required. The actual requirement related to the > > ReplicationPolicy is that the policy should be able to correctly tell > which > > topic was replicated from which cluster. Because of this, > > IdentityReplicationPolicy would not work, but DefaultReplicationPolicy, > or > > any other ReplicationPolicy implementations with a correctly implemented > > "topicSource" method should work. I will make an explicit note of this in > > the KIP. > > > > Thank you, > > Daniel > > > > Viktor Somogyi-Vass <viktor.somo...@cloudera.com.invalid> ezt írta > > (időpont: 2024. okt. 18., Pén 17:28): > > > > > Hey Dan, > > > > > > I think this is a very useful idea. Two questions: > > > > > > SVV1: Do you think we need the feature flag at all? I know that not > having > > > this flag may technically render the KIP unnecessary (however it may > still > > > be useful to discuss this topic and create a concensus). As you wrote > in > > > the KIP, we may be able to look up the target and source topics and if > we > > > can do this, we can probably detect if the replication is one-way or > > > prefixless (identity). So that means we don't need this flag to control > > > when we want to use this. Then it is really just there to act as > something > > > that can turn the feature on and off if needed, but I'm not really > sure if > > > there is a great risk in just enabling this by default. If we really > just > > > turn back the B -> A checkpoints and save them in the A -> B, then > maybe > > > it's not too risky and users would get this immediately by just > upgrading. > > > > > > SVV2: You write that we need DefaultReplicationPolicy to use this > feature, > > > but most of the functionality is there on interface level in > > > ReplicationPolicy. Is there anything that is missing from there and if > so, > > > what do you think about pulling it into the interface? If this > improvement > > > only works with the default replication policy, then it's somewhat > limiting > > > as users may have their own policy for various reasons, but if we can > make > > > it work on the interface level, then we could provide this feature to > > > everyone. Of course there can be replication policies like the > identity one > > > that by design disallows this feature, but for that, see my previous > point. > > > > > > Best, > > > Viktor > > > > > > On Fri, Oct 18, 2024 at 3:30 PM Dániel Urbán <urb.dani...@gmail.com> > > > wrote: > > > > > > > Hi everyone, > > > > > > > > I'd like to start the discussion on KIP-1098: Reverse Checkpointing ( > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1098%3A+Reverse+Checkpointing > > > > ) > > > > which aims to minimize message reprocessing for consumers in > failbacks. > > > > > > > > TIA, > > > > Daniel > > > > > > > >
