Hey all,
I've experimented with a solution and opened a PR:
https://github.com/apache/kafka/pull/13429
I'd appreciate any comments you might have, and please ask questions about
the core algorithm. I've done my best to explain the index in comments, and
you can see how an arbitrary state is constr
Thanks all for the discussion!
Mickael,
I agree. I think that while someone may appreciate the current high
availability, everyone expects at-least-once delivery from this feature.
Not meeting the expectation for at-least-once delivery may be more damaging
than excessive re-delivery in a failure
"it becomes impossible to ensure that MM2 is
able to sync offsets from consumer groups that are behind the last-synced
offset emitted by MirrorSourceTask."
That's sorta true. The idea is that offset syncs are exceptionally rare.
They basically occur when the tasks start or restart and that should
Hi Greg,
I agree with others that a KIP isn't necessarily required as long as we
don't alter public interface.
Reading between the lines (please correct me if I'm wrong), it seems like
part of the concern is that the design of the offset syncs topic (parts of
which are counted as part of the publ
Hi Greg,
Thanks for looking into this issue!
To me the current behavior is just buggy. I don't see how anyone could
rely on incorrect offset translation, at this point it's pretty much
like picking a random offset when consumers start on the target
cluster.
Apart if we have to introduce new conf
Luke,
Thanks for leaving your thoughts.
I agree that this does not directly change an API, and doesn't fall within
the typical KIP guidelines. These proposals would change the behavior of
translation within the bounds of the current API, and I wasn't sure whether
this fell under the KIP process.
Hi Greg,
Thanks for your summary.
I'm not very familiar with MM2, actually, but I'd like to give my answer to
some of the questions below:
Q1. Would an improvement to allow translation from earlier in the topic be
reasonable to propose in a KIP?
--> I checked the proposed 5 solutions in KAFKA-146
Hey all!
I realized that the information above is a bit in-the-weeds, and I think a
re-framing of the situation might be necessary.
Since the release of MM2, offsets translation has been limited to only
performing translation ahead of the most recent offset sync. This
limitation appears to have w
Hi all,
Recently, we've been experimenting with using MM2 to mirror topics that
were populated by transactional producers. We've noticed that MM2
replicates records but not transaction markers, causing certain offsets to
appear in the source topic but not destination topic. These behaviors can
als
