Hi, I agree with Rajini, I'm not sure it's worth re-adding that logic. If the migration/upgrade use case is important for you, targeting 3.0+ should already cover a large portion of outdated clusters.
Thanks, Mickael On Mon, Nov 2, 2026 at 7:41 PM Rajini Sivaram <[email protected]> wrote: > > Hi Federico, > > Thanks for the response. I think we removed the old code in > AbstractFetcherThread to reduce complexity and maintenance cost. It seems > odd that we would bring back all that complexity in a subclass. Not sure if > there are users who are still on old broker versions like 2.1 (unlike > clients). I would have thought that it is sufficient to support 3.0+. It > will be good to understand how much compatibility we expect to maintain for > 5.0 and beyond. Let's see what others think too. > > Regards, > > Rajini > > > On Mon, Nov 2, 2026 at 5:58 PM Federico Valeri <[email protected]> wrote: > > > Hi Rajini, thanks for letting us know. The schema is still there and we > > think it is important to keep support for 2.1 to help old Kafka versions > > users to migrate to latest version, so we would like to reintroduce that > > logic just for Cluster Mirroring. Wdyt? > > > > On Mon, Nov 2, 2026 at 2:39 AM Rajini Sivaram <[email protected]> > > wrote: > > > > > Hi Federico, > > > > > > KIP-1279 currently states that cluster mirroring source clusters may be > > > version >= 2.1. We have been removing support for older metadata versions > > > and > > > > > > > > https://github.com/apache/kafka/commit/b234a3a667055d867319c99fd8eca72200c780a8 > > > removes the explicit truncating phase that sends > > > OffsetForLeaderEpochRequest from AbstractFetcherThread. So inter-broker > > > replication now relies entirely on truncate-on-fetch. Are we planning to > > > introduce OffsetForLeaderEpochRequest in the cluster mirroring fetcher > > > because we think there is a requirement to migrate from older clusters? > > Or > > > are we planning to keep the code simpler and support newer clusters only? > > > > > > Thank you, > > > > > > Rajini > > > > > > > > > On Mon, Sep 7, 2026 at 12:09 PM Rajini Sivaram <[email protected]> > > > wrote: > > > > > > > Thanks Federico! The solutions for both RS25 and RS27 sound good. > > > > > > > > Regards, > > > > > > > > Rajini > > > > > > > > On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri <[email protected]> > > > > wrote: > > > > > > > >> Hi Rajini, > > > >> > > > >> >Not sure about RS25, but you mentioned you are looking into it again, > > > so > > > >> will wait for the response after that. > > > >> > > > >> The clamping first attempts to read LSO and LEO from the local > > > >> ReplicaManager, which is an in-memory lookup against the local > > > >> UnifiedLog instance. This works for any partition the coordinator > > > >> broker happens to host a replica for. For partitions without a local > > > >> replica, the coordinator falls back to the Admin listOffsets API, > > > >> which internally routes the request to the partition leader and > > > >> returns the offsets. We resolve all mirror partition offsets once at > > > >> the beginning of each sync cycle, then reuse them for every group. Not > > > >> one call per partition or per group. > > > >> > > > >> KIP Updated. > > > >> > > > >> > For RS27, not supporting unclean leader election for older clients > > may > > > >> be ok unless the fix is straightforward. > > > >> > > > >> We think it is possible to support old versions with the following > > > >> strategy: > > > >> > > > >> We detect the source cluster's fetch version dynamically. On the first > > > >> fetch request, the fetcher thread always includes lastFetchedEpoch in > > > >> the request builder. The NetworkClient then queries ApiVersions from > > > >> the source cluster and selects the latestUsableVersion. Two outcomes > > > >> are possible: > > > >> > > > >> 1. Source supports fetch v12+: the request succeeds with > > > >> lastFetchedEpoch, and truncation proceeds via the truncation-on-fetch > > > >> path. > > > >> 2. Source does not support fetch v12: NetworkClient throws > > > >> UnsupportedVersionException. We catch it, disable truncation-on-fetch, > > > >> and fall back to the OffsetsForLeaderEpochRequest path. > > > >> > > > >> This handles both old and new source clusters, including live upgrades > > > >> and downgrades. > > > >> > > > >> Unclean leader election handling: > > > >> > > > >> Previously we only handled unclean leader election (ULE) in the > > > >> truncation-on-fetch case. To support older clusters via > > > >> OffsetsForLeaderEpochRequest, we apply the same principle: log > > > >> truncation should never happen on the destination cluster during > > > >> mirroring because we only fetch committed data. If an > > > >> OffsetsForLeaderEpochResponse does trigger truncation, that truncation > > > >> must originate from a ULE on the source cluster. In that case, we > > > >> transition to the ULE_RECOVERY state and wait for all replicas to > > > >> catch up (when mirror.unclean.leader.election.enable=true). > > > >> > > > >> To guarantee that all destination truncations come from source ULEs, > > > >> we store the "last mirror offset and last mirror epoch" in the LME > > > >> record. Originally, the LME phase one truncated records with epoch > > > >> greater than the LME epoch, then phase two relied on normal > > > >> replication to handle log divergence. The problem: some truncations > > > >> are side effects of mirroring setup (e.g. reverse mirroring), not > > > >> ULEs. > > > >> > > > >> Example (cluster A mirrors from cluster B): > > > >> > > > >> Cluster A (dest) <---- Cluster B (source) > > > >> Offset 0: A (LE: 0) Offset 0: A (LE: 0) > > > >> Offset 1: A (LE: 0) Offset 1: A (LE: 0) > > > >> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10) > > > >> > > > >> When cluster A receives the first OffsetsForLeaderEpochResponse or > > > >> fetch response from cluster B, offset 2 on cluster A gets truncated. > > > >> This truncation is not caused by a ULE. Without the last mirror offset > > > >> and epoch in the LME record, we cannot distinguish it from a real ULE. > > > >> By storing these values in the LME record, the LOG_ALIGNMENT state > > > >> truncates only records where: record offset <= last mirror offset, AND > > > >> record epoch <= last mirror epoch. This correctly truncates offset 2 > > > >> in the example above during LOG_ALIGNMENT. After alignment completes, > > > >> any subsequent truncation on the destination is guaranteed to > > > >> originate from a source cluster ULE. > > > >> > > > >> Let us know if you are good with this solution. KIP update will > > follow. > > > >> > > > >> Thanks > > > >> Fede > > > >> > > > > > > > > >
