Thank you, Federico! On Tue, Nov 3, 2026 at 4:50 PM Federico Valeri <[email protected]> wrote:
> Hi Rajini, thanks for the info. KIP updated. > > Regards > Fede > > On Tue, Nov 3, 2026 at 2:11 AM Rajini Sivaram <[email protected]> > wrote: > > > > Hi Federico, > > > > Truncation-on-fetch was introduced in 2.7, but had edge cases, which were > > fixed later. Commits like fcc7c2de391 and 8e211eb72f9 didn't go into 2.7. > > There were also other fixes like KAFKA-12996 that came later than 2.8.0, > > though that one is in 2.8.1. There may be others too. 3.0.x seems like a > > safe option where we can expect truncation-on-fetch to be reliable. > > > > Regards, > > > > Rajini > > > > > > On Tue, Nov 3, 2026 at 5:40 AM Federico Valeri <[email protected]> > wrote: > > > > > Hi Rajini and Michael. Maintenance is a fair point and we agree on > > > that. Just one question for my understanding, why 3.0 and not 2.7 that > > > is where truncation-on-fetch was introduced? Do you see an any > > > specific problem related to the Fetch protocol? > > > > > > > > > On Mon, Nov 2, 2026 at 11:46 AM Mickael Maison < > [email protected]> > > > wrote: > > > > > > > > Hi, > > > > > > > > I agree with Rajini, I'm not sure it's worth re-adding that logic. > > > > If the migration/upgrade use case is important for you, targeting > 3.0+ > > > > should already cover a large portion of outdated clusters. > > > > > > > > Thanks, > > > > Mickael > > > > > > > > > > > > On Mon, Nov 2, 2026 at 7:41 PM Rajini Sivaram < > [email protected]> > > > wrote: > > > > > > > > > > Hi Federico, > > > > > > > > > > Thanks for the response. I think we removed the old code in > > > > > AbstractFetcherThread to reduce complexity and maintenance cost. It > > > seems > > > > > odd that we would bring back all that complexity in a subclass. Not > > > sure if > > > > > there are users who are still on old broker versions like 2.1 > (unlike > > > > > clients). I would have thought that it is sufficient to support > 3.0+. > > > It > > > > > will be good to understand how much compatibility we expect to > > > maintain for > > > > > 5.0 and beyond. Let's see what others think too. > > > > > > > > > > Regards, > > > > > > > > > > Rajini > > > > > > > > > > > > > > > On Mon, Nov 2, 2026 at 5:58 PM Federico Valeri < > [email protected]> > > > wrote: > > > > > > > > > > > Hi Rajini, thanks for letting us know. The schema is still there > and > > > we > > > > > > think it is important to keep support for 2.1 to help old Kafka > > > versions > > > > > > users to migrate to latest version, so we would like to > reintroduce > > > that > > > > > > logic just for Cluster Mirroring. Wdyt? > > > > > > > > > > > > On Mon, Nov 2, 2026 at 2:39 AM Rajini Sivaram < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Federico, > > > > > > > > > > > > > > KIP-1279 currently states that cluster mirroring source > clusters > > > may be > > > > > > > version >= 2.1. We have been removing support for older > metadata > > > versions > > > > > > > and > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/kafka/commit/b234a3a667055d867319c99fd8eca72200c780a8 > > > > > > > removes the explicit truncating phase that sends > > > > > > > OffsetForLeaderEpochRequest from AbstractFetcherThread. So > > > inter-broker > > > > > > > replication now relies entirely on truncate-on-fetch. Are we > > > planning to > > > > > > > introduce OffsetForLeaderEpochRequest in the cluster mirroring > > > fetcher > > > > > > > because we think there is a requirement to migrate from older > > > clusters? > > > > > > Or > > > > > > > are we planning to keep the code simpler and support newer > > > clusters only? > > > > > > > > > > > > > > Thank you, > > > > > > > > > > > > > > Rajini > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 7, 2026 at 12:09 PM Rajini Sivaram < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks Federico! The solutions for both RS25 and RS27 sound > good. > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > Rajini > > > > > > > > > > > > > > > > On Mon, Aug 31, 2026 at 5:12 PM Federico Valeri < > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > >> Hi Rajini, > > > > > > > >> > > > > > > > >> >Not sure about RS25, but you mentioned you are looking > into it > > > again, > > > > > > > so > > > > > > > >> will wait for the response after that. > > > > > > > >> > > > > > > > >> The clamping first attempts to read LSO and LEO from the > local > > > > > > > >> ReplicaManager, which is an in-memory lookup against the > local > > > > > > > >> UnifiedLog instance. This works for any partition the > > > coordinator > > > > > > > >> broker happens to host a replica for. For partitions > without a > > > local > > > > > > > >> replica, the coordinator falls back to the Admin listOffsets > > > API, > > > > > > > >> which internally routes the request to the partition leader > and > > > > > > > >> returns the offsets. We resolve all mirror partition offsets > > > once at > > > > > > > >> the beginning of each sync cycle, then reuse them for every > > > group. Not > > > > > > > >> one call per partition or per group. > > > > > > > >> > > > > > > > >> KIP Updated. > > > > > > > >> > > > > > > > >> > For RS27, not supporting unclean leader election for older > > > clients > > > > > > may > > > > > > > >> be ok unless the fix is straightforward. > > > > > > > >> > > > > > > > >> We think it is possible to support old versions with the > > > following > > > > > > > >> strategy: > > > > > > > >> > > > > > > > >> We detect the source cluster's fetch version dynamically. On > > > the first > > > > > > > >> fetch request, the fetcher thread always includes > > > lastFetchedEpoch in > > > > > > > >> the request builder. The NetworkClient then queries > ApiVersions > > > from > > > > > > > >> the source cluster and selects the latestUsableVersion. Two > > > outcomes > > > > > > > >> are possible: > > > > > > > >> > > > > > > > >> 1. Source supports fetch v12+: the request succeeds with > > > > > > > >> lastFetchedEpoch, and truncation proceeds via the > > > truncation-on-fetch > > > > > > > >> path. > > > > > > > >> 2. Source does not support fetch v12: NetworkClient throws > > > > > > > >> UnsupportedVersionException. We catch it, disable > > > truncation-on-fetch, > > > > > > > >> and fall back to the OffsetsForLeaderEpochRequest path. > > > > > > > >> > > > > > > > >> This handles both old and new source clusters, including > live > > > upgrades > > > > > > > >> and downgrades. > > > > > > > >> > > > > > > > >> Unclean leader election handling: > > > > > > > >> > > > > > > > >> Previously we only handled unclean leader election (ULE) in > the > > > > > > > >> truncation-on-fetch case. To support older clusters via > > > > > > > >> OffsetsForLeaderEpochRequest, we apply the same principle: > log > > > > > > > >> truncation should never happen on the destination cluster > during > > > > > > > >> mirroring because we only fetch committed data. If an > > > > > > > >> OffsetsForLeaderEpochResponse does trigger truncation, that > > > truncation > > > > > > > >> must originate from a ULE on the source cluster. In that > case, > > > we > > > > > > > >> transition to the ULE_RECOVERY state and wait for all > replicas > > > to > > > > > > > >> catch up (when mirror.unclean.leader.election.enable=true). > > > > > > > >> > > > > > > > >> To guarantee that all destination truncations come from > source > > > ULEs, > > > > > > > >> we store the "last mirror offset and last mirror epoch" in > the > > > LME > > > > > > > >> record. Originally, the LME phase one truncated records with > > > epoch > > > > > > > >> greater than the LME epoch, then phase two relied on normal > > > > > > > >> replication to handle log divergence. The problem: some > > > truncations > > > > > > > >> are side effects of mirroring setup (e.g. reverse > mirroring), > > > not > > > > > > > >> ULEs. > > > > > > > >> > > > > > > > >> Example (cluster A mirrors from cluster B): > > > > > > > >> > > > > > > > >> Cluster A (dest) <---- Cluster B (source) > > > > > > > >> Offset 0: A (LE: 0) Offset 0: A (LE: 0) > > > > > > > >> Offset 1: A (LE: 0) Offset 1: A (LE: 0) > > > > > > > >> Offset 2: A (LE: 0) Offset 2: PID_RESET (LE: 10) > > > > > > > >> > > > > > > > >> When cluster A receives the first > OffsetsForLeaderEpochResponse > > > or > > > > > > > >> fetch response from cluster B, offset 2 on cluster A gets > > > truncated. > > > > > > > >> This truncation is not caused by a ULE. Without the last > mirror > > > offset > > > > > > > >> and epoch in the LME record, we cannot distinguish it from a > > > real ULE. > > > > > > > >> By storing these values in the LME record, the LOG_ALIGNMENT > > > state > > > > > > > >> truncates only records where: record offset <= last mirror > > > offset, AND > > > > > > > >> record epoch <= last mirror epoch. This correctly truncates > > > offset 2 > > > > > > > >> in the example above during LOG_ALIGNMENT. After alignment > > > completes, > > > > > > > >> any subsequent truncation on the destination is guaranteed > to > > > > > > > >> originate from a source cluster ULE. > > > > > > > >> > > > > > > > >> Let us know if you are good with this solution. KIP update > will > > > > > > follow. > > > > > > > >> > > > > > > > >> Thanks > > > > > > > >> Fede > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
