Hi all, I have updated the KIP based on the feedback so far:
- Compatibility: added a "Disk sizing" section. For backfill data, local disk grows by about local.retention.ms * write rate per partition replica, with local.retention.bytes as the way to cap it. - Proposed Changes: clarified where the upload time comes from, and that local time-based deletion pauses while the remote log metadata cache is initializing, with no fallback to record timestamps. - Rejected Alternatives: added min(record timestamp, upload time). Please take a look and let me know if there are further comments. Thanks, Murali On Fri, Nov 13, 2026 at 10:29 PM Muralidhar Basani < [email protected]> wrote: > Hi Viktor, > > Thanks for the review, and nice point on disk sizing. > > Unless there are objections, I shall keep the uploadTime as base, add the > sizing note to compatibility, and move min(..) to rejected alternatives. > > I will update KIP. > > Chia, pls let me know if this addresses the concern you raised on Case C. > > Thanks, > Murali > > On Thu, Nov 12, 2026 at 11:32 AM Viktor Somogyi-Vass < > [email protected]> wrote: > >> Hi Murali, >> >> I also agree with Chia-Ping, taking uploadTime as the new base for >> local.retention.ms would generally increase local disk usage which might >> be surprising and might have disk sizing consequences as well. So the >> extra local disk required would be local.retention.ms * upload rate, I >> think we can add this to the KIP. I don't think that changing behavior is a >> blocker if it results in a more predictable behavior. >> I was also considering min(maxTimestamp, uploadTime), but I think that >> overall it would complicate the config itself, which I would not take >> compared to uploadTime only as that would create a common, broker-bound and >> predictable interpretation for local.retention.ms. >> >> Best, >> Viktor >> >> On Thu, Nov 5, 2026 at 10:03 PM Muralidhar Basani via dev < >> [email protected]> wrote: >> >>> Thanks Chia for the review. >>> >>> 1. Case C: Yes, this is a behavior change. It only affects topics that >>> set >>> local.retention.ms lower than the age of the backfilled data. >>> With the default (-2 for local.retention.ms, means use the same value as >>> retention.ms), backfilled data isn't deleted right after upload today >>> either. >>> For example, with retention.ms = 7 days and local.retention.ms left at >>> its >>> default of -2, local retention is also 7 days. Data stays on local disk >>> as >>> long as it's kept at all. >>> >>> I see 2 options. >>> >>> a. Keep upload-time semantics and document the change in the upgrade >>> notes. >>> Operators who want to bound local disk can set local.retention.bytes: it >>> is >>> per partition, ignores timestamps, and deletes the oldest uploaded >>> segments >>> once the limit is exceeded. For a one-off backfill, they can also lower >>> local.retention.ms on the topic temporarily. >>> >>> Ex for bytes : As the limit applies to each partition separately, with >>> 5gb >>> per partition, and with 10 partitions, broker can hold upto 50gb locally. >>> As it only looks at size, it will ignore the timestamps then. >>> >>> b. What do you think of an alternative which is to measure from >>> min(record >>> timestamp, upload time). That keeps today's backfill behavior and still >>> fixes future timestamps, but local.retention.ms would no longer mean the >>> same thing for all data. >>> >>> Which one do you think fits operators better? >>> >>> 2. Metadata cache initializing: No, we won't fall back. RLM tasks skip >>> their run until RemoteLogMetadataManager#isReady is true, so >>> highestOffsetInRemoteStorage stays -1 and the upload-before-delete guard >>> blocks time-based local deletion. >>> Basically deletion is paused until cache is ready. So the fallback isn't >>> triggered. >>> >>> Pls let me know your thoughts. >>> >>> Thanks, >>> Murali >>> >>> On Thu, Nov 5, 2026 at 7:06 PM Chia-Ping Tsai via dev < >>> [email protected]> >>> wrote: >>> >>> > Hi Murali, >>> > >>> > I'm worried about Case C: "the local copy is deleted around 11:05, 1 >>> hour >>> > after the upload." >>> > >>> > Would this surprise operators who assume backfilled data is removed >>> right >>> > after upload? I guess the suitable approach for them is to use >>> > local.retention.bytes? >>> > >>> > Also, what happens if the RemoteLogMetadataCache is still initializing >>> > when the broker has just started? Will we always take the fallback? >>> > >>> > Best, >>> > Chia-Ping >>> > >>> > >>> > >>> > On 2026-10-23 18:30 UTC Muralidhar Basani via dev wrote: >>> > > Hi all, >>> > > >>> > > I would like to start a discussion about tiered storage topics and >>> how >>> > > local segments are deleted by time, once they are uploaded to remote >>> > > storage. >>> > > >>> > > >>> > >>> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/451979130/KIP-1385+Measure+local.retention.ms+from+remote+upload+time+instead+of+record+timestamps >>> > > >>> > > Looking forward to your thoughts on this approach. >>> > > >>> > > Thanks, >>> > > Murali >>> > > >>> > >>> >>
