Thanks Chia for the review. 1. Case C: Yes, this is a behavior change. It only affects topics that set local.retention.ms lower than the age of the backfilled data. With the default (-2 for local.retention.ms, means use the same value as retention.ms), backfilled data isn't deleted right after upload today either. For example, with retention.ms = 7 days and local.retention.ms left at its default of -2, local retention is also 7 days. Data stays on local disk as long as it's kept at all.
I see 2 options. a. Keep upload-time semantics and document the change in the upgrade notes. Operators who want to bound local disk can set local.retention.bytes: it is per partition, ignores timestamps, and deletes the oldest uploaded segments once the limit is exceeded. For a one-off backfill, they can also lower local.retention.ms on the topic temporarily. Ex for bytes : As the limit applies to each partition separately, with 5gb per partition, and with 10 partitions, broker can hold upto 50gb locally. As it only looks at size, it will ignore the timestamps then. b. What do you think of an alternative which is to measure from min(record timestamp, upload time). That keeps today's backfill behavior and still fixes future timestamps, but local.retention.ms would no longer mean the same thing for all data. Which one do you think fits operators better? 2. Metadata cache initializing: No, we won't fall back. RLM tasks skip their run until RemoteLogMetadataManager#isReady is true, so highestOffsetInRemoteStorage stays -1 and the upload-before-delete guard blocks time-based local deletion. Basically deletion is paused until cache is ready. So the fallback isn't triggered. Pls let me know your thoughts. Thanks, Murali On Thu, Nov 5, 2026 at 7:06 PM Chia-Ping Tsai via dev <[email protected]> wrote: > Hi Murali, > > I'm worried about Case C: "the local copy is deleted around 11:05, 1 hour > after the upload." > > Would this surprise operators who assume backfilled data is removed right > after upload? I guess the suitable approach for them is to use > local.retention.bytes? > > Also, what happens if the RemoteLogMetadataCache is still initializing > when the broker has just started? Will we always take the fallback? > > Best, > Chia-Ping > > > > On 2026-10-23 18:30 UTC Muralidhar Basani via dev wrote: > > Hi all, > > > > I would like to start a discussion about tiered storage topics and how > > local segments are deleted by time, once they are uploaded to remote > > storage. > > > > > https://cwiki.apache.org/confluence/spaces/KAFKA/pages/451979130/KIP-1385+Measure+local.retention.ms+from+remote+upload+time+instead+of+record+timestamps > > > > Looking forward to your thoughts on this approach. > > > > Thanks, > > Murali > > >
