Thanks Chia for the review.

1. Case C: Yes, this is a behavior change. It only affects topics that set
local.retention.ms lower than the age of the backfilled data.
With the default (-2 for local.retention.ms, means use the same value as
retention.ms), backfilled data isn't deleted right after upload today
either.
For example, with retention.ms = 7 days and local.retention.ms left at its
default of -2, local retention is also 7 days. Data stays on local disk as
long as it's kept at all.

I see 2 options.

a. Keep upload-time semantics and document the change in the upgrade notes.
Operators who want to bound local disk can set local.retention.bytes: it is
per partition, ignores timestamps, and deletes the oldest uploaded segments
once the limit is exceeded. For a one-off backfill, they can also lower
local.retention.ms on the topic temporarily.

Ex for bytes : As the limit applies to each partition separately, with 5gb
per partition, and with 10 partitions, broker can hold upto 50gb locally.
As it only looks at size, it will ignore the timestamps then.

b. What do you think of an alternative which is to measure from min(record
timestamp, upload time). That keeps today's backfill behavior and still
fixes future timestamps, but local.retention.ms would no longer mean the
same thing for all data.

Which one do you think fits operators better?

2. Metadata cache initializing: No, we won't fall back. RLM tasks skip
their run until RemoteLogMetadataManager#isReady is true, so
highestOffsetInRemoteStorage stays -1 and the upload-before-delete guard
blocks time-based local deletion.
Basically deletion is paused until cache is ready. So the fallback isn't
triggered.

Pls let me know your thoughts.

Thanks,
Murali

On Thu, Nov 5, 2026 at 7:06 PM Chia-Ping Tsai via dev <[email protected]>
wrote:

> Hi Murali,
>
> I'm worried about Case C: "the local copy is deleted around 11:05, 1 hour
> after the upload."
>
> Would this surprise operators who assume backfilled data is removed right
> after upload? I guess the suitable approach for them is to use
> local.retention.bytes?
>
> Also, what happens if the RemoteLogMetadataCache is still initializing
> when the broker has just started? Will we always take the fallback?
>
> Best,
> Chia-Ping
>
>
>
> On 2026-10-23 18:30 UTC Muralidhar Basani via dev wrote:
> > Hi all,
> >
> > I would like to start a discussion about tiered storage topics and how
> > local segments are deleted by time, once they are uploaded to remote
> > storage.
> >
> >
> https://cwiki.apache.org/confluence/spaces/KAFKA/pages/451979130/KIP-1385+Measure+local.retention.ms+from+remote+upload+time+instead+of+record+timestamps
> >
> > Looking forward to your thoughts on this approach.
> >
> > Thanks,
> > Murali
> >
>

Reply via email to