[
https://issues.apache.org/jira/browse/KAFKA-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127217#comment-18127217
]
Amrit Sarkar commented on KAFKA-19952:
--------------------------------------
Hi [~fujian1115], we reproduced this on our own Kafka setup (trunk, single
KRaft broker, tiered storage with {{LocalTieredStorage}},
{{local.retention.ms=1000}}, {{retention.ms=120000}}). After making the remote
storage dir unwritable to simulate an outage, the copy task kept failing with
{{RemoteStorageException}} and all 8 local segments were still on disk 4
minutes later, twice past {{retention.ms}}. Once access came back they were
uploaded and cleaned up within a minute.
In practice this is what an operator hits when object store credentials expire,
during a regional S3/GCS outage, or on a network partition to the store: local
disk grows without bound until the store recovers. With KIP-1241's delayed
upload the exposure is a bit larger.
Now that KIP-1241 has landed
([#20913|https://github.com/apache/kafka/pull/20913]), are you planning to pick
this back up? [#21361|https://github.com/apache/kafka/pull/21361] was closed by
the stale bot in June. If not, I'd like to take it, following [~kamalcph]'s
review on [#21361|https://github.com/apache/kafka/pull/21361]: handle it in
{{UnifiedLog}} by advancing the log start offset for segments that breach the
full {{retention.ms}} / {{retention.bytes}} even if they were never uploaded,
scanning from the oldest segment and stopping at the first one that doesn't
breach (no deleting from the middle).
> When remote storage keep outage status. the local segement never get deleted
> ----------------------------------------------------------------------------
>
> Key: KAFKA-19952
> URL: https://issues.apache.org/jira/browse/KAFKA-19952
> Project: Kafka
> Issue Type: Bug
> Components: Tiered-Storage
> Reporter: fujian
> Priority: Major
>
> As titled.
> For example. If you save one topic local retention: 1 days and remote
> storage: 3 days.
> If the remote storage outage for 4 days. The local segments will save 4 days.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)