[ 
https://issues.apache.org/jira/browse/KAFKA-19952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127217#comment-18127217
 ] 

Amrit Sarkar commented on KAFKA-19952:
--------------------------------------

Hi [~fujian1115], we reproduced this on our own Kafka setup (trunk, single 
KRaft broker, tiered storage with {{LocalTieredStorage}}, 
{{local.retention.ms=1000}}, {{retention.ms=120000}}). After making the remote 
storage dir unwritable to simulate an outage, the copy task kept failing with 
{{RemoteStorageException}} and all 8 local segments were still on disk 4 
minutes later, twice past {{retention.ms}}. Once access came back they were 
uploaded and cleaned up within a minute.

In practice this is what an operator hits when object store credentials expire, 
during a regional S3/GCS outage, or on a network partition to the store: local 
disk grows without bound until the store recovers. With KIP-1241's delayed 
upload the exposure is a bit larger.

Now that KIP-1241 has landed 
([#20913|https://github.com/apache/kafka/pull/20913]), are you planning to pick 
this back up? [#21361|https://github.com/apache/kafka/pull/21361] was closed by 
the stale bot in June. If not, I'd like to take it, following [~kamalcph]'s 
review on [#21361|https://github.com/apache/kafka/pull/21361]: handle it in 
{{UnifiedLog}} by advancing the log start offset for segments that breach the 
full {{retention.ms}} / {{retention.bytes}} even if they were never uploaded, 
scanning from the oldest segment and stopping at the first one that doesn't 
breach (no deleting from the middle).


> When remote storage keep outage status. the local segement never get deleted
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-19952
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19952
>             Project: Kafka
>          Issue Type: Bug
>          Components: Tiered-Storage
>            Reporter: fujian
>            Priority: Major
>
> As titled. 
> For example. If you save one topic local retention: 1 days and remote 
> storage: 3 days.
> If the remote storage outage for 4 days. The local segments will save 4 days.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to