[
https://issues.apache.org/jira/browse/KAFKA-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17647807#comment-17647807
]
A. Sophie Blee-Goldman commented on KAFKA-14038:
------------------------------------------------
[~divijvaidya] bumped to 3.5 as we are past code freeze for 3.4.0
> Optimize calculation of size for log in remote tier
> ---------------------------------------------------
>
> Key: KAFKA-14038
> URL: https://issues.apache.org/jira/browse/KAFKA-14038
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Reporter: Divij Vaidya
> Assignee: Divij Vaidya
> Priority: Major
> Labels: needs-kip
> Fix For: 3.5.0
>
>
> {color:#24292f}As per the Tiered Storage feature introduced in
> [KIP-405|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage],
> users can configure the retention of remote tier based on time, by size, or
> both. The work of computing the log segments to be deleted based on the
> retention config is [owned by
> RemoteLogManager|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-1.RemoteLogManager(RLM)ThreadPool]
> (RLM).{color}
> {color:#24292f}To compute remote segments eligible for deletion based on
> retention by size config, {color}RLM needs to compute the
> {{total_remote_log_size}} i.e. the total size of logs available in the remote
> tier for that topic-partition. RLM could use the
> {{RemoteLogMetadataManager.listRemoteLogSegments()}} to fetch metadata for
> all the remote segments and then aggregate the segment sizes by using
> {{{}RemoteLogSegmentMetadata.segmentSizeInBytes(){}}}to find the total log
> size stored in the remote tier.
> The above method involves iterating through all metadata of all the segments
> i.e. O({color:#24292f}num_remote_segments{color}) on each execution of RLM
> thread. {color:#24292f}Since the main feature of tiered storage is storing a
> large amount of data, we expect num_remote_segments to be large and a
> frequent linear scan could be expensive (depending on the underlying storage
> used by RemoteLogMetadataManager).
> Segment offloads and segment deletions are run together in the same task and
> a fixed size thread pool is shared among all topic-partitions. A slow logic
> for calculation of total_log_size could result in the loss of availability as
> demonstrated in the following scenario:{color}
> # {color:#24292f}Calculation of total_size is slow and the threads in the
> thread pool are busy with segment deletions{color}
> # Segment offloads are delayed (since they run together with deletions)
> # Local disk fills up, since local deletion requires the segment to be
> offloaded
> # If local disk is completely full, Kafka fails
> Details are in KIP -
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
