chia7712 commented on code in PR #23520: URL: https://github.com/apache/kafka/pull/23520#discussion_r4067694140
########## docs/operations/tiered-storage.md: ########## @@ -167,6 +167,27 @@ $ bin/kafka-topics.sh --delete --topic tieredTopic --bootstrap-server localhost: After topics are deleted, you're safe to set `remote.log.storage.system.enable=false` in the broker configuration. +## Delay Upload + +By default, non-active log segments are uploaded to remote storage as soon as they are eligible, keeping data in remote storage as up to date as possible. As a result, some segments before local retention are stored redundantly in both the local and remote tiers. If you do not need remote storage to always hold the newest data, you can delay upload with `remote.copy.lag.ms` and `remote.copy.lag.bytes` to reduce this redundancy and save remote storage space. The remote log manager uses these time-based and size-based lag parameters to decide when a segment becomes eligible for upload; the same settings are also available at the broker level as `log.remote.copy.lag.ms` and `log.remote.copy.lag.bytes`. + +For example, you can configure both settings for one topic as follows: + +```properties Review Comment: Would you mind adding two examples to show how to use them? For example: case 1: keep the last 2 days of data on local disk, so reads of recent data don't hit remote storage case 2: keep the last 10 GB of data on local disk, so reads of recent data don't hit remote storage -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
