fxbing commented on issue #275: URL: https://github.com/apache/fluss/issues/275#issuecomment-4855183236
@loserwang1024 Hi, I’m new to the project and came across this issue while learning the tiered log and partitioned table code paths. From my current understanding, `table.log.tiered.local-segments` is a table-level option, but it is applied to each partition bucket. So for a time-partitioned table, old partitions may still keep the same number of local segments as the latest/hot partitions, which can lead to large local disk usage when there are many partitions and buckets. Would it make sense to address this in a limited scope first, only for tables with auto-partition strategy enabled? One possible approach could be: - Keep `table.log.tiered.local-segments` as the current/default setting, used for recent or hot partitions. - Add an optional setting for old partitions, for example `table.log.tiered.old-partition.local-segments`. - Add a small policy to decide which partitions are considered recent, for example `table.log.tiered.recent-partitions`. - For tables with auto-partition strategy enabled, the latest N time partitions use `table.log.tiered.local-segments`, while older partitions use `table.log.tiered.old-partition.local-segments`. - This should also cover manually created partitions in such tables, as long as they follow the table’s auto-partition time strategy. - For non-partitioned tables, and partitioned tables without auto-partition strategy, keep the current behavior unchanged. - For the first version, validate `table.log.tiered.old-partition.local-segments >= 1`. This can still reduce local disk usage for old partitions, while avoiding the extra edge cases of removing all local segments from a partition bucket. Supporting `0` could be considered separately after the local log lifecycle is reviewed more carefully. If `table.log.tiered.old-partition.local-segments` is not configured, the behavior can stay the same as today. Does this direction sound reasonable to the community? If so, I’d be happy to try working on a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
