[
https://issues.apache.org/jira/browse/FLINK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17569799#comment-17569799
]
ming li commented on FLINK-28390:
---------------------------------
[~yunta] sorry, maybe I didn't describe clearly.
In Flink we only need to provide FIFO related configuration (like
{{{}state.backend.rocksdb.fifo.max-size{}}}).
I have already reported the bug of FIFO TTL in RocksDB not taking effect to the
RockDB community (https://github.com/facebook/rocksdb/issues/10133).
> Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.
> -------------------------------------------------------------------
>
> Key: FLINK-28390
> URL: https://issues.apache.org/jira/browse/FLINK-28390
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Reporter: ming li
> Priority: Major
>
> We know that the fifo compaction strategy may silently delete data and may
> lose data for the business. But in some scenarios, FIFO compaction can be a
> very effective way to reduce CPU usage.
>
> Flink's Taskmanager is usually some small-scale processes, such as allocating
> 4 CPUs and 16G memory. When the state size is small, the CPU overhead
> occupied by RocksDB is not high, and as the state increases, RocksDB may
> frequently be in the compaction operation, which will occupy a large amount
> of CPU and affect the computing operation.
>
> We usually configure a TTL for the state, so when using FIFO we can configure
> it to be slightly longer than the TTL, so that the upper layer is the same as
> before.
>
> Although the FIFO Compaction strategy may bring space amplification, the disk
> is cheaper than the CPU after all, so the overall cost is reduced.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
