[
https://issues.apache.org/jira/browse/FLINK-28390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17564770#comment-17564770
]
Hangxiang Yu commented on FLINK-28390:
--------------------------------------
Hi, [~Ming Li] .
I have also seen some user question like :
[https://lists.apache.org/thread/n8q75r226byj7790m08nlkdpfj1mfsjo]
Currently IIUC, Flink should have supported to configure FIFO compaction and it
should work.
So you mean we need some improvements for this feature ?
> Allows RocksDB to configure FIFO Compaction to reduce CPU overhead.
> -------------------------------------------------------------------
>
> Key: FLINK-28390
> URL: https://issues.apache.org/jira/browse/FLINK-28390
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Reporter: ming li
> Priority: Major
>
> We know that the fifo compaction strategy may silently delete data and may
> lose data for the business. But in some scenarios, FIFO compaction can be a
> very effective way to reduce CPU usage.
>
> Flink's Taskmanager is usually some small-scale processes, such as allocating
> 4 CPUs and 16G memory. When the state size is small, the CPU overhead
> occupied by RocksDB is not high, and as the state increases, RocksDB may
> frequently be in the compaction operation, which will occupy a large amount
> of CPU and affect the computing operation.
>
> We usually configure a TTL for the state, so when using FIFO we can configure
> it to be slightly longer than the TTL, so that the upper layer is the same as
> before.
>
> Although the FIFO Compaction strategy may bring space amplification, the disk
> is cheaper than the CPU after all, so the overall cost is reduced.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
