[
https://issues.apache.org/jira/browse/FLINK-35853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867735#comment-17867735
]
Keith Lee commented on FLINK-35853:
-----------------------------------
Thanks for your response Jinzhong.
I understand that the FLINK-28699 was to reduce overhead in serde. However, it
does have the impact as I've described causing regression for jobs migrated
from 1.15.
It can be confusing for Flink users as we saw ever increasing checkpoint size
with RocksDB full checkpointing when our state size is expected to be static.
We actually spent significant time looking for state leak when we first
encountered the issue.
Is there a path forward here to prevent Flink users from going down similar
rabbit hole? For example:
1. Updating flink documentation on checkpoint size when using rocksdb full
checkpointing
2. Updating flink documentation to point to rocksdb configuration (if any) that
would allow for more frequent compaction
Thank you
> Regression in checkpoint size when performing full checkpointing in RocksDB
> ---------------------------------------------------------------------------
>
> Key: FLINK-35853
> URL: https://issues.apache.org/jira/browse/FLINK-35853
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 1.18.1
> Environment: amazon-linux-2023
> Reporter: Keith Lee
> Priority: Major
> Attachments: StaticStateSizeGenerator115.java,
> StaticStateSizeGenerator118.java
>
>
> We have a job with small and static state size (states are updated instead of
> added), the job is configured to use RocksDB + full checkpointng (incremental
> disabled) because the diff between checkpoint is larger than full checkpoint
> size.
> After migrating to 1.18, we observed significant and steady increase in full
> checkpoint size with RocksDB + full checkpointing. The increase was not
> observed with hashmap state backend.
> I managed to reproduce the issue with following code:
> [^StaticStateSizeGenerator115.java]
> [^StaticStateSizeGenerator118.java]
> Result:
> On Flink 1.15, RocksDB + full checkpointing, checkpoint size is constant at
> 250KiB.
> On Flink 1.18, RocksDB + full checkpointing, max checkpoint size got up to
> 38MiB before dropping (presumably due to compaction?)
> On Flink 1.18, Hashmap statebackend, checkpoint size is constant at 219KiB.
> Notes:
> One observation I have is that the issue is more pronounced with higher
> parallelism, the code uses 8 parallelism. The production application that we
> first saw the regression got up to GiB of checkpoint size, where we only
> expected and observed (in 1.15) at most a couple of MiB.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
