[
https://issues.apache.org/jira/browse/FLINK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137974#comment-18137974
]
Yordan Pavlov commented on FLINK-40327:
---------------------------------------
[~fanrui]
Yes it's safe.
The whole fix (0edbffc538a) changes one line. A ReadOptions that was created
for each batch and never closed (leaking native memory) is now created in a
try-with-resources block, so it gets closed:
{code:java}
- try {
- ReadOptions readOptions = new ReadOptions();
+ try (ReadOptions readOptions = new ReadOptions()) {
{code}
The change only makes the per-batch ReadOptions in
ForStGeneralMultiGetOperation get closed via try-with-resources. It's used only
by the synchronous multiGetAsList call inside the block, and the results are
copied into Java byte[], so nothing touches it after close. The overhead is one
JNI close per batch (not per record). There are no changes to state format,
checkpoints, config or API, and the commit cherry-picks cleanly onto
release-2.0–2.3 (2.4 already has it). So I believe it's safe to backport.
Cherry-picked onto release-2.0 to release-2.3 (clean, no conflicts);
ForStGeneralMultiGetOperationTest passes on all four.
> ForSt async multiGet leaks a native ReadOptions per read batch
> --------------------------------------------------------------
>
> Key: FLINK-40327
> URL: https://issues.apache.org/jira/browse/FLINK-40327
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0
> Reporter: Yordan Pavlov
> Assignee: Yordan Pavlov
> Priority: Critical
> Labels: pull-request-available
> Fix For: 2.4.0
>
> Attachments: after.png, before.png
>
>
> h4. Problem observed:
> The problem I observed unlimited memory increase of a Flink Kubernetes pod.
> Observed in production (Flink 2.3.0, ForSt disaggregated + async State V2,
> high read volume ~25k rec/s). TaskManager RSS grows ~2.6 GiB/h while
> heap/managed/network stay flat. OOMKill after ~10–14 h.
> h4. Investigation:
> jemalloc allocation profiling attributes the live-allocation
> growth to Java_org_forstdb_ReadOptions_newReadOptions (+530 MiB live in a
> 12-minute steady-state window).
>
> ForStGeneralMultiGetOperation.process() creates a new ReadOptions() inside
> the per-split executor lambda and no code path closes it — not the early
> returns, not the exception paths, not normal completion. ReadOptions is a JNI
> wrapper owning a native
> rocksdb::ReadOptions; without close() the native allocation is never freed,
> so memory leaks proportionally to the number of executed async read batches,
> outside all JVM/Flink memory accounting.
> All other ReadOptions/WriteOptions constructions in the module are
> lifecycle-managed (handlesToClose, toClose, closeQuietly in close()); this is
> the only unmanaged one.
> h4. Fix:
> Open the ReadOptions with try-with-resources spanning the lambda body. Safe
> because multiGetAsList is synchronous and returns byte[] copies, so nothing
> referencing the ReadOptions outlives the batch. We run this patched in
> production. Happy to open a PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
