Yordan Pavlov created FLINK-40327:
-------------------------------------
Summary: ForSt async multiGet leaks a native ReadOptions per read
batch
Key: FLINK-40327
URL: https://issues.apache.org/jira/browse/FLINK-40327
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 2.3.0, 2.2.0, 2.1.0, 2.0.0
Reporter: Yordan Pavlov
h4. Problem observed:
The problem I observed unlimited memory increase of a Flink Kubernetes pod.
Observed in production (Flink 2.3.0, ForSt disaggregated + async State V2, high
read volume ~25k rec/s). TaskManager RSS grows ~2.6 GiB/h while
heap/managed/network stay flat. OOMKill after ~10–14 h.
h4. Investigation:
jemalloc allocation profiling attributes the live-allocation
growth to Java_org_forstdb_ReadOptions_newReadOptions (+530 MiB live in a
12-minute steady-state window).
ForStGeneralMultiGetOperation.process() creates a new ReadOptions() inside the
per-split executor lambda and no code path closes it — not the early returns,
not the exception paths, not normal completion. ReadOptions is a JNI wrapper
owning a native
rocksdb::ReadOptions; without close() the native allocation is never freed, so
memory leaks proportionally to the number of executed async read batches,
outside all JVM/Flink memory accounting.
All other ReadOptions/WriteOptions constructions in the module are
lifecycle-managed (handlesToClose, toClose, closeQuietly in close()); this is
the only unmanaged one.
h4. Fix:
Open the ReadOptions with try-with-resources spanning the lambda body. Safe
because multiGetAsList is synchronous and returns byte[] copies, so nothing
referencing the ReadOptions outlives the batch. We run this patched in
production. Happy to open a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
