[
https://issues.apache.org/jira/browse/FLINK-18712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165194#comment-17165194
]
Yun Tang commented on FLINK-18712:
----------------------------------
[~lio_sy], do you have a simple job to reproduce this and what happened if we
move this job to YARN as it could also kill container once memory exceed the
memory limit. I ask this is because I just wonder whether k8s would take os
cache into account as the memory usage for that pod.
To know how much memory used in RocksDB, there existed two ways:
# Turn on block cache memory usage metrics:
[https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#state-backend-rocksdb-metrics-block-cache-usage]
when managed memory for RocksDB is enabled. And remember that all rocksDB
instances within one slot would report the same value.
# Use jemalloc and jeprof to see how much memory allocated from os by RocksDB,
this is much more precious than 1st solution, and you could refer to
[https://github.com/jeffgriffith/native-jvm-leaks/#going-native-with-jemalloc]
to re-build and pre load related ".so" file. For flink, you could refer to
[https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/config.html#forwarding-environment-variables]
to know how to pass environment variables {{LD_PRELOAD and }}{{MALLOC_CONF}}.
I think by doing this, you could know whether rocksDB has used too much
unexpected memory.
> Flink RocksDB statebackend memory leak issue
> ---------------------------------------------
>
> Key: FLINK-18712
> URL: https://issues.apache.org/jira/browse/FLINK-18712
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 1.10.0
> Reporter: Farnight
> Priority: Critical
>
> When using RocksDB as our statebackend, we found it will lead to memory leak
> when restarting job (manually or in recovery case).
>
> How to reproduce:
> # increase RocksDB blockcache size(e.g. 1G), it is easier to monitor and
> reproduce.
> # start a job using RocksDB statebackend.
> # when the RocksDB blockcache reachs maximum size, restart the job. and
> monitor the memory usage (k8s pod working set) of the TM.
> # go through step 2-3 few more times. and memory will keep raising.
>
> Any solution or suggestion for this? Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
