[
https://issues.apache.org/jira/browse/FLINK-21986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17321239#comment-17321239
]
Feifan Wang commented on FLINK-21986:
-------------------------------------
Hi [~yunta], I confirm that I found the cause of this problem :
org.apache.flink.contrib.streaming.state.RocksDBOperationUtils#addColumnFamilyOptionsToCloseLater()
{color:#FF0000}called ColumnFamilyHandle#getDescriptor() twice by
mistake{color}。
{code:java}
public static void addColumnFamilyOptionsToCloseLater(
List<ColumnFamilyOptions> columnFamilyOptions, ColumnFamilyHandle
columnFamilyHandle) {
try {
if (columnFamilyHandle != null && columnFamilyHandle.getDescriptor() !=
null) {
columnFamilyOptions.add(columnFamilyHandle.getDescriptor().getOptions());
}
} catch (RocksDBException e) {
// ignore
}
}
{code}
As every time call ColumnFamilyHandle#getDescriptor() create a new
ColumnFamilyDescriptor which hold a new ColumnFamilyOptions instance,
RocksDBOperationUtils#addColumnFamilyOptionsToCloseLater create two
ColumnFamilyOptions instances actually，but only second be add to "closeLater
list". As a result, some ColumnFamilyOptions instances has not been closed.
I modified this method that only call ColumnFamilyHandle#getDescriptor() once
and run the test job again which result shows the modification works.
Please assign this issue to me, I will open a pull request to fix this problem,
thanks.
> taskmanager native memory not release timely after restart
> ----------------------------------------------------------
>
> Key: FLINK-21986
> URL: https://issues.apache.org/jira/browse/FLINK-21986
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 1.12.1
> Environment: flink version：1.12.1
> run ：yarn session
> job type：mock source -> regular join
>
> checkpoint interval: 3m
> Taskmanager memory : 16G
>
> Reporter: Feifan Wang
> Priority: Major
> Attachments: 82544.svg, image-2021-03-25-15-53-44-214.png,
> image-2021-03-25-16-07-29-083.png, image-2021-03-26-11-46-06-828.png,
> image-2021-03-26-11-47-21-388.png
>
>
> I run a regular join job with flink_1.12.1 , and find taskmanager native
> memory not release timely after restart cause by exceeded checkpoint
> tolerable failure threshold.
> *problem job information：*
> # job first restart cause by exceeded checkpoint tolerable failure threshold.
> # then taskmanager be killed by yarn many times
> # in this case，tm heap is set to 7.68G，bug all tm heap size is under 4.2G
> !image-2021-03-25-15-53-44-214.png|width=496,height=103!
> # nonheap size increase after restart，but still under 160M.
>
> !https://km.sankuai.com/api/file/cdn/706284607/716474606?contentType=1&isNewContent=false&isNewContent=false|width=493,height=102!
> # taskmanager process memory increase 3-4G after restart（this figure show
> one of taskmanager）
> !image-2021-03-25-16-07-29-083.png|width=493,height=107!
>
> *my guess：*
> [RocksDB
> wiki|https://github.com/facebook/rocksdb/wiki/RocksJava-Basics#memory-management]
> mentioned ：Many of the Java Objects used in the RocksJava API will be backed
> by C++ objects for which the Java Objects have ownership. As C++ has no
> notion of automatic garbage collection for its heap in the way that Java
> does, we must explicitly free the memory used by the C++ objects when we are
> finished with them.
> So, is it possible that RocksDBStateBackend not call
> AbstractNativeReference#close() to release memory use by RocksDB C++ Object ?
> *I make a change:*
> Actively call System.gc() and System.runFinalization() every minute.
> *And run this test again:*
> # taskmanager process memory no obvious increase
> !image-2021-03-26-11-46-06-828.png|width=495,height=93!
> # job run for several days，and restart many times，but no taskmanager killed
> by yarn like before
>
> *Summary：*
> # first，there is some native memory can not release timely after restart in
> this situation
> # I guess it maybe RocksDB C++ object，but I hive not check it from source
> code of RocksDBStateBackend
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
