Thanks you for the response. We do see the heap actually shrink after
starting new jobs.
From: Matthias Pohl
Sent: Thursday, November 5, 2020 8:20 AM
To: Colletta, Edward
Cc: user@flink.apache.org
Subject: Re: a couple of memory questions
This email is from an external source - exercise caution regarding links and
attachments.
Hello Edward,
please find my answers within your message below:
On Wed, Nov 4, 2020 at 1:35 PM Colletta, Edward
mailto:edward.colle...@fmr.com>> wrote:
Using Flink 1.9.2 with FsStateBackend, Session cluster.
1. Does heap state get cleaned up when a job is cancelled?
We have jobs that we run on a daily basis. We start each morning and cancel
each evening. We noticed that the process size does not seem to shrink. We
are looking at the resident size of the process with ps and also the USED
column for Heap on the taskmanager page of the flink dashboard.
There is no explicit cleanup happening on the Flink side. The heap should be
cleaned up when GC kicks in.
1. How can I examine the usage of Flink Managed Memory?
The configuration documentation seems to indicate this is used for batch jobs,
and we are only using the Streaming API. I reduced
taskmanager.memory.fraction to 0.3, but I think this is still reserving too
much memory to an area we will not be using.
Unfortunately, I don't know of any way to monitor the managed memory for Flink
1.9.2 as is. We're going to introduce new metrics for managed memory [1],
network memory [2] and metaspace [3] in the upcoming release of Flink 1.12.0.
This should make it easier to monitor these memory pools.
I hope that helps a bit.
Best,
Matthias
[1] https://issues.apache.org/jira/browse/FLINK-14406
[2] https://issues.apache.org/jira/browse/FLINK-14422
[3] https://issues.apache.org/jira/browse/FLINK-19617
