Thanks for the insight Roman, and also for the GC tips. There are 2
reasons why I wanted to see this memory released. First as a way to just
confirm my understanding of Flink memory segment handling. Second is that
I run a single standalone cluster that runs both streaming and batch jobs,
and
Forced GC does not mean that JVM will even try to release the freed memory back
to the operating system. This highly depends on the JVM and garbage collector
used for your Flink setup, but most probably it's the jvm8 with the ParallelGC
collector.
ParallelGC is known to be not that aggressive
This part about the GC not cleaning up after the job finishes makes sense.
However, I o served that even after I run a "jcmd GC.run" on the
task manager process ID the memory is still not released. This is what
concerns me.
Tim
On Sat, Oct 12, 2019, 2:53 AM Xintong Song wrote:
> Generally
Thanks Xintong! In my case both of those parameters are set to false
(default). I think I am sort of following what's happening here.
I have one TM with heap size set to 1GB. When the cluster is started the
TM doesn't use that 1GB (no allocations). Once the first batch job is
submitted I can
I think it depends on your configurations.
- Are you using on-heap/off-heap managed memory? (configured by
'taskmanager.memory.off-heap', by default is false)
- Is managed memory pre-allocated? (configured by
'taskmanager.memory.preallocate', by default is ffalse)
If managed memory is
Hi Tim,
Do you mean the user heap memory used by the tasks of finished jobs is not
freed up? If this is the case,
the memory usage of taskmanger will increase as more and more jobs
finished. However this does not
happen, the memory will be freed up by jvm gc.
BTW, flink has its own memory
After a batch job finishes in a flink standalone cluster, I notice that the
memory isn't freed up. I understand Flink uses it's own memory manager
and just allocates a large tenured byte array that is not GC'ed. But does
the memory used in this byte array get released when the batch job is
