Thanks Xintong! In my case both of those parameters are set to false (default). I think I am sort of following what's happening here.
I have one TM with heap size set to 1GB. When the cluster is started the TM doesn't use that 1GB (no allocations). Once the first batch job is submitted I can see the memory roughly go up by 1GB. I presume this is when TM allocates its 1GB on the heap, and if I read correctly this is essentially a large byte buffer that is tenured so that it is never GCed. Flink writes any pojos (serializes) to this byte buffer and this is to essentially circumvent GC for performance. Once the job is done, this byte buffer remains on the heap, and the task manager keeps it to use for the next batch job. This is why I never see the memory go down after a batch job is complete. Does this make sense? Please let me know what you think. Thanks Tim On Thu, Oct 10, 2019, 11:16 PM Xintong Song <tonysong...@gmail.com> wrote: > I think it depends on your configurations. > - Are you using on-heap/off-heap managed memory? (configured by > 'taskmanager.memory.off-heap', by default is false) > > - Is managed memory pre-allocated? (configured by > 'taskmanager.memory.preallocate', by default is ffalse) > > > If managed memory is pre-allocated, then the allocated memory segments > will never be released. If it's not pre-allocated, memory segments should > be released when the task is finished, but the actual memory will not be > de-allocated until next GC. Since the job is finished, there may not be > enough heap activities to trigger the GC. If on-heap memory is used, you > may not be able to observe the decreasing of TM memory usage, because JVM > heap size does not scale down. Only if off-heap memory is used, you might > be able to observe the decreasing of TM memory usage after a GC, but not > from a jmap dump because jmap dumps heap memory usage only. > > > Besides, I don't think you need to worry about whether memory is released > after one job is finished. Sometimes flink/jvm do not release memory after > jobs/tasks finished, so that it can be reused directly by other jobs/tasks, > for the purpose of reducing allocate/deallocated overheads and optimizing > performance. > > > Thank you~ > > Xintong Song > > > > On Thu, Oct 10, 2019 at 7:55 PM Timothy Victor <vict...@gmail.com> wrote: > >> After a batch job finishes in a flink standalone cluster, I notice that >> the memory isn't freed up. I understand Flink uses it's own memory >> manager and just allocates a large tenured byte array that is not GC'ed. >> But does the memory used in this byte array get released when the batch >> job is done? >> >> The scenario I am facing is that I am running a series of scheduled batch >> jobs on a standalone cluster with 1 TM and 1 Slot. I notice that after a >> job is complete the memory used in the TM isn't freed up. I can confirm >> this by running jmap dump. >> >> Has anyone else run into this issue? This is on 1.9. >> >> Thanks >> >> Tim >> >
