[
https://issues.apache.org/jira/browse/BEAM-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991691#comment-16991691
]
Maximilian Michels commented on BEAM-8901:
------------------------------------------
{quote}
Heap dump inspection shows that byte[] objects are occupying most of the memory
and I can see half-million empty HybridMemorySegments each owning byte[] of
32,792B.
{quote}
That is Flink's managed memory. Those segments will be used by for some of the
algorithms which manage their own memory (to avoid OOM errors), e.g. sorting,
large record handling, spilling to disk. So they are nothing unusual and they
should be GC-collected when the references to the cluster go away. Possibly,
the job server (or the GRPC classes, etc.) hold on to a cluster reference,
which never lets the GC kick in.
> add experimental flag for reusing flink local environment
> ---------------------------------------------------------
>
> Key: BEAM-8901
> URL: https://issues.apache.org/jira/browse/BEAM-8901
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Heejong Lee
> Assignee: Heejong Lee
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Flink job server launches a new mini cluster every time we run the pipeline
> on Flink local environment. To prevent OOM, we need to reuse existing Flink
> local environment if possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
