[
https://issues.apache.org/jira/browse/BEAM-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989570#comment-16989570
]
Maximilian Michels commented on BEAM-8901:
------------------------------------------
There was a memory issue in the past due to the job server: BEAM-6923
One thing that we could test is setting {{MaxMetaspaceSize}}, e.g.
{{-XX:MaxMetaspaceSize=512M.}} because, by default, meta space grows
boundlessly in Java 8: FLINK-10317. Each Flink job will construct a new class
loader which reloads all pipeline related classes, leading to a growing meta
space.
I'd run some profiling on the JVM, to see which where most memory is allocated.
> add experimental flag for reusing flink local environment
> ---------------------------------------------------------
>
> Key: BEAM-8901
> URL: https://issues.apache.org/jira/browse/BEAM-8901
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Heejong Lee
> Assignee: Heejong Lee
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Flink job server launches a new mini cluster every time we run the pipeline
> on Flink local environment. To prevent OOM, we need to reuse existing Flink
> local environment if possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
