[
https://issues.apache.org/jira/browse/FLINK-19005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181871#comment-17181871
]
Matthias edited comment on FLINK-19005 at 8/21/20, 1:09 PM:
------------------------------------------------------------
Thanks for the update:
A quick diff shows already that there is a growing number of classes generated
through reflection:
{code:bash}
# analysis on the "after 1 execution" heap dump
FLINK-19005 grep -o "class .*</a>" after_1/out.html | sed -e 's~class ~~g' -e
's~</a>~~g' | cut -d'$' -f1 | sed 's/[0-9]*$//g'| sort | uniq -c | sort -rn |
head
287 jdk.internal.reflect.GeneratedSerializationConstructorAccessor
150 jdk.internal.reflect.GeneratedMethodAccessor
136 oracle.jdbc.driver.Redirector
41 org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache
37 akka.remote.WireFormats
37 akka.remote.EndpointManager
35
org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.cache.LocalCache
35 akka.remote.RemoteSettings
33 org.apache.flink.shaded.hadoop2.com.google.common.cache.LocalCache
31 akka.remote.serialization.MiscMessageSerializer
{code}
{code:bash}
# analysis on the "after 10 executions" heap dump
FLINK-19005 grep -o "class .*</a>" after_10/out.html | sed -e 's~class ~~g' -e
's~</a>~~g' | cut -d'$' -f1 | sed 's/[0-9]*$//g'| sort | uniq -c | sort -rn |
head
575 jdk.internal.reflect.GeneratedSerializationConstructorAccessor
223 jdk.internal.reflect.GeneratedMethodAccessor
136 oracle.jdbc.driver.Redirector
49 com.sun.proxy.
41 org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache
37 akka.remote.WireFormats
37 akka.remote.EndpointManager
36 jdk.internal.reflect.GeneratedConstructorAccessor
35
org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.cache.LocalCache
35 akka.remote.RemoteSettings
{code}
was (Author: mapohl):
Thanks for the update:
A quick diff shows already that there is a growing number of class generated
through reflection:
{code:bash}
# analysis on the "after 1 execution" heap dump
FLINK-19005 grep -o "class .*</a>" after_1/out.html | sed -e 's~class ~~g' -e
's~</a>~~g' | cut -d'$' -f1 | sed 's/[0-9]*$//g'| sort | uniq -c | sort -rn |
head
287 jdk.internal.reflect.GeneratedSerializationConstructorAccessor
150 jdk.internal.reflect.GeneratedMethodAccessor
136 oracle.jdbc.driver.Redirector
41 org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache
37 akka.remote.WireFormats
37 akka.remote.EndpointManager
35
org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.cache.LocalCache
35 akka.remote.RemoteSettings
33 org.apache.flink.shaded.hadoop2.com.google.common.cache.LocalCache
31 akka.remote.serialization.MiscMessageSerializer
{code}
{code:bash}
# analysis on the "after 10 executions" heap dump
FLINK-19005 grep -o "class .*</a>" after_10/out.html | sed -e 's~class ~~g' -e
's~</a>~~g' | cut -d'$' -f1 | sed 's/[0-9]*$//g'| sort | uniq -c | sort -rn |
head
575 jdk.internal.reflect.GeneratedSerializationConstructorAccessor
223 jdk.internal.reflect.GeneratedMethodAccessor
136 oracle.jdbc.driver.Redirector
49 com.sun.proxy.
41 org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache
37 akka.remote.WireFormats
37 akka.remote.EndpointManager
36 jdk.internal.reflect.GeneratedConstructorAccessor
35
org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.cache.LocalCache
35 akka.remote.RemoteSettings
{code}
> used metaspace grow on every execution
> --------------------------------------
>
> Key: FLINK-19005
> URL: https://issues.apache.org/jira/browse/FLINK-19005
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet, Client / Job Submission
> Affects Versions: 1.11.1
> Reporter: Guillermo Sánchez
> Assignee: Chesnay Schepler
> Priority: Major
> Attachments: heap_dump_after_10_executions.zip,
> heap_dump_after_1_execution.zip
>
>
> Hi !
> Im running a 1.11.1 flink cluster, where I execute batch jobs made with
> DataSet API.
> I submit these jobs every day to calculate daily data.
> In every execution, cluster's used metaspace increase by 7MB and its never
> released.
> This ends up with an OutOfMemoryError caused by Metaspace every 15 days and i
> need to restart the cluster to clean the metaspace
> taskmanager.memory.jvm-metaspace.size is set to 512mb
> Any idea of what could be causing this metaspace grow and why is it not
> released ?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
