???????????????? OutOfMemoryError: Metaspace ????

Thu, 01 Apr 2021 01:19:33 -0700 

??????????
???????????????????????????????? sql ?? s3 ???????????????????????????????? 
clickhouse ?? kafka
?????????????? task-manager ???? OutOfMemoryError: Metaspace ????


??????????
flink ??????1.12.2
?????????? standalone kubernetes session ????
??????????

        &nbsp; &nbsp; <dependency&gt;

        &nbsp; &nbsp; &nbsp;&nbsp;<groupId&gt;org.apache.flink</groupId&gt;

        &nbsp; &nbsp; 
&nbsp;&nbsp;<artifactId&gt;flink-connector-kafka_2.11</artifactId&gt;

        &nbsp; &nbsp; &nbsp;&nbsp;<version&gt;${flink.version}</version&gt;

        &nbsp; &nbsp;&nbsp;</dependency&gt;

        &nbsp; &nbsp;&nbsp;<dependency&gt;

        &nbsp; &nbsp; &nbsp; <groupId&gt;com.google.code.gson</groupId&gt;

        &nbsp; &nbsp; &nbsp; <artifactId&gt;gson</artifactId&gt;

        &nbsp; &nbsp; &nbsp; <version&gt;2.8.5</version&gt;

        &nbsp; &nbsp; </dependency&gt;

        &nbsp; &nbsp; <dependency&gt;

        &nbsp; &nbsp; &nbsp; <groupId&gt;org.apache.flink</groupId&gt;

        &nbsp; &nbsp; &nbsp; 
<artifactId&gt;flink-connector-jdbc_2.11</artifactId&gt;

        &nbsp; &nbsp; &nbsp; <version&gt;${flink.version}</version&gt;&nbsp; 
&nbsp;

        &nbsp; &nbsp; </dependency&gt;

        &nbsp; &nbsp; <dependency&gt;

        &nbsp; &nbsp; &nbsp; <groupId&gt;ru.yandex.clickhouse</groupId&gt;

        &nbsp; &nbsp; &nbsp; <artifactId&gt;clickhouse-jdbc</artifactId&gt;

        &nbsp; &nbsp; &nbsp; <version&gt;0.3.0</version&gt;

        &nbsp; &nbsp; </dependency&gt;

        &nbsp; &nbsp; <dependency&gt;

        &nbsp; &nbsp; &nbsp; <groupId&gt;org.apache.flink</groupId&gt;

&nbsp;  &nbsp; &nbsp; 
&nbsp;&nbsp;<artifactId&gt;flink-parquet_2.11</artifactId&gt;

&nbsp;  &nbsp; &nbsp; &nbsp;&nbsp;<version&gt;${flink.version}</version&gt;

        &nbsp; &nbsp;&nbsp;</dependency&gt;

        &nbsp; &nbsp;&nbsp;<dependency&gt;

&nbsp;  &nbsp; &nbsp; &nbsp; &nbsp;<groupId&gt;org.apache.flink</groupId&gt;

&nbsp;  &nbsp; &nbsp; &nbsp; &nbsp;<artifactId&gt;flink-json</artifactId&gt;

&nbsp;  &nbsp; &nbsp; &nbsp; &nbsp;<version&gt;${flink.version}</version&gt;

        &nbsp; &nbsp;&nbsp;</dependency&gt;




heap dump1:

Leak Suspects

System Overview

&nbsp;Leaks&nbsp;

&nbsp;Overview


&nbsp;&nbsp;Problem Suspect 1

21 instances of&nbsp;"org.apache.flink.util.ChildFirstClassLoader", loaded 
by&nbsp;"sun.misc.Launcher$AppClassLoader @ 
0x73b2d42e0"&nbsp;occupy&nbsp;29,656,880 (41.16%)&nbsp;bytes.&nbsp;

Biggest instances:

org.apache.flink.util.ChildFirstClassLoader @ 0x73ca2a1e8&nbsp;-&nbsp;1,474,760 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73d2af820&nbsp;-&nbsp;1,474,168 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73cdcaa10&nbsp;-&nbsp;1,474,160 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6aab0&nbsp;-&nbsp;1,474,160 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73d1111d8&nbsp;-&nbsp;1,474,160 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73d2bb108&nbsp;-&nbsp;1,474,128 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73de202e0&nbsp;-&nbsp;1,474,120 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73dadc778&nbsp;-&nbsp;1,474,112 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73d5f70e8&nbsp;-&nbsp;1,474,064 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73d93aa38&nbsp;-&nbsp;1,474,064 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73e179638&nbsp;-&nbsp;1,474,064 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73dc80418&nbsp;-&nbsp;1,474,056 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73dfcda60&nbsp;-&nbsp;1,474,056 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73e4bcd38&nbsp;-&nbsp;1,474,056 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73d6006e8&nbsp;-&nbsp;1,474,032 
(2.05%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73c7d2ad8&nbsp;-&nbsp;1,461,944 
(2.03%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73ca1bb98&nbsp;-&nbsp;1,460,752 
(2.03%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73bf203f0&nbsp;-&nbsp;1,460,744 
(2.03%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73e3284a8&nbsp;-&nbsp;1,445,232 
(2.01%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73e65de00&nbsp;-&nbsp;1,445,232 
(2.01%) bytes.&nbsp;


Keywords
org.apache.flink.util.ChildFirstClassLoader
sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0

Details ?0?3



&nbsp;&nbsp;Problem Suspect 2

34,407 instances of&nbsp;"org.apache.flink.core.memory.HybridMemorySegment", 
loaded by&nbsp;"sun.misc.Launcher$AppClassLoader @ 
0x73b2d42e0"&nbsp;occupy&nbsp;7,707,168 (10.70%)&nbsp;bytes.&nbsp;

Keywords
org.apache.flink.core.memory.HybridMemorySegment
sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0


Details ?0?3










heap dump2:

Leak Suspects

System Overview

&nbsp;Leaks&nbsp;

&nbsp;Overview


&nbsp;&nbsp;Problem Suspect 1

21 instances of&nbsp;"org.apache.flink.util.ChildFirstClassLoader", loaded 
by&nbsp;"sun.misc.Launcher$AppClassLoader @ 
0x73b2d42e0"&nbsp;occupy&nbsp;26,061,408 (30.68%)&nbsp;bytes.&nbsp;

Biggest instances:

org.apache.flink.util.ChildFirstClassLoader @ 0x73e9e9930&nbsp;-&nbsp;1,474,224 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73edce0b8&nbsp;-&nbsp;1,474,224 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73f1ad7d0&nbsp;-&nbsp;1,474,168 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73f3e5118&nbsp;-&nbsp;1,474,168 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73f5d3fe0&nbsp;-&nbsp;1,474,168 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73ebd8d28&nbsp;-&nbsp;1,474,160 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73efc00c0&nbsp;-&nbsp;1,474,160 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73e2251a8&nbsp;-&nbsp;1,474,136 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73cc24af0&nbsp;-&nbsp;1,474,064 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73cdca3e0&nbsp;-&nbsp;1,474,064 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6f860&nbsp;-&nbsp;1,474,064 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73d114768&nbsp;-&nbsp;1,474,064 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73ca6f878&nbsp;-&nbsp;1,474,056 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73d2b7640&nbsp;-&nbsp;1,474,056 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73d2c1d80&nbsp;-&nbsp;1,474,040 
(1.74%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73c7e2868&nbsp;-&nbsp;1,469,720 
(1.73%) bytes.&nbsp;

org.apache.flink.util.ChildFirstClassLoader @ 0x73bf34a98&nbsp;-&nbsp;1,460,808 
(1.72%) bytes.&nbsp;


Keywords
org.apache.flink.util.ChildFirstClassLoader
sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0

Details ?0?3



&nbsp;&nbsp;Problem Suspect 2

4 instances 
of&nbsp;"org.apache.flink.streaming.runtime.tasks.OneInputStreamTask", loaded 
by&nbsp;"sun.misc.Launcher$AppClassLoader @ 
0x73b2d42e0"&nbsp;occupy&nbsp;11,644,200 (13.71%)&nbsp;bytes.&nbsp;

Biggest instances:

org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 
0x73e2d0cb0&nbsp;-&nbsp;4,364,536 (5.14%) bytes.&nbsp;

org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 
0x73d62fb88&nbsp;-&nbsp;3,643,576 (4.29%) bytes.&nbsp;

org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 
0x73dae0270&nbsp;-&nbsp;3,635,952 (4.28%) bytes.&nbsp;


Keywords
sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask

Details ?0?3

回复