I have tried this method, but the problem still exist.
by heap dump analysis, is 21 instances of
"org.apache.flink.util.ChildFirstClassLoader" normal?
------------------ 原始邮件 ------------------
发件人:
"Yangze Guo"
<karma...@gmail.com>;
发送时间: 2021年4月6日(星期二) 下午4:32
收件人: "太平洋"<495635...@qq.com>;
抄送: "user"<user@flink.apache.org>;
主题: Re: period batch job lead to OutOfMemoryError: Metaspace problem
I think you can try to increase the JVM metaspace option for
TaskManagers through taskmanager.memory.jvm-metaspace.size. [1]
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#outofmemoryerror-metaspace
Best,
Yangze Guo
Best,
Yangze Guo
On Tue, Apr 6, 2021 at 4:22 PM 太平洋 <495635...@qq.com> wrote:
>
> batch job：
> read data from s3 by sql，then by some operators and write data to
clickhouse and kafka.
> after some times, task-manager quit with OutOfMemoryError: Metaspace.
>
> env：
> flink version：1.12.2
> task-manager slot count: 5
> deployment： standalone kubernetes session 模式
> dependencies：
>
> <dependency>
>
>
<groupId>org.apache.flink</groupId>
>
>
<artifactId>flink-connector-kafka_2.11</artifactId>
>
>
<version>${flink.version}</version>
>
> </dependency>
>
> <dependency>
>
>
<groupId>com.google.code.gson</groupId>
>
> <artifactId>gson</artifactId>
>
> <version>2.8.5</version>
>
> </dependency>
>
> <dependency>
>
>
<groupId>org.apache.flink</groupId>
>
>
<artifactId>flink-connector-jdbc_2.11</artifactId>
>
>
<version>${flink.version}</version>
>
> </dependency>
>
> <dependency>
>
>
<groupId>ru.yandex.clickhouse</groupId>
>
>
<artifactId>clickhouse-jdbc</artifactId>
>
> <version>0.3.0</version>
>
> </dependency>
>
> <dependency>
>
>
<groupId>org.apache.flink</groupId>
>
>
<artifactId>flink-parquet_2.11</artifactId>
>
>
<version>${flink.version}</version>
>
> </dependency>
>
> <dependency>
>
>
<groupId>org.apache.flink</groupId>
>
>
<artifactId>flink-json</artifactId>
>
>
<version>${flink.version}</version>
>
> </dependency>
>
>
> heap dump1:
>
> Leak Suspects
>
> System Overview
>
> Leaks
>
> Overview
>
>
> Problem Suspect 1
>
> 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by
"sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 29,656,880 (41.16%)
bytes.
>
> Biggest instances:
>
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ca2a1e8 - 1,474,760
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2af820 - 1,474,168
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cdcaa10 - 1,474,160
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6aab0 - 1,474,160
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d1111d8 - 1,474,160
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2bb108 - 1,474,128
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73de202e0 - 1,474,120
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73dadc778 - 1,474,112
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d5f70e8 - 1,474,064
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d93aa38 - 1,474,064
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e179638 - 1,474,064
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73dc80418 - 1,474,056
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73dfcda60 - 1,474,056
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e4bcd38 - 1,474,056
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d6006e8 - 1,474,032
(2.05%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73c7d2ad8 - 1,461,944
(2.03%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ca1bb98 - 1,460,752
(2.03%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73bf203f0 - 1,460,744
(2.03%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e3284a8 - 1,445,232
(2.01%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e65de00 - 1,445,232
(2.01%) bytes.
>
>
>
> Keywords
> org.apache.flink.util.ChildFirstClassLoader
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> Details »
>
> Problem Suspect 2
>
> 34,407 instances of "org.apache.flink.core.memory.HybridMemorySegment",
loaded by "sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 7,707,168
(10.70%) bytes.
>
> Keywords
> org.apache.flink.core.memory.HybridMemorySegment
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
>
> Details »
>
>
>
> heap dump2:
>
> Leak Suspects
>
> System Overview
>
> Leaks
>
> Overview
>
> Problem Suspect 1
>
> 21 instances of "org.apache.flink.util.ChildFirstClassLoader", loaded by
"sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 26,061,408 (30.68%)
bytes.
>
> Biggest instances:
>
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e9e9930 - 1,474,224
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73edce0b8 - 1,474,224
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73f1ad7d0 - 1,474,168
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73f3e5118 - 1,474,168
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73f5d3fe0 - 1,474,168
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ebd8d28 - 1,474,160
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73efc00c0 - 1,474,160
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73e2251a8 - 1,474,136
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cc24af0 - 1,474,064
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cdca3e0 - 1,474,064
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73cf6f860 - 1,474,064
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d114768 - 1,474,064
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73ca6f878 - 1,474,056
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2b7640 - 1,474,056
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73d2c1d80 - 1,474,040
(1.74%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73c7e2868 - 1,469,720
(1.73%) bytes.
> org.apache.flink.util.ChildFirstClassLoader @ 0x73bf34a98 - 1,460,808
(1.72%) bytes.
>
>
>
> Keywords
> org.apache.flink.util.ChildFirstClassLoader
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> Details »
>
> Problem Suspect 2
>
> 4 instances of
"org.apache.flink.streaming.runtime.tasks.OneInputStreamTask", loaded by
"sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0" occupy 11,644,200 (13.71%)
bytes.
>
> Biggest instances:
>
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73e2d0cb0
- 4,364,536 (5.14%) bytes.
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73d62fb88
- 3,643,576 (4.29%) bytes.
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask @ 0x73dae0270
- 3,635,952 (4.28%) bytes.
>
>
>
> Keywords
> sun.misc.Launcher$AppClassLoader @ 0x73b2d42e0
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask
> Details »
>
>
