Still failed after every 12 tasks.
And the exception stack of failed tasks is different.
such as the recent failed tasks's exception info:
Caused by: java.lang.OutOfMemoryError: Metaspace
at java.lang.ClassLoader.defineClass1(Native Method)
at
java.lang.ClassLoader.defineClass(ClassLoader.java:757)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at
java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at
java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at
java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at
java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native
Method)
at
java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at
org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:66)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at
org.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:93)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:117)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:95)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:90)
at
ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:94)
at
ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:80)
at
ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55)
at
ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:47)
at
ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:29)
at
java.sql.DriverManager.getConnection(DriverManager.java:664)
at
java.sql.DriverManager.getConnection(DriverManager.java:270)
at
org.apache.flink.api.java.io.jdbc.AbstractJDBCOutputFormat.establishConnection(AbstractJDBCOutputFormat.java:68)
at
com.xxx.clickhouse.ClickHouseJDBCOutputFormat.open(ClickHouseJDBCOutputFormat.java:53)
at
org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205)
at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
at java.lang.Thread.run(Thread.java:748)
is different with the exception info in last email.
So analyse the dump file is the key.
------------------ ???????? ------------------
??????:
"??????"
<[email protected]>;
????????: 2020??8??24??(??????) ????4:33
??????: "dev"<[email protected]>;
????: ??????OutOfMemoryError: Metaspace on Batch Task When Write into
Clickhouse
Additional info:
The exception info in Flink Manager Page:
Caused by: java.lang.OutOfMemoryError: Metaspace
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at
java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at
org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:66)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at
org.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:93)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:117)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:95)
at
ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:90)
at
ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:94)
at
ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:80)
at
ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55)
at
ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:47)
at
ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:29)
at
java.sql.DriverManager.getConnection(DriverManager.java:664)
at
java.sql.DriverManager.getConnection(DriverManager.java:270)
at
org.apache.flink.api.java.io.jdbc.AbstractJDBCOutputFormat.establishConnection(AbstractJDBCOutputFormat.java:68)
at
com.xx.xx.xx.ClickHouseJDBCOutputFormat.open(ClickHouseJDBCOutputFormat.java:53)
at
org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205)
at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
at java.lang.Thread.run(Thread.java:748)
------------------ ???????? ------------------
??????:
"??????"
<[email protected]>;
????????: 2020??8??24??(??????) ????4:20
??????: "dev"<[email protected]>;
????: OutOfMemoryError: Metaspace on Batch Task When Write into Clickhouse
Hi,
I catch "OutOfMemoryError: Metaspace" on Batch Task When Write into
Clickhouse.
Attached *.java file is my task code.
And I find that, after running 12 tasks, the 13th task will be failed. And the
exception always is "OutOfMemoryError: Metaspace". see "task-failed.png"
I conf -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/hprofFile
and dump the hprof file.
I analyse this hprof file. And find this error occurs may not caused by my
user-code.
So I came here ask for your help. To confirm whether the memory leak should be
caused by Flink.
Attached file "java_pid29294.hprof" is the dump file.
Thanks.
