Hi, could you share with us the Flink cluster logs? This would help answering a lot of questions around your setup and the Flink version you are using. Thanks a lot!
Cheers, Till On Mon, Aug 24, 2020 at 10:48 AM 耿延杰 <[email protected]> wrote: > Still failed after every 12 tasks. > And the exception stack of failed tasks is different. > > > such as the recent failed tasks's exception info: > Caused by: java.lang.OutOfMemoryError: Metaspace > at java.lang.ClassLoader.defineClass1(Native > Method) > at > java.lang.ClassLoader.defineClass(ClassLoader.java:757) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at > java.net.URLClassLoader.defineClass(URLClassLoader.java:468) > at > java.net.URLClassLoader.access$100(URLClassLoader.java:74) > at > java.net.URLClassLoader$1.run(URLClassLoader.java:369) > at > java.net.URLClassLoader$1.run(URLClassLoader.java:363) > at > java.security.AccessController.doPrivileged(Native Method) > at > java.net.URLClassLoader.findClass(URLClassLoader.java:362) > at > org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:66) > at > java.lang.ClassLoader.loadClass(ClassLoader.java:352) > at > org.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:93) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:117) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:95) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:90) > at > ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:94) > at > ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:80) > at > ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55) > at > ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:47) > at > ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:29) > at > java.sql.DriverManager.getConnection(DriverManager.java:664) > at > java.sql.DriverManager.getConnection(DriverManager.java:270) > at org.apache.flink.api.java.io > .jdbc.AbstractJDBCOutputFormat.establishConnection(AbstractJDBCOutputFormat.java:68) > at > com.xxx.clickhouse.ClickHouseJDBCOutputFormat.open(ClickHouseJDBCOutputFormat.java:53) > at > org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205) > at > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) > at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) > at java.lang.Thread.run(Thread.java:748) > > > > > is different with the exception info in last email. > > > So analyse the dump file is the key. > > > > > > > ------------------ 原始邮件 ------------------ > 发件人: > "耿延杰" > < > [email protected]>; > 发送时间: 2020年8月24日(星期一) 下午4:33 > 收件人: "dev"<[email protected]>; > > 主题: 回复：OutOfMemoryError: Metaspace on Batch Task When Write into > Clickhouse > > > > Additional info: > > > The exception info in Flink Manager Page: > > > Caused by: java.lang.OutOfMemoryError: Metaspace > at java.lang.ClassLoader.defineClass1(Native Method) > at > java.lang.ClassLoader.defineClass(ClassLoader.java:757) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at > java.net.URLClassLoader.defineClass(URLClassLoader.java:468) > at > java.net.URLClassLoader.access$100(URLClassLoader.java:74) > at > java.net.URLClassLoader$1.run(URLClassLoader.java:369) > at > java.net.URLClassLoader$1.run(URLClassLoader.java:363) > at java.security.AccessController.doPrivileged(Native > Method) > at > java.net.URLClassLoader.findClass(URLClassLoader.java:362) > at > org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:66) > at java.lang.ClassLoader.loadClass(ClassLoader.java:352) > at > org.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:93) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:117) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:95) > at > ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:90) > at > ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:94) > at > ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:80) > at > ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55) > at > ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:47) > at > ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:29) > at > java.sql.DriverManager.getConnection(DriverManager.java:664) > at > java.sql.DriverManager.getConnection(DriverManager.java:270) > at org.apache.flink.api.java.io > .jdbc.AbstractJDBCOutputFormat.establishConnection(AbstractJDBCOutputFormat.java:68) > at > com.xx.xx.xx.ClickHouseJDBCOutputFormat.open(ClickHouseJDBCOutputFormat.java:53) > at > org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:205) > at > org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) > at > org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) > at java.lang.Thread.run(Thread.java:748) > > > > > > > ------------------ 原始邮件 ------------------ > 发件人: > "耿延杰" > < > [email protected]>; > 发送时间: 2020年8月24日(星期一) 下午4:20 > 收件人: "dev"<[email protected]>; > > 主题: OutOfMemoryError: Metaspace on Batch Task When Write into > Clickhouse > > > > Hi, > > > I catch "OutOfMemoryError: Metaspace" on Batch Task When Write into > Clickhouse. > Attached *.java file is my task code. > > And I find that, after running 12 tasks, the 13th task will be failed. And > the exception always is "OutOfMemoryError: Metaspace". see "task-failed.png" > > > I conf -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/hprofFile > and dump the hprof file. > I analyse this hprof file. And find this error occurs may not caused by my > user-code. > So I came here ask for your help. To confirm whether the memory leak > should be caused by Flink. > > > Attached file "java_pid29294.hprof" is the dump file. > > > Thanks.
