tisonkun commented on PR #23003: URL: https://github.com/apache/flink/pull/23003#issuecomment-1638530747
``` java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either Flink Master requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'jobmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The Flink Master has to be shutdown... ``` The OOM is about class loading. The backtrace is: ``` [JobManager] STDOUT: 2023-07-17 12:09:55,039 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Fatal error occurred in the cluster entrypoint. [JobManager] STDOUT: java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either Flink Master requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'jobmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The Flink Master has to be shutdown... [JobManager] STDOUT: 2023-07-17 12:09:55,039 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Fatal error occurred in the cluster entrypoint. [JobManager] STDOUT: java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either Flink Master requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'jobmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The Flink Master has to be shutdown... [JobManager] STDOUT: at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_342] [JobManager] STDOUT: at java.lang.ClassLoader.defineClass(ClassLoader.java:756) ~[?:1.8.0_342] [JobManager] STDOUT: at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_342] [JobManager] STDOUT: at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) ~[?:1.8.0_342] [JobManager] STDOUT: at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_342] [JobManager] STDOUT: at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_342] [JobManager] STDOUT: at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_342] [JobManager] STDOUT: at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_342] [JobManager] STDOUT: at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_342] [JobManager] STDOUT: at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:71) ~[flink-dist-1.17.0.jar:1.17.0] [JobManager] STDOUT: at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51) ~[flink-dist-1.17.0.jar:1.17.0] [JobManager] STDOUT: at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_342] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:86) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:881) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.channel.AbstractChannel.write(AbstractChannel.java:301) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext.writeQuery(DnsQueryContext.java:178) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext.sendQuery(DnsQueryContext.java:141) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsQueryContext.query(DnsQueryContext.java:136) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.query0(DnsNameResolver.java:1322) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:450) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:1154) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.internalResolve(DnsResolveContext.java:362) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsResolveContext.resolve(DnsResolveContext.java:215) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.resolveNow(DnsNameResolver.java:1208) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.doResolveAllUncached0(DnsNameResolver.java:1194) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver.access$500(DnsNameResolver.java:93) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.resolver.dns.DnsNameResolver$7.run(DnsNameResolver.java:1142) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] [JobManager] STDOUT: at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[blob_p-b27adc8f726f7be55998719e732c5f4ecfae67b5-de2cfa0e85f194197c05016b94980ff1:4.0-SNAPSHOT] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
