[
https://issues.apache.org/jira/browse/BEAM-11570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kyle Weaver updated BEAM-11570:
-------------------------------
Description:
When running multiple jobs in local mode on the Flink 1.12 job server, pipeline
construction for subsequent jobs fails with the error below. This error was
caused by the addition of a new wrapper classloader to fix FLINK-16245.
The classloader is probably being leaked by GRPC threads that we can't shut
down cleanly.
RuntimeError: java.lang.IllegalStateException: Trying to access closed
classloader. Please check if you store classloaders directly or indirectly in
static fields. If the stacktrace suggests that the leak occurs in a third party
library and cannot be fixed immediately, you can disable this check with the
configuration 'classloader.check-leaked-classloader'. at
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:161)
at
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:168)
at java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:348) at
org.apache.beam.sdk.options.PipelineOptionsFactory.findCallersClassName(PipelineOptionsFactory.java:418)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.access$100(PipelineOptionsFactory.java:122)
at
org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.<init>(PipelineOptionsFactory.java:212)
at
org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.<init>(PipelineOptionsFactory.java:208)
at
org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.<init>(PipelineOptionsFactory.java:199)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.create(PipelineOptionsFactory.java:131)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:146) at
org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:394)
at
org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:491)
at
org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:232)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:817)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Commit/1707/testReport/junit/apache_beam.runners.portability.flink_runner_test/FlinkRunnerTestStreaming/test_external_transform/
was:
When running multiple jobs in local mode on the Flink 1.12 job server, pipeline
construction for subsequent jobs fails with the error below. This error was
caused by the addition of a new wrapper classloader to fix FLINK-16245.
RuntimeError: java.lang.IllegalStateException: Trying to access closed
classloader. Please check if you store classloaders directly or indirectly in
static fields. If the stacktrace suggests that the leak occurs in a third party
library and cannot be fixed immediately, you can disable this check with the
configuration 'classloader.check-leaked-classloader'. at
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:161)
at
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:168)
at java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:348) at
org.apache.beam.sdk.options.PipelineOptionsFactory.findCallersClassName(PipelineOptionsFactory.java:418)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.access$100(PipelineOptionsFactory.java:122)
at
org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.<init>(PipelineOptionsFactory.java:212)
at
org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.<init>(PipelineOptionsFactory.java:208)
at
org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.<init>(PipelineOptionsFactory.java:199)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.create(PipelineOptionsFactory.java:131)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:146) at
org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:394)
at
org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:491)
at
org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:232)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:817)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Commit/1707/testReport/junit/apache_beam.runners.portability.flink_runner_test/FlinkRunnerTestStreaming/test_external_transform/
> Flink 1.12 job server error: Trying to access closed classloader.
> -----------------------------------------------------------------
>
> Key: BEAM-11570
> URL: https://issues.apache.org/jira/browse/BEAM-11570
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: P2
> Labels: portability-flink
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When running multiple jobs in local mode on the Flink 1.12 job server,
> pipeline construction for subsequent jobs fails with the error below. This
> error was caused by the addition of a new wrapper classloader to fix
> FLINK-16245.
> The classloader is probably being leaked by GRPC threads that we can't shut
> down cleanly.
> RuntimeError: java.lang.IllegalStateException: Trying to access closed
> classloader. Please check if you store classloaders directly or indirectly in
> static fields. If the stacktrace suggests that the leak occurs in a third
> party library and cannot be fixed immediately, you can disable this check
> with the configuration 'classloader.check-leaked-classloader'. at
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:161)
> at
> org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:168)
> at java.lang.Class.forName0(Native Method) at
> java.lang.Class.forName(Class.java:348) at
> org.apache.beam.sdk.options.PipelineOptionsFactory.findCallersClassName(PipelineOptionsFactory.java:418)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.access$100(PipelineOptionsFactory.java:122)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.<init>(PipelineOptionsFactory.java:212)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.<init>(PipelineOptionsFactory.java:208)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.<init>(PipelineOptionsFactory.java:199)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.create(PipelineOptionsFactory.java:131)
> at org.apache.beam.sdk.Pipeline.create(Pipeline.java:146) at
> org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:394)
> at
> org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:491)
> at
> org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:232)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:817)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Commit/1707/testReport/junit/apache_beam.runners.portability.flink_runner_test/FlinkRunnerTestStreaming/test_external_transform/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
