DavidFricker commented on pull request #15491:
URL: https://github.com/apache/flink/pull/15491#issuecomment-874873342
We have implemented something similar and started receiving classloader
exceptions as below, however the job continues to work fine and stream data
from Kinesis:
```
Job has been submitted with JobID 31975670b3660829abf7e69c3a13c2c6
Exception in thread "Thread-11" java.lang.IllegalStateException: Trying to
access closed classloader. Please check if you store classloaders directly or
indirectly in static fields. If the stacktrace suggests that the leak occurs in
a third party library and cannot be fixed immediately, you can disable this
check with the configuration 'classloader.check-leaked-classloader'.
at
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:164)
at
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResource(FlinkUserCodeClassLoaders.java:183)
at java.lang.ClassLoader.getResource(ClassLoader.java:1084)
at
org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2779)
at
org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3035)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2994)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2967)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2847)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1199)
at
org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1811)
at
org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1788)
at
org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183)
at
org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145)
at
org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102)
```
We also tried the implementation from this PR and got the same error as
above.
Happy to contribute back here and help to debug if someone can point us in
the right direction. We are using:
- Flink 1.13.1
- apache-flink==1.13.1 (for pyflink)
- YARN on AWS EMR
- `flink-sql-connector-kinesis_2.11-1.13.1.jar`
- `flink run -t yarn-per-job --detached -py /path/to/main.py --pyFiles
/path/to/dependencies.zip -pyexec /usr/bin/python3.7 --parallelism 4` to submit
the job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
