HI ALL:
   
背景:我使用flink客户端脚本提交作业到Yarn集群,通过-C参数指定udf依赖包的URL,yarn.ship-files参数指定了依赖包目录,在yarn-per-job模式下作业可正常运行得到正确结果,但在yarn-session模式下,却报ClassNotFoundException,查看taskmanager.log,依赖包确实没有加载到classpath。
    两种提交命令分别如下:
    1、flink-1.13.2/bin/flink run -t yarn-per-job 
-Dyarn.ship-files=/home/chenqizhu/myflink/usrlib -C 
file:///home/chenqizhu/myflink/usrlib/flink-sql-udf-1.0-SNAPSHOT.jar 
flink-test-1.0-SNAPSHOT.jar
    2、flink-1.13.2/bin/flink run -t yarn-session 
-Dyarn.application.id=application_1607512101880_345557 
-Dyarn.ship-files=/home/chenqizhu/myflink/usrlib -C 
file:///home/chenqizhu/myflink/usrlib/flink-sql-udf-1.0-SNAPSHOT.jar 
flink-test-1.0-SNAPSHOT.jar
   
   问题:为什么在yarn-session模式下报错,per-job模式却可正常运行
   
如果以上命令不适用于Yarn-session模式,请问我用yarn-session模式提交作业时,在不把udf依赖打进作业jar的情况下,该如何加载udf依赖包?


客户端版本:1.13.2
taskmanager.log报错信息:
ClassLoader info: URLClassLoader:
Classnot resolvable through given classloader.
    at 
org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:336)
    at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperator(OperatorChain.java:656)
    at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:629)
    at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:569)
    at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:186)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:548)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
    at 
org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
    at java.lang.Thread.run(Thread.java:748)
Causedby: java.lang.ClassNotFoundException: 
com.cqz.component.flink.sql.udf.SumFunction
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at 
org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
    at 
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
    at 
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at 
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:172)
    at java.lang.Class.forName0(NativeMethod)
    at java.lang.Class.forName(Class.java:348)
    at 
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:76)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1975)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
    at 
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615)
    at 
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600)
    at 
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587)
    at 
org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541)
    at 
org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:322)
    ... 10 more
   

回复