yarn.provided.lib.dirs在flink1.11 yarn提交不生效

Tue, 22 Dec 2020 00:56:36 -0800 


flink1.11 on yarn模式，我提前将flink
lib下的依赖及自定义函数jar上传到hdfs上，提交时使用yarn.provided.lib.dirs
指定hdfs的依赖路径。原本设想程序中使用反射去寻找自定义函数的类并且实例化，但是提交时报错，程序并没有找到自定义函数的路径

提交命令:/usr/hdp/flink1.11/bin/flink run -m yarn-cluster -d -ynm udf-test -yD
yarn.provided.lib.dirs=hdfs://ip:8020/flink-yarn/jars -c
com.ly.common.udf.demo.FlinkUDFDemo  /data/bigdata/jars/udf-test.jar

相关信息如下:
2020-12-22 08:41:11,157 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli     
          
[] - Dynamic Property set:
yarn.provided.lib.dirs=hdfs://chaitin/flink-yarn/jars
2020-12-22 08:41:11,157 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli     
          
[] - Dynamic Property set:
yarn.provided.lib.dirs=hdfs://chaitin/flink-yarn/jars
-- class path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: object com.ly.third.udf.flink.SortKey not found.
        at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
        at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
        at 
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
        at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232)
        at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
        at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
Caused by: scala.ScalaReflectionException: object
com.ly.third.udf.flink.SortKey not found.
        at 
scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:162)
        at 
scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:22)
        at
com.ly.common.udf.reflect.RegisterFlinkFunction$.loadFlinkFunction(RegisterFlinkFunction.scala:14)
        at com.ly.common.udf.demo.FlinkUDFDemo$.main(FlinkUDFDemo.scala:27)
        at com.ly.common.udf.demo.FlinkUDFDemo.main(FlinkUDFDemo.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
        ... 11 more



--
Sent from: http://apache-flink.147419.n8.nabble.com/






















					回复