Hi everyone,
Flink 1.0.2
Hadoop 2.4.0
I am running Flink on Yarn by using FlinkYarnClient to launch a Flink
cluster and Flink Client to submit a PackagedProgram. To keep it simple,
for batch jobs I use the WordCount example and for streaming the
IterateExample and IncrementalLearning ones without args.
Batch job executes successfully. However, the streaming ones fail with
ClassNotFoundException.
For example the IncrementalLearning job throws this exception:
Caused by: java.lang.RuntimeException: Could not look up the
main(String[]) method from the class
org.apache.flink.streaming.examples.ml.IncrementalLearningSkeleton:
org/apache/flink/streaming/api/functions/source/SourceFunction
at
org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:479)
at
org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
at
org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:106)
[..]
Caused by: java.lang.NoClassDefFoundError:
org/apache/flink/streaming/api/functions/source/SourceFunction
org/apache/flink/streaming/api/functions/source/SourceFunction.class
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2615)
at java.lang.Class.getMethod0(Class.java:2856)
at java.lang.Class.getMethod(Class.java:1668)
at
org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
... 45 more
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.streaming.api.functions.source.SourceFunction
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 50 more
The taskmanager classpath includes the following:
Classpath:
/srv/hadoop-2.4.0/tmp/nm-local-dir/usercache/myuser/appcache/application_1462487692793_0012/container_1462487692793_0012_01_000002/flink-dist_2.10-1.0.2.jar
It could be my pom Yarn dependency which I am not so sure about if I'm
using the proper version:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.10</artifactId>
<version>1.0.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-yarn_2.10</artifactId>
<version>1.1-SNAPSHOT</version>
</dependency>
Thanks for you help!
Cheers,
Theo
