You can also have a look at the YARN client logs, which should print which JARs are uploaded. The container logs should also log the class path.
On Sun, Jul 3, 2016 at 6:04 PM, Jamie Grier <ja...@data-artisans.com> wrote: > Hi Bruce, > > I just spun up an EMR cluster and tried this out. Hadoop 2.7.2 and Flink > 1.0.3. I ran the exact same command as you and everything works just fine. > > Please verify one thing, though. In your command you do not specify the > path to the Flink executable, which means it's just getting picked up from > your PATH. Is it possible you're running a flink-1.0.3 example program but > using a different version of Flink? Try the following: > > ./flink-1.0.3/bin/flink run -m yarn-cluster -yn 1 -ynm test1:WordCount > ./flink-1.0.3/examples/streaming/WordCount.jar > > > > -Jamie > > > > On Thu, Jun 30, 2016 at 11:28 AM, Hanson, Bruce <bruce.han...@here.com> > wrote: > >> I’m trying to submit a stand-alone Flink job to a YARN cluster running on >> EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container >> for the Job Manager, it fails. The error message from the container is >> below. The command I’m using is: >> >> $ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount >> ./flink-1.0.3/examples/streaming/WordCount.jar >> >> I have tried adding log4j and slf4j libraries to the classpath using -C >> and that doesn’t help. >> >> This does not happen on other YARN clusters I have that are not EMR >> nodes. And it doesn’t happen on my EMR cluster if I use "yarn-session.sh" >> to create a Flink cluster in the YARN cluster and then use “flink run …” to >> submit the job to the Flink cluster. >> >> Does anyone out there know how I could fix this? >> >> Thanks in advance for any help you can give. >> >> Error message in the jobmanager.err file: >> >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >> >> SLF4J: Defaulting to no-operation (NOP) logger implementation >> >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for >> further details. >> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/log4j/Level >> >> at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357) >> >> at java.lang.Class.forName0(Native Method) >> >> at java.lang.Class.forName(Class.java:278) >> >> at >> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134) >> >> at >> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95) >> >> at >> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) >> >> at >> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) >> >> at org.apache.hadoop.security.Groups.<init>(Groups.java:79) >> >> at org.apache.hadoop.security.Groups.<init>(Groups.java:74) >> >> at >> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303) >> >> at >> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283) >> >> at >> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260) >> >> at >> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790) >> >> at >> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760) >> >> at >> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633) >> >> at >> org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64) >> >> at >> org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36) >> >> at >> org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala) >> >> Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> >> ... 18 more >> >> >> *Bruce Hanson* >> >> Software Engineer >> >> HERE Predictive Analytics >> >> >> >> *HERE Seattle* >> >> 701 Pike St, Suite 2000, Seattle, WA 98101 >> >> *47° 36' 41" N. 122° 19' 57" W >> <http://here.com/usa/seattle/98101/pike-st/701/map=47.611439,-122.332741,17/title=HERE%20Seattle%20-%20701%20Pike%20Street>* >> >> >> >> <http://360.here.com/> <https://twitter.com/here> >> <https://www.facebook.com/here> >> <https://linkedin.com/company/heremaps> >> <https://www.instagram.com/here> >> >> >> >> > > > -- > > Jamie Grier > data Artisans, Director of Applications Engineering > @jamiegrier <https://twitter.com/jamiegrier> > ja...@data-artisans.com > >
