Attila Sasvari created OOZIE-2944:
-------------------------------------
Summary: Shell action example does not work with Oozie on Yarn on
hadoop 2.6
Key: OOZIE-2944
URL: https://issues.apache.org/jira/browse/OOZIE-2944
Project: Oozie
Issue Type: Bug
Reporter: Attila Sasvari
I tried to execute example shell action with a pseudo hadoop 2.6.0. The
workflow failed with the exception below :
{code}
$ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId
application_1497357254962_0005
17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032
17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Container: container_1497357254962_0005_01_000001 on 172.30.64.221_55437
==========================================================================
LogType:stderr
Log Upload Time:13-Jun-2017 15:58:45
LogLength:1397
Log Contents:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapred/Mapper
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 15 more
LogType:stdout
Log Upload Time:13-Jun-2017 15:58:45
LogLength:2152
Log Contents:
Submitter user is: asasvari
log4j: Trying to find [container-log4j.properties] using context classloader
sun.misc.Launcher$AppClassLoader@6bc7c054.
log4j: Using URL
[jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties]
for automatic log4j configuration.
log4j: Reading configuration from URL
jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties
log4j: Hierarchy threshold set to [ALL].
log4j: Parsing for [root] with value=[INFO,CLA, EventCounter].
log4j: Level token is [INFO].
log4j: Category root set to INFO
log4j: Parsing appender named "CLA".
log4j: Parsing layout options for "CLA".
log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n].
log4j: End of parsing for "CLA".
log4j: Setting property [containerLogDir] to
[/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_000001].
log4j: Setting property [totalLogFileSize] to [0].
log4j: setFile called:
/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_000001/syslog,
true
log4j: setFile ended
log4j: Parsed "CLA" options.
log4j: Parsing appender named "EventCounter".
log4j: Parsed "EventCounter" options.
log4j: Finished configuring.
Using login user for UGI
Launcher AM configuration loaded
Starting the execution of prepare actions
Completed the execution of prepare actions successfully
Failing Oozie Launcher, null
Oozie Launcher, uploading action data to HDFS sequence file:
hdfs://localhost:9000/user/asasvari/oozie-asas/0000002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq
Stopping AM
Callback notification attempts left 0
Callback notification trying
http://0.0.0.0:11000/oozie/callback?id=0000002-170613150915408-oozie-asas-W@shell-node&status=FAILED
Callback notification to
http://0.0.0.0:11000/oozie/callback?id=0000002-170613150915408-oozie-asas-W@shell-node&status=FAILED
succeeded
Callback notification succeeded
LogType:syslog
Log Upload Time:13-Jun-2017 15:58:45
LogLength:1215
Log Contents:
2017-06-13 15:58:43,105 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2017-06-13 15:58:43,271 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at /0.0.0.0:8030
2017-06-13 15:58:43,844 INFO [main] org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor [.deflate]
2017-06-13 15:58:43,972 INFO [main]
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl: Waiting for application
to be successfully unregistered.
2017-06-13 15:58:44,078 INFO [AMRM Callback Handler Thread]
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl: Interrupted
while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
{code}
I believe this issue is related to
[OOZIE-2918|https://issues.apache.org/jira/browse/OOZIE-2918].
{{ShellActionExecutor}} extends {{JavaActionExecutor}} whose
{{needToAddMapReduceToClassPath()}} returns {{false}}.
{{LauncherMapper.getMainArguments()}} is called from {{LauncherAM}} via its
{{run()}} method. {{LauncherMapper}} extends
{{org.apache.hadoop.mapred.Mapper}} that is not on the classpath.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
