[jira] [Commented] (OOZIE-2941) Old map-reduce workflow example can't work with OYA

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396795#comment-16396795
 ] 

Andras Piros commented on OOZIE-2941:
-

[~Dongying Jiao] can you please re-test the issue after installing the Oozie 
sharelib? Thanks!

> Old map-reduce workflow example can't work with OYA
> ---
>
> Key: OOZIE-2941
> URL: https://issues.apache.org/jira/browse/OOZIE-2941
> Project: Oozie
>  Issue Type: Sub-task
>  Components: workflow
>Affects Versions: trunk
>Reporter: Dongying Jiao
>Priority: Major
> Attachments: RM_UI.jpg
>
>
> map-reduce workflow in Oozie examples run failed after OYA merged. The 
> exception in yarn log is:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/mapred/Mapper
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540)
>   at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170)
>   at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 15 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2941) Old map-reduce workflow example can't work with OYA

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054450#comment-16054450
 ] 

Peter Bacsko commented on OOZIE-2941:
-

So what's the status of this?

Is there a consensus on whether it's a bug or not? 

> Old map-reduce workflow example can't work with OYA
> ---
>
> Key: OOZIE-2941
> URL: https://issues.apache.org/jira/browse/OOZIE-2941
> Project: Oozie
>  Issue Type: Sub-task
>  Components: workflow
>Affects Versions: trunk
>Reporter: Dongying Jiao
> Attachments: RM_UI.jpg
>
>
> map-reduce workflow in Oozie examples run failed after OYA merged. The 
> exception in yarn log is:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/mapred/Mapper
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540)
>   at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170)
>   at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 15 more



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [jira] [Commented] (OOZIE-2941) Old map-reduce workflow example can't work with OYA

[Attila Sasvari]

Dong Ying, I have not changed anything. I have built oozie from source with
- mvn  clean install  assembly:single -DskipTests  -Dhadoop.version-2.6.0
-Puber

Note: -Puber is used so that dependencies of the Oozie server are copied

Then
- cd distro/target/oozie-5.0.0-SNAPSHOT

- I made sure that my hadoop configuration (e.g. fs.defaultFS points to the
right namenode in core-site.xml in conf/hadoop-conf ) was correct (see
https://oozie.apache.org/docs/4.2.0/AG_HadoopConfiguration.html)

- Started  Oozie server
bin/oozied.sh start

- Installed Sharelib
bin/oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib
oozie-sharelib-5.0.0-SNAPSHOT.tar.gz   -concurrency 4

- Updated Sharelib
bin/oozie admin -sharelibupdate

- Extracted, uploaded examples to HDFS
- Submitted map-reduce example
bin/oozie job -oozie http://localhost:11000/oozie -config
examples/apps/map-reduce/job.properties -run
-DnameNode=hdfs://localhost:9000 -DjobTracker=localhost:8032

(I am working with a pseudo, single-node, Hadoop 2.6 cluster in my dev
environment)

JavaActionExecutor should add MR to classpath in case of a MapReduce action:

https://github.com/apache/oozie/blob/83d4ddf45aa16649bd9fae367fa915379d5781cd/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1091

In my debugger I could see the following env was added to launcherJobConf:

{CLASSPATH=$PWD:$PWD/*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*}

It is later added to appContext used for submitting the application to YARN
-
https://github.com/apache/oozie/blob/83d4ddf45aa16649bd9fae367fa915379d5781cd/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1028


On Wed, Jun 14, 2017 at 5:39 AM, Dongying Jiao 
wrote:
>
> Hi Attilia:
> I have Oozie share lib, it is setup during oozie installation, but oozie
> share lib doesn't have map-reduce folder. Did you modify anything to make
> the example map-reduce workflow run successfully?
>
> Peter:
> About the commit link you mentioned, I see MR jars are added from
> map-reduce property:“mapreduce.application.classpath”, the classpath in my
> cluster is:
>
/etc/hadoop/conf/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/etc/hadoop/conf/secure:/usr/lib/hadoop-lzo/lib/*:/usr/iop/current/ext/hadoop/*
>
> Not sure if my map-reduce classpath is not corrcet.
> I used the 08/Jun build in oozie master branch.
>
> Thanks for help.
>
> Best Regards,
> Dong Ying
>
>
>
> 2017-06-13 23:27 GMT+08:00 Attila Sasvari (JIRA) :
>
> >
> > [ https://issues.apache.org/jira/browse/OOZIE-2941?page=
> > com.atlassian.jira.plugin.system.issuetabpanels:comment-
> > tabpanel=16048022#comment-16048022 ]
> >
> > Attila Sasvari commented on OOZIE-2941:
> > ---
> >
> > [~Dongying Jiao] I re-tested the mapreduce example in my local
> > environment, and the workflow successfully finished. Before running the
MR
> > workflow, you should first install Oozie sharelib and make sure jhs is
> > running. As per your previous commend, I suspect sharelib was not
installed
> > in your case. So I believe this issue is not the same as my experience
with
> > the shell action example.
> >
> > > Old map-reduce workflow example can't work with OYA
> > > ---
> > >
> > > Key: OOZIE-2941
> > > URL: https://issues.apache.org/jira/browse/OOZIE-2941
> > > Project: Oozie
> > >  Issue Type: Sub-task
> > >  Components: workflow
> > >Affects Versions: trunk
> > >Reporter: Dongying Jiao
> > > Attachments: RM_UI.jpg
> > >
> > >
> > > map-reduce workflow in Oozie examples run failed after OYA merged. The
> > exception in yarn log is:
> > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/apache/hadoop/mapred/Mapper
> > >   at java.lang.ClassLoader.defineClass1(Native Method)
> > >   at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> > >   at java.security.SecureClassLoader.defineClass(
> > SecureClassLoader.java:142)
> > >   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > >   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > >   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > >   at 

Re: [jira] [Commented] (OOZIE-2941) Old map-reduce workflow example can't work with OYA

[Dongying Jiao]

Hi Attilia:
I have Oozie share lib, it is setup during oozie installation, but oozie
share lib doesn't have map-reduce folder. Did you modify anything to make
the example map-reduce workflow run successfully?

Peter:
About the commit link you mentioned, I see MR jars are added from
map-reduce property:“mapreduce.application.classpath”, the classpath in my
cluster is:
/etc/hadoop/conf/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/etc/hadoop/conf/secure:/usr/lib/hadoop-lzo/lib/*:/usr/iop/current/ext/hadoop/*

Not sure if my map-reduce classpath is not corrcet.
I used the 08/Jun build in oozie master branch.

Thanks for help.

Best Regards,
Dong Ying



2017-06-13 23:27 GMT+08:00 Attila Sasvari (JIRA) :

>
> [ https://issues.apache.org/jira/browse/OOZIE-2941?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel=16048022#comment-16048022 ]
>
> Attila Sasvari commented on OOZIE-2941:
> ---
>
> [~Dongying Jiao] I re-tested the mapreduce example in my local
> environment, and the workflow successfully finished. Before running the MR
> workflow, you should first install Oozie sharelib and make sure jhs is
> running. As per your previous commend, I suspect sharelib was not installed
> in your case. So I believe this issue is not the same as my experience with
> the shell action example.
>
> > Old map-reduce workflow example can't work with OYA
> > ---
> >
> > Key: OOZIE-2941
> > URL: https://issues.apache.org/jira/browse/OOZIE-2941
> > Project: Oozie
> >  Issue Type: Sub-task
> >  Components: workflow
> >Affects Versions: trunk
> >Reporter: Dongying Jiao
> > Attachments: RM_UI.jpg
> >
> >
> > map-reduce workflow in Oozie examples run failed after OYA merged. The
> exception in yarn log is:
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapred/Mapper
> >   at java.lang.ClassLoader.defineClass1(Native Method)
> >   at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> >   at java.security.SecureClassLoader.defineClass(
> SecureClassLoader.java:142)
> >   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> >   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> >   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> >   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> >   at java.security.AccessController.doPrivileged(Native Method)
> >   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> >   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> >   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >   at org.apache.oozie.action.hadoop.LauncherAM.
> getMainArguments(LauncherAM.java:540)
> >   at org.apache.oozie.action.hadoop.LauncherAM.run(
> LauncherAM.java:170)
> >   at org.apache.oozie.action.hadoop.LauncherAM.main(
> LauncherAM.java:148)
> > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.
> Mapper
> >   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> >   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >   ... 15 more
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)
>

[jira] [Commented] (OOZIE-2941) Old map-reduce workflow example can't work with OYA

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048022#comment-16048022
 ] 

Attila Sasvari commented on OOZIE-2941:
---

[~Dongying Jiao] I re-tested the mapreduce example in my local environment, and 
the workflow successfully finished. Before running the MR workflow, you should 
first install Oozie sharelib and make sure jhs is running. As per your previous 
commend, I suspect sharelib was not installed in your case. So I believe this 
issue is not the same as my experience with the shell action example.

> Old map-reduce workflow example can't work with OYA
> ---
>
> Key: OOZIE-2941
> URL: https://issues.apache.org/jira/browse/OOZIE-2941
> Project: Oozie
>  Issue Type: Sub-task
>  Components: workflow
>Affects Versions: trunk
>Reporter: Dongying Jiao
> Attachments: RM_UI.jpg
>
>
> map-reduce workflow in Oozie examples run failed after OYA merged. The 
> exception in yarn log is:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/mapred/Mapper
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540)
>   at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170)
>   at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 15 more



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2941) Old map-reduce workflow example can't work with OYA

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047989#comment-16047989
 ] 

Attila Sasvari commented on OOZIE-2941:
---

[~Dongying Jiao], [~gezapeti] Today I experienced the same issue with the shell 
action example (I was using a pseudo hadoop 2.6.0 for my test).

{code}
$ ~/workspace/pseudo/hadoop-2.6.0/bin/yarn logs --applicationId 
application_1497357254962_0005

17/06/13 15:59:09 INFO client.RMProxy: Connecting to ResourceManager at 
/0.0.0.0:8032
17/06/13 15:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable

Container: container_1497357254962_0005_01_01 on 172.30.64.221_55437
==

LogType:stderr
Log Upload Time:13-Jun-2017 15:58:45
LogLength:1397
Log Contents:

Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/hadoop/mapred/Mapper

at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at 
org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 15 more

LogType:stdout
Log Upload Time:13-Jun-2017 15:58:45
LogLength:2152
Log Contents:

Submitter user is: asasvari

log4j: Trying to find [container-log4j.properties] using context classloader 
sun.misc.Launcher$AppClassLoader@6bc7c054.
log4j: Using URL 
[jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties]
 for automatic log4j configuration.
log4j: Reading configuration from URL 
jar:file:/Users/asasvari/workspace/pseudo/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.0.jar!/container-log4j.properties
log4j: Hierarchy threshold set to [ALL].
log4j: Parsing for [root] with value=[INFO,CLA, EventCounter].
log4j: Level token is [INFO].
log4j: Category root set to INFO
log4j: Parsing appender named "CLA".
log4j: Parsing layout options for "CLA".
log4j: Setting property [conversionPattern] to [%d{ISO8601} %p [%t] %c: %m%n].
log4j: End of parsing for "CLA".
log4j: Setting property [containerLogDir] to 
[/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01].
log4j: Setting property [totalLogFileSize] to [0].
log4j: setFile called: 
/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1497357254962_0005/container_1497357254962_0005_01_01/syslog,
 true
log4j: setFile ended
log4j: Parsed "CLA" options.
log4j: Parsing appender named "EventCounter".
log4j: Parsed "EventCounter" options.
log4j: Finished configuring.
Using login user for UGI
Launcher AM configuration loaded

Starting the execution of prepare actions
Completed the execution of prepare actions successfully

Failing Oozie Launcher, null

Oozie Launcher, uploading action data to HDFS sequence file: 
hdfs://localhost:9000/user/asasvari/oozie-asas/002-170613150915408-oozie-asas-W/shell-node--shell/action-data.seq

Stopping AM
Callback notification attempts left 0
Callback notification trying 
http://0.0.0.0:11000/oozie/callback?id=002-170613150915408-oozie-asas-W@shell-node=FAILED
Callback notification to 
http://0.0.0.0:11000/oozie/callback?id=002-170613150915408-oozie-asas-W@shell-node=FAILED
 succeeded

Callback notification succeeded

LogType:syslog
Log Upload Time:13-Jun-2017 15:58:45
LogLength:1215
Log Contents:

2017-06-13 15:58:43,105 WARN [main] org.apache.hadoop.util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... using builtin-java 
classes where applicable
2017-06-13 15:58:43,271 INFO [main] org.apache.hadoop.yarn.client.RMProxy: 
Connecting to ResourceManager at /0.0.0.0:8030
2017-06-13 15:58:43,844 INFO [main] 

[jira] [Commented] (OOZIE-2941) Old map-reduce workflow example can't work with OYA

[Peter Cseh (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047951#comment-16047951
 ] 

Peter Cseh commented on OOZIE-2941:
---

Thanks for looking into this!

This failure is strange. The MR action should have the MR jars on it's 
classpath just like it had before.
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1090
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java#L346


> Old map-reduce workflow example can't work with OYA
> ---
>
> Key: OOZIE-2941
> URL: https://issues.apache.org/jira/browse/OOZIE-2941
> Project: Oozie
>  Issue Type: Sub-task
>  Components: workflow
>Affects Versions: trunk
>Reporter: Dongying Jiao
> Attachments: RM_UI.jpg
>
>
> map-reduce workflow in Oozie examples run failed after OYA merged. The 
> exception in yarn log is:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/mapred/Mapper
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540)
>   at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170)
>   at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 15 more



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2941) Old map-reduce workflow example can't work with OYA

[Dongying Jiao (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047357#comment-16047357
 ] 

Dongying Jiao commented on OOZIE-2941:
--

This is due to lack of map-reduce lib. My WA is:
Create map-reduce folder in Oozie sharelib on HDFS, for example, 
/user/oozie/share/lib/lib_{timestamp}/map-reduce.
Put all map-reduce jars in hadoop map-reduce lib to this folder.
Add below property in workflow:

  oozie.action.sharelib.for.map-reduce
  map-reduce

Set oozie.use.system.libpath=true in job.properties.
Then the map-reduce workflow example can run successfully.
But seen from the yarn resource manager UI, the second application name is N/A. 
I attach the screenshot.





> Old map-reduce workflow example can't work with OYA
> ---
>
> Key: OOZIE-2941
> URL: https://issues.apache.org/jira/browse/OOZIE-2941
> Project: Oozie
>  Issue Type: Sub-task
>  Components: workflow
>Affects Versions: trunk
>Reporter: Dongying Jiao
>
> map-reduce workflow in Oozie examples run failed after OYA merged. The 
> exception in yarn log is:
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/apache/hadoop/mapred/Mapper
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.apache.oozie.action.hadoop.LauncherAM.getMainArguments(LauncherAM.java:540)
>   at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:170)
>   at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:148)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.Mapper
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   ... 15 more



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)