Hi folks,
I searched through the archives and googled my brains out and I'm flat out
stuck. I'm hoping someone can help me here :) We're getting pretty desperate.
CDH 4.1.3
Oozie v3.2.0
Using MR2 (YARN)
We have a Java action configured in workflow.xml thusly:
<action name="convert-file">
<java>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<main-class>com.sdl.hlab.infrastructure.SM2ApiResultsFileToHdfsConverter</main-class>
<java-opts>-Djava.library.path=${sapNativePath}
-Dcom.sdl.sap.conf.directory="." -Xmx1900m
-Dlogback.configurationFile="conf/${SAP_ENV}/log4j.xml"
-Dlog4j.configuration="conf/${SAP_ENV}/log4j.xml"</java-opts>
<arg>${filePath}</arg>
<file>files/sap.properties</file>
</java>
<ok to="cleanup"/>
<error to="convert-file-Email"/>
</action>
This action works perfectly on two different environments (all same CDH and
Oozie version), but not on our new third environment. In the first two
environments, the java-opts block is respected and I can see it in the Job
History logs in stdout under "Java System Properties". In the third environment
(which is new) none of the java-opts appear except for java.library.path, which
points to the wrong libraries.
I checked everywhere for entries for mapred.child.java.opts and there appears
to be one entry buried in mapred-defaults.xml and that's it.
I also tried to move the java opts to a property entry, no luck.
The job log doesn't have anything interesting:
2013-12-19 00:41:05,419 INFO ActionStartXCommand:539 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@:start:] Start action
[0000012-131218165312793-oozie-oozi-W@:start:] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-12-19 00:41:05,419 WARN ActionStartXCommand:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@:start:]
[***0000012-131218165312793-oozie-oozi-W@:start:***]Action status=DONE
2013-12-19 00:41:05,419 WARN ActionStartXCommand:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@:start:]
[***0000012-131218165312793-oozie-oozi-W@:start:***]Action updated in DB!
2013-12-19 00:41:05,561 INFO ActionStartXCommand:539 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] Start action
[0000012-131218165312793-oozie-oozi-W@convert-file] with user-retry state :
userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-12-19 00:41:07,226 WARN JavaActionExecutor:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] credentials is null
for the action
2013-12-19 00:41:09,812 INFO JavaActionExecutor:539 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] checking action,
external ID [job_1387408789132_0015] status [RUNNING]
2013-12-19 00:41:09,815 WARN ActionStartXCommand:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file]
[***0000012-131218165312793-oozie-oozi-W@convert-file***]Action status=RUNNING
2013-12-19 00:41:09,815 WARN ActionStartXCommand:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file]
[***0000012-131218165312793-oozie-oozi-W@convert-file***]Action updated in DB!
2013-12-19 00:41:28,470 INFO CallbackServlet:539 - USER[-] GROUP[-] TOKEN[-]
APP[-] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] callback for action
[0000012-131218165312793-oozie-oozi-W@convert-file]
2013-12-19 00:41:28,755 INFO JavaActionExecutor:539 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] action completed,
external ID [job_1387408789132_0015]
2013-12-19 00:41:28,787 WARN JavaActionExecutor:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] Launcher ERROR,
reason: Main class [our main class], main() threw exception,
conf/qa/our.properties (No such file or directory)
2013-12-19 00:41:28,787 WARN JavaActionExecutor:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] Launcher exception:
conf/qa/our.properties (No such file or directory)
java.io.FileNotFoundException: conf/qa/our.properties (No such file or
directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at java.io.FileInputStream.<init>(FileInputStream.java:79)
at
com.sdl.sap.util.BaseConfiguration.configurationFile(Configuration.scala:78)
at
com.sdl.sap.util.BaseConfiguration.loadProperties(Configuration.scala:88)
at
com.sdl.sap.util.BaseConfiguration.properties$lzycompute(Configuration.scala:38)
at
com.sdl.sap.util.BaseConfiguration.properties(Configuration.scala:38)
at
com.sdl.sap.util.BaseConfiguration.stringValue(Configuration.scala:49)
at
com.sdl.hlab.infrastructure.SM2ApiResultsFileToHdfsConverter$.buildFileInfo(SM2ApiResultsFileToHdfsConverter.scala:74)
at
com.sdl.hlab.infrastructure.SM2ApiResultsFileToHdfsConverter$.main(SM2ApiResultsFileToHdfsConverter.scala:68)
at
com.sdl.hlab.infrastructure.SM2ApiResultsFileToHdfsConverter.main(SM2ApiResultsFileToHdfsConverter.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:472)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
2013-12-19 00:41:28,864 INFO ActionEndXCommand:539 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] ERROR is considered
as FAILED for SLA
2013-12-19 00:41:29,021 INFO ActionStartXCommand:539 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-Email] Start action
[0000012-131218165312793-oozie-oozi-W@convert-file-Email] with user-retry state
: userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-12-19 00:41:29,112 WARN ActionStartXCommand:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-Email]
[***0000012-131218165312793-oozie-oozi-W@convert-file-Email***]Action
status=DONE
2013-12-19 00:41:29,113 WARN ActionStartXCommand:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-Email]
[***0000012-131218165312793-oozie-oozi-W@convert-file-Email***]Action updated
in DB!
2013-12-19 00:41:29,259 INFO ActionStartXCommand:539 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-fail] Start action
[0000012-131218165312793-oozie-oozi-W@convert-file-fail] with user-retry state
: userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-12-19 00:41:29,260 WARN ActionStartXCommand:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-fail]
[***0000012-131218165312793-oozie-oozi-W@convert-file-fail***]Action status=DONE
2013-12-19 00:41:29,260 WARN ActionStartXCommand:542 - USER[hdfs] GROUP[-]
TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W]
ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-fail]
[***0000012-131218165312793-oozie-oozi-W@convert-file-fail***]Action updated in
DB!
2013-12-19 00:41:29,440 WARN CoordActionUpdateXCommand:542 - USER[hdfs]
GROUP[-] TOKEN[] APP[hlab-oozie-workflow]
JOB[0000012-131218165312793-oozie-oozi-W] ACTION[-] E1100: Command precondition
does not hold before execution, [, coord action is null], Error Code: E1100
The FIleNotFoundException is the result of the property
com.sdl.sap.conf.directory not being set.
Any help would be really appreciated!
</pre><font face="arial" size="2" color="#736F6E">
<a
href="http://www.sdl.com/?utm_source=Email&utm_medium=Email%2BSignature&utm_campaign=SDL%2BStandard%2BEmail%2BSignature";>
<img
src="http://www.sdl.com/Content/themes/common/images/SDL_logo_strapline_GCEM_EmailSig_150x68px.jpg";
border=0><br><br>www.sdl.com
</a><br><br>
<font face="arial" size="1" color="#736F6E">
<b>SDL PLC confidential, all rights reserved.</b>
If you are not the intended recipient of this mail SDL requests and requires
that you delete it without acting upon or copying any of its contents,
and we further request that you advise us.<BR>
SDL PLC is a public limited company registered in England and Wales.
Registered number: 02675207.<BR>
Registered address: Globe House, Clivemont Road, Maidenhead, Berkshire SL6 7DY,
UK.
</font>
This message has been scanned for malware by Websense. www.websense.com
![http://www.sdl.com/Content/themes/common/images/SDL_logo_strapline_GCEM_EmailSig_150x68px.jpg"](http://www.sdl.com/Content/themes/common/images/SDL_logo_strapline_GCEM_EmailSig_150x68px.jpg"){kind=link}