What Robert said. It can especially happen if you use Cloudera Manager in your environment, and the CM -> MapReduce configs have "Client Overrides" set for things like heap sizes of tasks. This administrative enforcement settings could unfortunately cause any overrides of the mapred.child.java.opts (and others) passed by clients to be ignored by the TaskTrackers.
On Fri, Dec 20, 2013 at 6:28 AM, Robert Kanter <[email protected]> wrote: > I remember seeing something similar with java-opts where they were being > set somewhere else (mapred-site?) and declared as final, so when Oozie > tried to set them, it was ignored. Perhaps you’re running into the same > problem? > > > On Thu, Dec 19, 2013 at 12:23 AM, Jose Fernandez <[email protected]> wrote: > >> Hi folks, >> >> I searched through the archives and googled my brains out and I'm flat out >> stuck. I'm hoping someone can help me here :) We're getting pretty >> desperate. >> >> CDH 4.1.3 >> Oozie v3.2.0 >> Using MR2 (YARN) >> >> We have a Java action configured in workflow.xml thusly: >> >> <action name="convert-file"> >> <java> >> <job-tracker>${jobTracker}</job-tracker> >> <name-node>${nameNode}</name-node> >> <configuration> >> <property> >> <name>mapred.job.queue.name</name> >> <value>${queueName}</value> >> </property> >> </configuration> >> >> <main-class>com.sdl.hlab.infrastructure.SM2ApiResultsFileToHdfsConverter</main-class> >> <java-opts>-Djava.library.path=${sapNativePath} >> -Dcom.sdl.sap.conf.directory="." -Xmx1900m >> -Dlogback.configurationFile="conf/${SAP_ENV}/log4j.xml" >> -Dlog4j.configuration="conf/${SAP_ENV}/log4j.xml"</java-opts> >> <arg>${filePath}</arg> >> <file>files/sap.properties</file> >> </java> >> <ok to="cleanup"/> >> <error to="convert-file-Email"/> >> </action> >> >> This action works perfectly on two different environments (all same CDH >> and Oozie version), but not on our new third environment. In the first two >> environments, the java-opts block is respected and I can see it in the Job >> History logs in stdout under "Java System Properties". In the third >> environment (which is new) none of the java-opts appear except for >> java.library.path, which points to the wrong libraries. >> >> I checked everywhere for entries for mapred.child.java.opts and there >> appears to be one entry buried in mapred-defaults.xml and that's it. >> >> I also tried to move the java opts to a property entry, no luck. >> >> The job log doesn't have anything interesting: >> >> 2013-12-19 00:41:05,419 INFO ActionStartXCommand:539 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@:start:] Start action >> [0000012-131218165312793-oozie-oozi-W@:start:] with user-retry state : >> userRetryCount [0], userRetryMax [0], userRetryInterval [10] >> 2013-12-19 00:41:05,419 WARN ActionStartXCommand:542 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@:start:] >> [***0000012-131218165312793-oozie-oozi-W@:start:***]Action status=DONE >> 2013-12-19 00:41:05,419 WARN ActionStartXCommand:542 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@:start:] >> [***0000012-131218165312793-oozie-oozi-W@:start:***]Action updated in DB! >> 2013-12-19 00:41:05,561 INFO ActionStartXCommand:539 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] Start action >> [0000012-131218165312793-oozie-oozi-W@convert-file] with user-retry state >> : userRetryCount [0], userRetryMax [0], userRetryInterval [10] >> 2013-12-19 00:41:07,226 WARN JavaActionExecutor:542 - USER[hdfs] GROUP[-] >> TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] credentials is >> null for the action >> 2013-12-19 00:41:09,812 INFO JavaActionExecutor:539 - USER[hdfs] GROUP[-] >> TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] checking >> action, external ID [job_1387408789132_0015] status [RUNNING] >> 2013-12-19 00:41:09,815 WARN ActionStartXCommand:542 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] >> [***0000012-131218165312793-oozie-oozi-W@convert-file***]Action >> status=RUNNING >> 2013-12-19 00:41:09,815 WARN ActionStartXCommand:542 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] >> [***0000012-131218165312793-oozie-oozi-W@convert-file***]Action updated >> in DB! >> 2013-12-19 00:41:28,470 INFO CallbackServlet:539 - USER[-] GROUP[-] >> TOKEN[-] APP[-] JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] callback for >> action [0000012-131218165312793-oozie-oozi-W@convert-file] >> 2013-12-19 00:41:28,755 INFO JavaActionExecutor:539 - USER[hdfs] GROUP[-] >> TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] action >> completed, external ID [job_1387408789132_0015] >> 2013-12-19 00:41:28,787 WARN JavaActionExecutor:542 - USER[hdfs] GROUP[-] >> TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] Launcher ERROR, >> reason: Main class [our main class], main() threw exception, >> conf/qa/our.properties (No such file or directory) >> 2013-12-19 00:41:28,787 WARN JavaActionExecutor:542 - USER[hdfs] GROUP[-] >> TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] Launcher >> exception: conf/qa/our.properties (No such file or directory) >> java.io.FileNotFoundException: conf/qa/our.properties (No such file or >> directory) >> at java.io.FileInputStream.open(Native Method) >> at java.io.FileInputStream.<init>(FileInputStream.java:120) >> at java.io.FileInputStream.<init>(FileInputStream.java:79) >> at >> com.sdl.sap.util.BaseConfiguration.configurationFile(Configuration.scala:78) >> at >> com.sdl.sap.util.BaseConfiguration.loadProperties(Configuration.scala:88) >> at >> com.sdl.sap.util.BaseConfiguration.properties$lzycompute(Configuration.scala:38) >> at >> com.sdl.sap.util.BaseConfiguration.properties(Configuration.scala:38) >> at >> com.sdl.sap.util.BaseConfiguration.stringValue(Configuration.scala:49) >> at >> com.sdl.hlab.infrastructure.SM2ApiResultsFileToHdfsConverter$.buildFileInfo(SM2ApiResultsFileToHdfsConverter.scala:74) >> at >> com.sdl.hlab.infrastructure.SM2ApiResultsFileToHdfsConverter$.main(SM2ApiResultsFileToHdfsConverter.scala:68) >> at >> com.sdl.hlab.infrastructure.SM2ApiResultsFileToHdfsConverter.main(SM2ApiResultsFileToHdfsConverter.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:472) >> at >> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) >> at >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) >> at >> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) >> at java.security.AccessController.doPrivileged(Native >> Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) >> at >> org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) >> >> 2013-12-19 00:41:28,864 INFO ActionEndXCommand:539 - USER[hdfs] GROUP[-] >> TOKEN[] APP[hlab-oozie-workflow] JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file] ERROR is >> considered as FAILED for SLA >> 2013-12-19 00:41:29,021 INFO ActionStartXCommand:539 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-Email] Start >> action [0000012-131218165312793-oozie-oozi-W@convert-file-Email] with >> user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval >> [10] >> 2013-12-19 00:41:29,112 WARN ActionStartXCommand:542 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-Email] >> [***0000012-131218165312793-oozie-oozi-W@convert-file-Email***]Action >> status=DONE >> 2013-12-19 00:41:29,113 WARN ActionStartXCommand:542 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-Email] >> [***0000012-131218165312793-oozie-oozi-W@convert-file-Email***]Action >> updated in DB! >> 2013-12-19 00:41:29,259 INFO ActionStartXCommand:539 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-fail] Start >> action [0000012-131218165312793-oozie-oozi-W@convert-file-fail] with >> user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval >> [10] >> 2013-12-19 00:41:29,260 WARN ActionStartXCommand:542 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-fail] >> [***0000012-131218165312793-oozie-oozi-W@convert-file-fail***]Action >> status=DONE >> 2013-12-19 00:41:29,260 WARN ActionStartXCommand:542 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] >> ACTION[0000012-131218165312793-oozie-oozi-W@convert-file-fail] >> [***0000012-131218165312793-oozie-oozi-W@convert-file-fail***]Action >> updated in DB! >> 2013-12-19 00:41:29,440 WARN CoordActionUpdateXCommand:542 - USER[hdfs] >> GROUP[-] TOKEN[] APP[hlab-oozie-workflow] >> JOB[0000012-131218165312793-oozie-oozi-W] ACTION[-] E1100: Command >> precondition does not hold before execution, [, coord action is null], >> Error Code: E1100 >> >> The FIleNotFoundException is the result of the property >> com.sdl.sap.conf.directory not being set. >> >> Any help would be really appreciated! >> </pre><font face="arial" size="2" color="#736F6E"> >> >> >> >> <a href=" >> http://www.sdl.com/?utm_source=Email&utm_medium=Email%2BSignature&utm_campaign=SDL%2BStandard%2BEmail%2BSignature >> "> >> <img src=" >> http://www.sdl.com/Content/themes/common/images/SDL_logo_strapline_GCEM_EmailSig_150x68px.jpg"; >> border=0><br><br>www.sdl.com >> </a><br><br> >> >> <font face="arial" size="1" color="#736F6E"> >> >> <b>SDL PLC confidential, all rights reserved.</b> >> >> If you are not the intended recipient of this mail SDL requests and >> requires that you delete it without acting upon or copying any of its >> contents, >> and we further request that you advise us.<BR> >> SDL PLC is a public limited company registered in England and Wales. >> Registered number: 02675207.<BR> >> Registered address: Globe House, Clivemont Road, Maidenhead, Berkshire SL6 >> 7DY, UK. >> </font> >> >> >> This message has been scanned for malware by Websense. www.websense.com >> -- Harsh J
![http://www.sdl.com/Content/themes/common/images/SDL_logo_strapline_GCEM_EmailSig_150x68px.jpg"](http://www.sdl.com/Content/themes/common/images/SDL_logo_strapline_GCEM_EmailSig_150x68px.jpg"){kind=link}