Cool On 14-Jan-2016 13:06, "Asanka Sanjaya Herath" <[email protected]> wrote:
> Hi, > > I think I figure out the problem. It is a permission issue. When I look the > yarn application log, I see following error. > > E0507: Could not access to > [hdfs://xx.xxx.xx.xx8020/user/hdfs/email-dispatcher-tenant-manager-1.0/], > Permission denied: user=yarn, access=EXECUTE, > inode="/user/hdfs/email-dispatcher-tenant-manager-1.0":hdfs:hdfs:drwx------ > at > > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) > at > > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259) > at > > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205) > at > > org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:300) > at > > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) > at > > org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1771) > at > > org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3857) > at > > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1012) > at > > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843) > at > > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137) at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2133) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:415) at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2131) > at > org.apache.oozie.client.OozieClient.handleError(OozieClient.java:612) > at > org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:695) > at > org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:665) > at > > org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:560) > at org.apache.oozie.client.OozieClient.run(OozieClient.java:850) > at > > com.sonasoft.sonacloud.email.dispatcher.manager.Utils.submitDispatcherJobForTenantToOozie(Utils.java:155) > at > > com.sonasoft.sonacloud.email.dispatcher.manager.Utils.spawnEmailDispatcherJobsPerTenant(Utils.java:43) > at > > com.sonasoft.sonacloud.email.dispatcher.manager.MainClass.main(MainClass.java:27) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56) > at > org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) > at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > End of LogType:stderr > > > > > On Wed, Jan 13, 2016 at 7:05 PM, Asanka Sanjaya Herath <[email protected] > > > wrote: > > > No. Only the child job runs using java API. The manager job is submitted > > to Oozie manually using the command line. The above provided workflow.xml > > is for the child job. > > > > On Wed, Jan 13, 2016 at 7:00 PM, Hanmay Udgiri < > [email protected]> > > wrote: > > > >> I guess these two jobs run using the java API. What about workflow.xml > of > >> child job? > >> On 13-Jan-2016 6:26 pm, "Asanka Sanjaya Herath" <[email protected]> > >> wrote: > >> > >> > There are basically two jobs. > >> > > >> > 1. manager job > >> > 2. child job > >> > > >> > > >> > Manager job is running on Oozie and responsible for creating new > >> instances > >> > of child jobs in Oozie when needed. > >> > > >> > I can see the manager job run in Oozie without giving any error. When > >> the > >> > submitDispatcherJobForTenantToOozie(String tenantId) method invoked in > >> > manager job, it should submit a new child job to Oozie. The method is > >> > invoking properly. But I cannot see any child job in Oozie console. > >> > > >> > But When I run the manager job directly in hadoop(Without using > Oozie), > >> it > >> > spawns new child jobs in Oozie properly. > >> > > >> > > >> > > >> > On Wed, Jan 13, 2016 at 6:20 PM, Hanmay Udgiri < > >> [email protected]> > >> > wrote: > >> > > >> > > So are u able to see the job running in HUE?.U can also view job in > >> the > >> > > oozie web console ? What about the workflow.xml of tenantapp?Is > >> > > The MainClass job able to call tenantapp workflow? > >> > > >> > On 13-Jan-2016 6:09 pm, "Asanka Sanjaya Herath" <[email protected]> > >> > wrote: > >> > > > >> > > > Java method to create a job in Oozie. > >> > > > > >> > > > private void submitDispatcherJobForTenantToOozie(String tenantId) > >> > > > throws OozieClientException { > >> > > > > >> > > > // get a OozieClient for local Oozie > >> > > > OozieClient wc = new > >> > > > > >> > > > >> > > >> > OozieClient(PropertyValuesProvider.getInstance().getPropValue("oozieURL")); > >> > > > > >> > > > // create a workflow job configuration and set the workflow > >> > > > application path > >> > > > Properties conf = wc.createConfiguration(); > >> > > > conf.setProperty(OozieClient.APP_PATH, > >> > > > PropertyValuesProvider.getInstance(). > >> > > > getPropValue("dispatcherTenantManagerAppPath")); > >> > > > > >> > > > // setting workflow parameters > >> > > > conf.setProperty("jobTracker", "sandbox.hortonworks.com:8050 > "); > >> > > > conf.setProperty("nameNode", "hdfs:// > >> sandbox.hortonworks.com:8020 > >> > "); > >> > > > conf.setProperty("queueName", "default"); > >> > > > conf.setProperty("appRoot", > >> "email-dispatcher-tenant-manager-1.0"); > >> > > > > >> > > > //set tenantId and retention period > >> > > > conf.setProperty("tenantId", tenantId); > >> > > > > >> > > > // submit and start the workflow job > >> > > > String jobId = wc.run(conf); > >> > > > } > >> > > > > >> > > > > >> > > > Workfolw.xml of child job: > >> > > > > >> > > > > >> > > > <workflow-app xmlns="uri:oozie:workflow:0.2" name="java-main-wf"> > >> > > > <start to="java-node"/> > >> > > > <action name="java-node"> > >> > > > <java> > >> > > > <job-tracker>${jobTracker}</job-tracker> > >> > > > <name-node>${nameNode}</name-node> > >> > > > <configuration> > >> > > > <property> > >> > > > <name>mapred.job.queue.name</name> > >> > > > <value>${queueName}</value> > >> > > > </property> > >> > > > </configuration> > >> > > > <main-class>com.MainClass</main-class> > >> > > > </java> > >> > > > <ok to="end"/> > >> > > > <error to="fail"/> > >> > > > </action> > >> > > > <kill name="fail"> > >> > > > <message>Java failed, error > >> > > > message[${wf:errorMessage(wf:lastErrorNode())}]</message> > >> > > > </kill> > >> > > > <end name="end"/> > >> > > > </workflow-app> > >> > > > > >> > > > > >> > > > > >> > > > On Wed, Jan 13, 2016 at 6:00 PM, Hanmay Udgiri < > >> > > [email protected]> > >> > > > wrote: > >> > > > > >> > > > > Can u share the workflow.xml or the java code? > >> > > > > On 13-Jan-2016 17:34, "Asanka Sanjaya Herath" < > [email protected] > >> > > >> > > > wrote: > >> > > > > > >> > > > > > Hi Hanmayya, > >> > > > > > > >> > > > > > I followed the given link but it failed to spawn new Oozie > jobs. > >> > But > >> > > > > when I > >> > > > > > run the same program directly using yarn jar command It spawns > >> the > >> > > jobs > >> > > > > in > >> > > > > > Oozie. The yarn log is not giving any errors too. > >> > > > > > > >> > > > > > On Wed, Jan 13, 2016 at 11:42 AM, Asanka Sanjaya Herath < > >> > > > > > [email protected]> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hi Hanmayya, > >> > > > > > > > >> > > > > > > Thank you for the response. I'll follow the given link and > >> update > >> > > > > status. > >> > > > > > > > >> > > > > > > > >> > > > > > >> Do you want to execute multiple Oozie workflows from the > Java > >> > > > class? > >> > > > > > > > >> > > > > > > > >> > > > > > > No I want to execute set of instances of one Oozie workflow. > >> > > > > > > > >> > > > > > > > >> > > > > > > On Wed, Jan 13, 2016 at 10:43 AM, Hanmay Udgiri < > >> > > > > > [email protected] > >> > > > > > > > wrote: > >> > > > > > > > >> > > > > > >> You can use OozieClient Java class. > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://www.hadooped.blogspot.in/2013/06/apache-oozie-part-8-java-api-for.html > >> > > > > > >> > >> > > > > > >> Do you want to execute multiple Oozie workflows from the > Java > >> > > class? > >> > > > > > >> > >> > > > > > >> Thanks > >> > > > > > >> Hanmayya > >> > > > > > >> > >> > > > > > >> On Wed, Jan 13, 2016 at 9:57 AM, Asanka Sanjaya Herath < > >> > > > > > >> [email protected]> > >> > > > > > >> wrote: > >> > > > > > >> > >> > > > > > >> > Hi, > >> > > > > > >> > > >> > > > > > >> > I'm writing a simple Java Oozie job. That job is supposed > >> to > >> > > spawn > >> > > > > new > >> > > > > > >> > Oozie jobs or directly submit new jobs to Hadoop. Is > there > >> a > >> > way > >> > > > to > >> > > > > > >> achieve > >> > > > > > >> > this using Oozie API? > >> > > > > > >> > > >> > > > > > >> > -- > >> > > > > > >> > Thanks, > >> > > > > > >> > Regards, > >> > > > > > >> > ASH > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> -- > >> > > > > > >> Thanks and Regards > >> > > > > > >> Hanmayya Udgiri > >> > > > > > >> > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > -- > >> > > > > > > Thanks, > >> > > > > > > Regards, > >> > > > > > > ASH > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > -- > >> > > > > > Thanks, > >> > > > > > Regards, > >> > > > > > ASH > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > -- > >> > > > Thanks, > >> > > > Regards, > >> > > > ASH > >> > > > > >> > > > >> > > >> > > >> > > >> > -- > >> > Thanks, > >> > Regards, > >> > ASH > >> > > >> > > > > > > > > -- > > Thanks, > > Regards, > > ASH > > > > > > -- > Thanks, > Regards, > ASH >
