[
https://issues.apache.org/jira/browse/OOZIE-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804883#comment-16804883
]
Jerry Chabot commented on OOZIE-3458:
-------------------------------------
Unfortunately, this means we need a workaround because EMR 5.22 has critical
HBase bug fix. Upgrading or patching Oozie within EMR 5.22 is not an option.
> Can no longer submit a coordinator job which has oozie.libpath and
> oozie.coord.application.path set
> ---------------------------------------------------------------------------------------------------
>
> Key: OOZIE-3458
> URL: https://issues.apache.org/jira/browse/OOZIE-3458
> Project: Oozie
> Issue Type: Bug
> Affects Versions: 5.1.0
> Reporter: Jerry Chabot
> Priority: Critical
>
> This bug is related to https://issues.apache.org/jira/browse/OOZIE-3457. Our
> Production environment is currently running Amazon EMR 5.19. We want to
> upgrade to EMR 5.22. Oozie will be upgraded from 5.0.0 to 5.1.0. Our
> deployment scripts can no longer submit coordinator jobs after upgrading a
> test environment. Here is a sample post from the oozie client.
> POST [http://hbase.qa.local:11000/oozie/v2/jobs?action=start]
> <?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
> <property><name>nameNode</name><value>hdfs://hbase.qa.local:8020</value></property>
> <property><name>oozie.use.system.libpath</name><value>true</value></property>
> <property><name>startTime</name><value>2018-03-28T00:00Z</value></property>
> <property><name>queueName</name><value>oozie</value></property>
> <property><name>jobTracker</name><value>hbase.qa.local:8032</value></property>
> <property><name>oozie.libpath</name><value>hdfs://hbase.qa.local:8020/user/oozie/cmx</value></property>
> <property><name>endTime</name><value>2020-01-01:00Z</value></property>
> <property><name>applicationConfigFile</name><value>hdfs://hbase.qa.local:8020/usr/share/cmx/conf/app.properties,hdfs://hbase.qa.local:8020/usr/share/cmx/conf/crypto.properties</value></property>
> <property><name>frequency</name><value>240</value></property>
> <property><name>user.name</name><value>oozie</value></property>
> <property><name>oozie.coord.application.path</name><value>hdfs://hbase.qa.local:8020/user/oozie/reporting/workspaceDimToVertica/</value></property>
> </configuration>
> Error: E0307 : E0307: Runtime error [App directory [null] does not exist and
> app definition cannot be created because of missing config value
> [oozie.jobs.api.generated.xml]]
> Note, our code specifies oozie.libpath and oozie.coord.application.path. Is
> this no longer allowed?
> I've narrowed down the problem to the checkAndWriteApplicationXMLToHDFS(...)
> method in V1JobsServlet.java. It is related to the new feature for submitting
> embedded XML within the job submission. This method is check whether the
> workflow.xml or coodinator.xml exists in either oozie.wf.application.path,
> oozie.libpath, oozie.coord.application.path or oozie.bundle.application.path.
> First, why is it checking oozie.libpath? That is not an application path.
> Second, it stops looking as soon as findAppPathsWithFileNames() returns a
> value. Which it will always do when oozie.libpath is set even if an xml
> doesn't exist there. It does not continue looking for an xml file in
> oozie.coord.application.path or oozie.bundle.application.path. I realize
> only one application path should be specified. IIMO, its the oozie.libpath
> check that is the problem.
> Third, the logic is inefficient. Here are two lines. Why call the same method
> again with the same params. You already have the result!
> if (!findAppPathsWithFileNames(appPath, "workflow.xml").isEmpty()) {
> appPathsWithFileNames = findAppPathsWithFileNames(appPath, "workflow.xml");
> }
> It is critical that we upgrade to EMR 5.22. However, we are blocked until we
> figure out a way to get the job submitted and completing successfully. Any
> suggested workarounds would be greatly appreciated. But, I am filing this bug
> to get it fixed so an upgraded Oozie will make it into a future EMR release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
