[jira] [Commented] (OOZIE-3458) Can no longer submit a coordinator job which has oozie.libpath and oozie.coord.application.path set
[
https://issues.apache.org/jira/browse/OOZIE-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805289#comment-16805289
]
Jerry Chabot commented on OOZIE-3458:
-
I've decided to use cron jobs to submit workflow jobs since we can no longer
submit coordinator jobs. I sure hope a test case was added to ensure this
scenario passes in future releases!
> Can no longer submit a coordinator job which has oozie.libpath and
> oozie.coord.application.path set
> ---
>
> Key: OOZIE-3458
> URL: https://issues.apache.org/jira/browse/OOZIE-3458
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Jerry Chabot
>Priority: Critical
>
> This bug is related to https://issues.apache.org/jira/browse/OOZIE-3457. Our
> Production environment is currently running Amazon EMR 5.19. We want to
> upgrade to EMR 5.22. Oozie will be upgraded from 5.0.0 to 5.1.0. Our
> deployment scripts can no longer submit coordinator jobs after upgrading a
> test environment. Here is a sample post from the oozie client.
> POST [http://hbase.qa.local:11000/oozie/v2/jobs?action=start]
>
> nameNodehdfs://hbase.qa.local:8020
> oozie.use.system.libpathtrue
> startTime2018-03-28T00:00Z
> queueNameoozie
> jobTrackerhbase.qa.local:8032
> oozie.libpathhdfs://hbase.qa.local:8020/user/oozie/cmx
> endTime2020-01-01:00Z
> applicationConfigFilehdfs://hbase.qa.local:8020/usr/share/cmx/conf/app.properties,hdfs://hbase.qa.local:8020/usr/share/cmx/conf/crypto.properties
> frequency240
> user.nameoozie
> oozie.coord.application.pathhdfs://hbase.qa.local:8020/user/oozie/reporting/workspaceDimToVertica/
>
> Error: E0307 : E0307: Runtime error [App directory [null] does not exist and
> app definition cannot be created because of missing config value
> [oozie.jobs.api.generated.xml]]
> Note, our code specifies oozie.libpath and oozie.coord.application.path. Is
> this no longer allowed?
> I've narrowed down the problem to the checkAndWriteApplicationXMLToHDFS(...)
> method in V1JobsServlet.java. It is related to the new feature for submitting
> embedded XML within the job submission. This method is check whether the
> workflow.xml or coodinator.xml exists in either oozie.wf.application.path,
> oozie.libpath, oozie.coord.application.path or oozie.bundle.application.path.
> First, why is it checking oozie.libpath? That is not an application path.
> Second, it stops looking as soon as findAppPathsWithFileNames() returns a
> value. Which it will always do when oozie.libpath is set even if an xml
> doesn't exist there. It does not continue looking for an xml file in
> oozie.coord.application.path or oozie.bundle.application.path. I realize
> only one application path should be specified. IIMO, its the oozie.libpath
> check that is the problem.
> Third, the logic is inefficient. Here are two lines. Why call the same method
> again with the same params. You already have the result!
> if (!findAppPathsWithFileNames(appPath, "workflow.xml").isEmpty()) {
> appPathsWithFileNames = findAppPathsWithFileNames(appPath, "workflow.xml");
> }
> It is critical that we upgrade to EMR 5.22. However, we are blocked until we
> figure out a way to get the job submitted and completing successfully. Any
> suggested workarounds would be greatly appreciated. But, I am filing this bug
> to get it fixed so an upgraded Oozie will make it into a future EMR release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OOZIE-3458) Can no longer submit a coordinator job which has oozie.libpath and oozie.coord.application.path set
[
https://issues.apache.org/jira/browse/OOZIE-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon resolved OOZIE-3458.
---
Resolution: Duplicate
> Can no longer submit a coordinator job which has oozie.libpath and
> oozie.coord.application.path set
> ---
>
> Key: OOZIE-3458
> URL: https://issues.apache.org/jira/browse/OOZIE-3458
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Jerry Chabot
>Priority: Critical
>
> This bug is related to https://issues.apache.org/jira/browse/OOZIE-3457. Our
> Production environment is currently running Amazon EMR 5.19. We want to
> upgrade to EMR 5.22. Oozie will be upgraded from 5.0.0 to 5.1.0. Our
> deployment scripts can no longer submit coordinator jobs after upgrading a
> test environment. Here is a sample post from the oozie client.
> POST [http://hbase.qa.local:11000/oozie/v2/jobs?action=start]
>
> nameNodehdfs://hbase.qa.local:8020
> oozie.use.system.libpathtrue
> startTime2018-03-28T00:00Z
> queueNameoozie
> jobTrackerhbase.qa.local:8032
> oozie.libpathhdfs://hbase.qa.local:8020/user/oozie/cmx
> endTime2020-01-01:00Z
> applicationConfigFilehdfs://hbase.qa.local:8020/usr/share/cmx/conf/app.properties,hdfs://hbase.qa.local:8020/usr/share/cmx/conf/crypto.properties
> frequency240
> user.nameoozie
> oozie.coord.application.pathhdfs://hbase.qa.local:8020/user/oozie/reporting/workspaceDimToVertica/
>
> Error: E0307 : E0307: Runtime error [App directory [null] does not exist and
> app definition cannot be created because of missing config value
> [oozie.jobs.api.generated.xml]]
> Note, our code specifies oozie.libpath and oozie.coord.application.path. Is
> this no longer allowed?
> I've narrowed down the problem to the checkAndWriteApplicationXMLToHDFS(...)
> method in V1JobsServlet.java. It is related to the new feature for submitting
> embedded XML within the job submission. This method is check whether the
> workflow.xml or coodinator.xml exists in either oozie.wf.application.path,
> oozie.libpath, oozie.coord.application.path or oozie.bundle.application.path.
> First, why is it checking oozie.libpath? That is not an application path.
> Second, it stops looking as soon as findAppPathsWithFileNames() returns a
> value. Which it will always do when oozie.libpath is set even if an xml
> doesn't exist there. It does not continue looking for an xml file in
> oozie.coord.application.path or oozie.bundle.application.path. I realize
> only one application path should be specified. IIMO, its the oozie.libpath
> check that is the problem.
> Third, the logic is inefficient. Here are two lines. Why call the same method
> again with the same params. You already have the result!
> if (!findAppPathsWithFileNames(appPath, "workflow.xml").isEmpty()) {
> appPathsWithFileNames = findAppPathsWithFileNames(appPath, "workflow.xml");
> }
> It is critical that we upgrade to EMR 5.22. However, we are blocked until we
> figure out a way to get the job submitted and completing successfully. Any
> suggested workarounds would be greatly appreciated. But, I am filing this bug
> to get it fixed so an upgraded Oozie will make it into a future EMR release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3179) Adding a configurable config-default.xml location to a workflow
[
https://issues.apache.org/jira/browse/OOZIE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805068#comment-16805068
]
Hadoop QA commented on OOZIE-3179:
--
Testing JIRA OOZIE-3179
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [4] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:red}-1{color} There are [4] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 207]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 177]: At BulkJPAExecutor.java:[line 176]
.At BulkJPAExecutor.java:[line 206]: At BulkJPAExecutor.java:[line 200]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 207]
.At BulkJPAExecutor.java:[line 112]: At BulkJPAExecutor.java:[line 128]
.{color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
.You can find the SpotBugs diff here: server/findbugs-new.html
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3170
.{color:orange}Tests failed at first run:{color}
TestCoordActionInputCheckXCommand#testNone
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1068/
> Adding a configurable config-default.xml location to a workflow
> ---
>
> Key: OOZIE-3179
> URL: https://issues.apache.org/jira/browse/OOZIE-3179
> Project: Oozie
> Issue Type: Improvement
>Reporter: Jason Phelps
>Assignee: Jason Phelps
>Priority: Major
> Attachments: OOZIE-3179-1.patch, OOZIE-3179-2.patch,
> OOZIE-3179-3.patch, OOZIE-3179-4.patch
>
>
> Recently a customer requested a way to
Failed: OOZIE-3179 PreCommit Build #1068
Jira: https://issues.apache.org/jira/browse/OOZIE-3179 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1068/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 1.93 MB...] [DEBUG] You can find the SpotBugs diff here (look for the red and orange ones): core/findbugs-new.html [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [WARN] There are [4] new bugs found in [server]. [DEBUG] You can find the SpotBugs diff here: server/findbugs-new.html [ERROR] There are [4] new bugs found below threshold in total that must be fixed. [TRACE] SpotBugs diffs checked and reports created [TRACE] Summary file size is 3441 bytes [TRACE] Full summary file size is 3249 bytes [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar] removed [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5] removed Running test-patch task BACKWARDS_COMPATIBILITY Running test-patch task TESTS Running test-patch task DISTRO Testing JIRA OOZIE-3179 Cleaning local git workspace +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS +1 the patch does not introduce any @author tags +1 the patch does not introduce any tabs +1 the patch does not introduce any trailing spaces +1 the patch does not introduce any star imports +1 the patch does not introduce any line longer than 132 +1 the patch adds/modifies 1 testcase(s) +1 RAT +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC +1 Javadoc generation succeeded with the patch +1 the patch does not seem to introduce new Javadoc warning(s) +1 COMPILE +1 HEAD compiles +1 patch compiles +1 the patch does not seem to introduce new javac warnings -1 There are [4] new bugs found below threshold in total that must be fixed. +1 There are no new bugs found in [examples]. +1 There are no new bugs found in [sharelib/sqoop]. +1 There are no new bugs found in [sharelib/hcatalog]. +1 There are no new bugs found in [sharelib/hive2]. +1 There are no new bugs found in [sharelib/oozie]. +1 There are no new bugs found in [sharelib/spark]. +1 There are no new bugs found in [sharelib/hive]. +1 There are no new bugs found in [sharelib/pig]. +1 There are no new bugs found in [sharelib/streaming]. +1 There are no new bugs found in [sharelib/git]. +1 There are no new bugs found in [sharelib/distcp]. +1 There are no new bugs found in [webapp]. +1 There are no new bugs found in [client]. +1 There are no new bugs found in [fluent-job/fluent-job-api]. +1 There are no new bugs found in [docs]. +1 There are no new bugs found in [tools]. -1 There are [4] new bugs found below threshold in [core] that must be fixed. You can find the SpotBugs diff here (look for the red and orange ones): core/findbugs-new.html The most important SpotBugs errors are: At BulkJPAExecutor.java:[line 207]: This use of javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query; can be vulnerable to SQL/JPQL injection At BulkJPAExecutor.java:[line 177]: At BulkJPAExecutor.java:[line 176] At BulkJPAExecutor.java:[line 206]: At BulkJPAExecutor.java:[line 200] This use of javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query; can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 207] At BulkJPAExecutor.java:[line 112]: At BulkJPAExecutor.java:[line 128] 0 There are [4] new bugs found in [server] that would be nice to have fixed. You can find the SpotBugs diff here: server/findbugs-new.html +1 BACKWARDS_COMPATIBILITY +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations +1 the patch does not modify JPA files +1 TESTS Tests run: 3170 Tests failed at first run: TestCoordActionInputCheckXCommand#testNone For the complete list of flaky tests, see TEST-SUMMARY-FULL files. +1 DISTRO +1 distro tarball builds with the patch -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at https://builds.apache.org/job/PreCommit-OOZIE-Build/1068/ Adding comment to JIRA % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
[jira] [Commented] (OOZIE-3450) Investigate and clean git sharelib
[
https://issues.apache.org/jira/browse/OOZIE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805038#comment-16805038
]
Hadoop QA commented on OOZIE-3450:
--
Testing JIRA OOZIE-3450
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:red}-1{color} There are [1] new bugs found below threshold in
[sharelib/oozie] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
sharelib/oozie/findbugs-new.html
.The most important SpotBugs errors are:
.At ShellMain.java:[line 92]: This usage of
java/lang/ProcessBuilder.init(Ljava/util/List;)V can be vulnerable to
Command Injection
.At ShellMain.java:[line 90]: At ShellMain.java:[line 89]
.At ShellMain.java:[line 91]
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3169
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeableBundleUnpurgeableCoordinatorUnpurgeableWorkflow
TestPurgeXCommand#testPurgeableBundleUnpurgeableCoordinatorUnpurgebleWorkflowPurgeableSubWorkflow
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1067/
> Investigate and clean git sharelib
> --
>
> Key: OOZIE-3450
> URL: https://issues.apache.org/jira/browse/OOZIE-3450
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Mate Juhasz
>Priority: Major
> Attachments: OOZIE-3450-v2.patch, OOZIE-3540-v1.patch
>
>
> I've checked the number of jars in the Oozie sharelibs and realized that git
> sharelib contains the highest number of jars (203), it's much more than the
> hive (85), pig (67). Not to mention that we have really small sharelibs like
> distcp (3).
> I don't really understand the reason for this, we need to check if we really
> need all the jars here. The huge number of jars make it slower and it's more
> likely that we get
Failed: OOZIE-3450 PreCommit Build #1067
Jira: https://issues.apache.org/jira/browse/OOZIE-3450 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1067/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 1.93 MB...] [TRACE] SpotBugs diffs checked and reports created [TRACE] Summary file size is 2998 bytes [TRACE] Full summary file size is 1987 bytes [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar] removed [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5] removed Running test-patch task BACKWARDS_COMPATIBILITY Running test-patch task TESTS /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/TESTS/results/sharelib/pig/target/surefire-reports/TEST-org.apache.oozie.action.hadoop.TestPigActionExecutor.xml:126: parser error : Unregistered error message /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/TESTS/results/sharelib/pig/target/surefire-reports/TEST-org.apache.oozie.action.hadoop.TestPigActionExecutor.xml:126: parser error : Premature end of data in tag system-out line 126 /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/TESTS/results/sharelib/pig/target/surefire-reports/TEST-org.apache.oozie.action.hadoop.TestPigActionExecutor.xml:126: parser error : Premature end of data in tag testcase line 74 /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/TESTS/results/sharelib/pig/target/surefire-reports/TEST-org.apache.oozie.action.hadoop.TestPigActionExecutor.xml:126: parser error : Premature end of data in tag testsuite line 2 /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/TESTS/results/sharelib/pig/target/surefire-reports/TEST-org.apache.oozie.action.hadoop.TestPigActionExecutor.xml:126: parser error : Unregistered error message /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/TESTS/results/sharelib/pig/target/surefire-reports/TEST-org.apache.oozie.action.hadoop.TestPigActionExecutor.xml:126: parser error : Premature end of data in tag system-out line 126 /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/TESTS/results/sharelib/pig/target/surefire-reports/TEST-org.apache.oozie.action.hadoop.TestPigActionExecutor.xml:126: parser error : Premature end of data in tag testcase line 74 /home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/TESTS/results/sharelib/pig/target/surefire-reports/TEST-org.apache.oozie.action.hadoop.TestPigActionExecutor.xml:126: parser error : Premature end of data in tag testsuite line 2 Running test-patch task DISTRO Testing JIRA OOZIE-3450 Cleaning local git workspace +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS +1 the patch does not introduce any @author tags +1 the patch does not introduce any tabs +1 the patch does not introduce any trailing spaces +1 the patch does not introduce any star imports +1 the patch does not introduce any line longer than 132 -1 the patch does not add/modify any testcase +1 RAT +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC +1 Javadoc generation succeeded with the patch +1 the patch does not seem to introduce new Javadoc warning(s) +1 COMPILE +1 HEAD compiles +1 patch compiles +1 the patch does not seem to introduce new javac warnings -1 There are [1] new bugs found below threshold in total that must be fixed. +1 There are no new bugs found in [sharelib/hive2]. +1 There are no new bugs found in [sharelib/spark]. -1 There are [1] new bugs found below threshold in [sharelib/oozie] that must be fixed. You can find the SpotBugs diff here (look for the red and orange ones): sharelib/oozie/findbugs-new.html The most important SpotBugs errors are: At ShellMain.java:[line 92]: This usage of java/lang/ProcessBuilder.init(Ljava/util/List;)V can be vulnerable to Command Injection At ShellMain.java:[line 90]: At ShellMain.java:[line 89] At ShellMain.java:[line 91] +1 There are no new bugs found in [sharelib/pig]. +1 There are no new bugs found in [sharelib/streaming]. +1 There are no new bugs found in [sharelib/hive]. +1 There are no new bugs found in [sharelib/distcp]. +1 There are no new bugs found in [sharelib/hcatalog]. +1 There are no new bugs found in [sharelib/sqoop]. +1 There are no new bugs found in [sharelib/git]. +1 There are no new bugs found in [client]. +1 There are no new bugs found in [docs]. +1 There are no new bugs found in [tools]. +1 There are no new bugs found in [fluent-job/fluent-job-api]. +1 There are no new bugs found in [server]. +1 There are no new bugs found in
[jira] [Commented] (OOZIE-3179) Adding a configurable config-default.xml location to a workflow
[ https://issues.apache.org/jira/browse/OOZIE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804945#comment-16804945 ] Hadoop QA commented on OOZIE-3179: -- PreCommit-OOZIE-Build started > Adding a configurable config-default.xml location to a workflow > --- > > Key: OOZIE-3179 > URL: https://issues.apache.org/jira/browse/OOZIE-3179 > Project: Oozie > Issue Type: Improvement >Reporter: Jason Phelps >Assignee: Jason Phelps >Priority: Major > Attachments: OOZIE-3179-1.patch, OOZIE-3179-2.patch, > OOZIE-3179-3.patch, OOZIE-3179-4.patch > > > Recently a customer requested a way to include several job.properties > variables to every workflow in their cluster. > Normally the config-default.xml file would be the best way to include a set > of variables, but it does not scale very well if that file needs to be > updated, as it would require it to be updated in every workflow application > path. > What I am requesting is another job.properties variable that can point to a > config-default.xml file, or a comma separated list of config-default.xml > files that will all be applied to the workflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65636: OOZIE-3179 - Adding a configurable config-default.xml location to a workflow
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65636/ --- (Updated March 29, 2019, 1 p.m.) Review request for oozie. Changes --- Rebased patch which includes the changes made in OOZIE-3370 Repository: oozie-git Description --- Added another client property: `oozie.default.configuration.path` which allows for a comma seperated list of HDFS XML property files to be applied to the job. This is useful if multiple workflows have shared properties (i.e. database names, table names for Sqoop or Hive, spark arguments, etc). This is similar to the `oozie.libpath` property which allows user's to define common paths workflows to load jars. Diffs (updated) - client/src/main/java/org/apache/oozie/client/OozieClient.java 2862d33f core/src/main/java/org/apache/oozie/command/wf/SubmitXCommand.java 70b9adc1 core/src/test/java/org/apache/oozie/command/wf/TestSubmitXCommand.java 2bc0baa4 Diff: https://reviews.apache.org/r/65636/diff/3/ Changes: https://reviews.apache.org/r/65636/diff/2-3/ Testing --- Thanks, Jason Phelps
[jira] [Created] (OOZIE-3459) Oozie cannot be built using Java 11
Denes Bodo created OOZIE-3459:
-
Summary: Oozie cannot be built using Java 11
Key: OOZIE-3459
URL: https://issues.apache.org/jira/browse/OOZIE-3459
Project: Oozie
Issue Type: Bug
Components: core, fluent-job
Affects Versions: 5.1.0
Reporter: Denes Bodo
Using OpenJDK 11 I am not able to build Oozie using {{mvn clean install}}.
I found two issues:
* Fluent job API build fails due to Jaxb2 maven plugin.
* No {{com.sun.tools.}} package is available so *TestMetricsInstrumentation*
will not work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3179) Adding a configurable config-default.xml location to a workflow
[ https://issues.apache.org/jira/browse/OOZIE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804930#comment-16804930 ] Jason Phelps commented on OOZIE-3179: - Review board diff had a few extra changes I made (been a while since then) from Peter's review, which I just added back into patch #4. Will upload the same diff to Reviewboard [~asalamon74] > Adding a configurable config-default.xml location to a workflow > --- > > Key: OOZIE-3179 > URL: https://issues.apache.org/jira/browse/OOZIE-3179 > Project: Oozie > Issue Type: Improvement >Reporter: Jason Phelps >Assignee: Jason Phelps >Priority: Major > Attachments: OOZIE-3179-1.patch, OOZIE-3179-2.patch, > OOZIE-3179-3.patch, OOZIE-3179-4.patch > > > Recently a customer requested a way to include several job.properties > variables to every workflow in their cluster. > Normally the config-default.xml file would be the best way to include a set > of variables, but it does not scale very well if that file needs to be > updated, as it would require it to be updated in every workflow application > path. > What I am requesting is another job.properties variable that can point to a > config-default.xml file, or a comma separated list of config-default.xml > files that will all be applied to the workflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3179) Adding a configurable config-default.xml location to a workflow
[ https://issues.apache.org/jira/browse/OOZIE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Phelps updated OOZIE-3179: Attachment: OOZIE-3179-4.patch > Adding a configurable config-default.xml location to a workflow > --- > > Key: OOZIE-3179 > URL: https://issues.apache.org/jira/browse/OOZIE-3179 > Project: Oozie > Issue Type: Improvement >Reporter: Jason Phelps >Assignee: Jason Phelps >Priority: Major > Attachments: OOZIE-3179-1.patch, OOZIE-3179-2.patch, > OOZIE-3179-3.patch, OOZIE-3179-4.patch > > > Recently a customer requested a way to include several job.properties > variables to every workflow in their cluster. > Normally the config-default.xml file would be the best way to include a set > of variables, but it does not scale very well if that file needs to be > updated, as it would require it to be updated in every workflow application > path. > What I am requesting is another job.properties variable that can point to a > config-default.xml file, or a comma separated list of config-default.xml > files that will all be applied to the workflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3450) Investigate and clean git sharelib
[ https://issues.apache.org/jira/browse/OOZIE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804899#comment-16804899 ] Mate Juhasz commented on OOZIE-3450: Thanks [~asalamon74] for the testing. Could you please give the new patch a try as well? Interesting that each of the sharelib projects are using the oozie-core as a provided dependency, the only difference is that while most of them just reffering to static constants in the ActionExecutor-s, sharelib-git's GitMain tries to call a method in GitActionExecutor$ActionConfVerifier. This method is the ActionConfVerifier#checkAndGetTrimmed, which only returns an action conf value, not connecting strongly anywhere, so I moved it to the GitMain class. > Investigate and clean git sharelib > -- > > Key: OOZIE-3450 > URL: https://issues.apache.org/jira/browse/OOZIE-3450 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-3450-v2.patch, OOZIE-3540-v1.patch > > > I've checked the number of jars in the Oozie sharelibs and realized that git > sharelib contains the highest number of jars (203), it's much more than the > hive (85), pig (67). Not to mention that we have really small sharelibs like > distcp (3). > I don't really understand the reason for this, we need to check if we really > need all the jars here. The huge number of jars make it slower and it's more > likely that we get strange errors because of jar conflicts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3450) Investigate and clean git sharelib
[ https://issues.apache.org/jira/browse/OOZIE-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804896#comment-16804896 ] Hadoop QA commented on OOZIE-3450: -- PreCommit-OOZIE-Build started > Investigate and clean git sharelib > -- > > Key: OOZIE-3450 > URL: https://issues.apache.org/jira/browse/OOZIE-3450 > Project: Oozie > Issue Type: Improvement >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Mate Juhasz >Priority: Major > Attachments: OOZIE-3450-v2.patch, OOZIE-3540-v1.patch > > > I've checked the number of jars in the Oozie sharelibs and realized that git > sharelib contains the highest number of jars (203), it's much more than the > hive (85), pig (67). Not to mention that we have really small sharelibs like > distcp (3). > I don't really understand the reason for this, we need to check if we really > need all the jars here. The huge number of jars make it slower and it's more > likely that we get strange errors because of jar conflicts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OOZIE-3457) App path is null in error message when submit an Oozie coordinator job
[ https://issues.apache.org/jira/browse/OOZIE-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon resolved OOZIE-3457. --- Resolution: Duplicate > App path is null in error message when submit an Oozie coordinator job > -- > > Key: OOZIE-3457 > URL: https://issues.apache.org/jira/browse/OOZIE-3457 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.1.0 >Reporter: Jerry Chabot >Priority: Minor > > We can no longer submit a coordinator job after upgrading from 5.0.0 to > 5.1.0. While we investigate this (there is an email in the oozie user mailing > list), I've been looking at the oozie code trying to determine why the error > message has "null" in it. > Error: E0307 : E0307: Runtime error [App directory [null] does not exist and > app definition cannot be created because of missing config value > [oozie.jobs.api.generated.xml]] > We are submitting a coordinator job so it specifies > oozie.coord.application.path. > Here is a sample submission: > sudo -u oozie oozie job -oozie [http://hbase.qa.local:11000/oozie/] -config > /var/lib/cmx/jobs/workspaceDimToVertica/coordinator.properties -run -D > nameNode=hdfs://hbase.qa.local:8020 -D jobTracker=hbase.qa.local:8032 -D > frequency=240 -D startTime=2018-03-28T00:00Z -D endTime=2020-01-01:00Z -debug > Auth type : null > POST [http://hbase.qa.local:11000/oozie/v2/jobs?action=start] > > nameNodehdfs://hbase.qa.local:8020 > oozie.use.system.libpathtrue > startTime2018-03-28T00:00Z > queueNameoozie > jobTrackerhbase.qa.local:8032 > oozie.libpathhdfs://hbase.qa.local:8020/user/oozie/cmx > endTime2020-01-01:00Z > applicationConfigFilehdfs://hbase.qa.local:8020/usr/share/cmx/conf/app.properties,hdfs://hbase.qa.local:8020/usr/share/cmx/conf/crypto.properties > frequency240 > user.nameoozie > oozie.coord.application.pathhdfs://hbase.qa.local:8020/user/oozie/reporting/workspaceDimToVertica/ > > V1JobsServlet.java:175 creates an error message using the appPath from > oozie.wf.application.path. But, as you will see the method also checks other > appPath. Building the error message should consider these other appPaths as > it would help troubleshoot errors such as these. > Here is the server-side the stack trace. > 2019-03-28 20:00:56,101 WARN V1JobsServlet:523 - > SERVER[ip-172-23-113-207.ec2.internal] USER[-] GROUP[-] TOKEN[-] APP[-] > JOB[-] ACTION[-] URL[POST http://hbase.qa.local:11000/oozie/v2/jobs] > error[E0307], E0307: Runtime error [App directory [null] does not exist and > app definition cannot be created because of missing config value > [oozie.jobs.api.generated.xml]] > org.apache.oozie.servlet.XServletException: E0307: Runtime error [App > directory [null] does not exist and app definition cannot be created because > of missing config value [oozie.jobs.api.generated.xml]] > at > org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:175) > at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) > at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) > at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) > at >
[jira] [Commented] (OOZIE-3458) Can no longer submit a coordinator job which has oozie.libpath and oozie.coord.application.path set
[
https://issues.apache.org/jira/browse/OOZIE-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804883#comment-16804883
]
Jerry Chabot commented on OOZIE-3458:
-
Unfortunately, this means we need a workaround because EMR 5.22 has critical
HBase bug fix. Upgrading or patching Oozie within EMR 5.22 is not an option.
> Can no longer submit a coordinator job which has oozie.libpath and
> oozie.coord.application.path set
> ---
>
> Key: OOZIE-3458
> URL: https://issues.apache.org/jira/browse/OOZIE-3458
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Jerry Chabot
>Priority: Critical
>
> This bug is related to https://issues.apache.org/jira/browse/OOZIE-3457. Our
> Production environment is currently running Amazon EMR 5.19. We want to
> upgrade to EMR 5.22. Oozie will be upgraded from 5.0.0 to 5.1.0. Our
> deployment scripts can no longer submit coordinator jobs after upgrading a
> test environment. Here is a sample post from the oozie client.
> POST [http://hbase.qa.local:11000/oozie/v2/jobs?action=start]
>
> nameNodehdfs://hbase.qa.local:8020
> oozie.use.system.libpathtrue
> startTime2018-03-28T00:00Z
> queueNameoozie
> jobTrackerhbase.qa.local:8032
> oozie.libpathhdfs://hbase.qa.local:8020/user/oozie/cmx
> endTime2020-01-01:00Z
> applicationConfigFilehdfs://hbase.qa.local:8020/usr/share/cmx/conf/app.properties,hdfs://hbase.qa.local:8020/usr/share/cmx/conf/crypto.properties
> frequency240
> user.nameoozie
> oozie.coord.application.pathhdfs://hbase.qa.local:8020/user/oozie/reporting/workspaceDimToVertica/
>
> Error: E0307 : E0307: Runtime error [App directory [null] does not exist and
> app definition cannot be created because of missing config value
> [oozie.jobs.api.generated.xml]]
> Note, our code specifies oozie.libpath and oozie.coord.application.path. Is
> this no longer allowed?
> I've narrowed down the problem to the checkAndWriteApplicationXMLToHDFS(...)
> method in V1JobsServlet.java. It is related to the new feature for submitting
> embedded XML within the job submission. This method is check whether the
> workflow.xml or coodinator.xml exists in either oozie.wf.application.path,
> oozie.libpath, oozie.coord.application.path or oozie.bundle.application.path.
> First, why is it checking oozie.libpath? That is not an application path.
> Second, it stops looking as soon as findAppPathsWithFileNames() returns a
> value. Which it will always do when oozie.libpath is set even if an xml
> doesn't exist there. It does not continue looking for an xml file in
> oozie.coord.application.path or oozie.bundle.application.path. I realize
> only one application path should be specified. IIMO, its the oozie.libpath
> check that is the problem.
> Third, the logic is inefficient. Here are two lines. Why call the same method
> again with the same params. You already have the result!
> if (!findAppPathsWithFileNames(appPath, "workflow.xml").isEmpty()) {
> appPathsWithFileNames = findAppPathsWithFileNames(appPath, "workflow.xml");
> }
> It is critical that we upgrade to EMR 5.22. However, we are blocked until we
> figure out a way to get the job submitted and completing successfully. Any
> suggested workarounds would be greatly appreciated. But, I am filing this bug
> to get it fixed so an upgraded Oozie will make it into a future EMR release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3457) App path is null in error message when submit an Oozie coordinator job
[ https://issues.apache.org/jira/browse/OOZIE-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804875#comment-16804875 ] Jerry Chabot commented on OOZIE-3457: - It looks like the fix for https://issues.apache.org/jira/browse/OOZIE-3415 also fixed the error message I reported in this bug. > App path is null in error message when submit an Oozie coordinator job > -- > > Key: OOZIE-3457 > URL: https://issues.apache.org/jira/browse/OOZIE-3457 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.1.0 >Reporter: Jerry Chabot >Priority: Minor > > We can no longer submit a coordinator job after upgrading from 5.0.0 to > 5.1.0. While we investigate this (there is an email in the oozie user mailing > list), I've been looking at the oozie code trying to determine why the error > message has "null" in it. > Error: E0307 : E0307: Runtime error [App directory [null] does not exist and > app definition cannot be created because of missing config value > [oozie.jobs.api.generated.xml]] > We are submitting a coordinator job so it specifies > oozie.coord.application.path. > Here is a sample submission: > sudo -u oozie oozie job -oozie [http://hbase.qa.local:11000/oozie/] -config > /var/lib/cmx/jobs/workspaceDimToVertica/coordinator.properties -run -D > nameNode=hdfs://hbase.qa.local:8020 -D jobTracker=hbase.qa.local:8032 -D > frequency=240 -D startTime=2018-03-28T00:00Z -D endTime=2020-01-01:00Z -debug > Auth type : null > POST [http://hbase.qa.local:11000/oozie/v2/jobs?action=start] > > nameNodehdfs://hbase.qa.local:8020 > oozie.use.system.libpathtrue > startTime2018-03-28T00:00Z > queueNameoozie > jobTrackerhbase.qa.local:8032 > oozie.libpathhdfs://hbase.qa.local:8020/user/oozie/cmx > endTime2020-01-01:00Z > applicationConfigFilehdfs://hbase.qa.local:8020/usr/share/cmx/conf/app.properties,hdfs://hbase.qa.local:8020/usr/share/cmx/conf/crypto.properties > frequency240 > user.nameoozie > oozie.coord.application.pathhdfs://hbase.qa.local:8020/user/oozie/reporting/workspaceDimToVertica/ > > V1JobsServlet.java:175 creates an error message using the appPath from > oozie.wf.application.path. But, as you will see the method also checks other > appPath. Building the error message should consider these other appPaths as > it would help troubleshoot errors such as these. > Here is the server-side the stack trace. > 2019-03-28 20:00:56,101 WARN V1JobsServlet:523 - > SERVER[ip-172-23-113-207.ec2.internal] USER[-] GROUP[-] TOKEN[-] APP[-] > JOB[-] ACTION[-] URL[POST http://hbase.qa.local:11000/oozie/v2/jobs] > error[E0307], E0307: Runtime error [App directory [null] does not exist and > app definition cannot be created because of missing config value > [oozie.jobs.api.generated.xml]] > org.apache.oozie.servlet.XServletException: E0307: Runtime error [App > directory [null] does not exist and app definition cannot be created because > of missing config value [oozie.jobs.api.generated.xml]] > at > org.apache.oozie.servlet.V1JobsServlet.checkAndWriteApplicationXMLToHDFS(V1JobsServlet.java:175) > at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:111) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:305) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) > at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:171) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) > at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) > at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[jira] [Commented] (OOZIE-3179) Adding a configurable config-default.xml location to a workflow
[ https://issues.apache.org/jira/browse/OOZIE-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804735#comment-16804735 ] Andras Salamon commented on OOZIE-3179: --- Thanks for the rebase [~jphelps]. Yes, it's very important that we handle the OOZIE-3370 changes here. Can you upload the patch to the [ReviewBoard|https://reviews.apache.org/dashboard/]? It's much easier to review the code there. > Adding a configurable config-default.xml location to a workflow > --- > > Key: OOZIE-3179 > URL: https://issues.apache.org/jira/browse/OOZIE-3179 > Project: Oozie > Issue Type: Improvement >Reporter: Jason Phelps >Assignee: Jason Phelps >Priority: Major > Attachments: OOZIE-3179-1.patch, OOZIE-3179-2.patch, > OOZIE-3179-3.patch > > > Recently a customer requested a way to include several job.properties > variables to every workflow in their cluster. > Normally the config-default.xml file would be the best way to include a set > of variables, but it does not scale very well if that file needs to be > updated, as it would require it to be updated in every workflow application > path. > What I am requesting is another job.properties variable that can point to a > config-default.xml file, or a comma separated list of config-default.xml > files that will all be applied to the workflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3458) Can no longer submit a coordinator job which has oozie.libpath and oozie.coord.application.path set
[
https://issues.apache.org/jira/browse/OOZIE-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804692#comment-16804692
]
Andras Salamon commented on OOZIE-3458:
---
Could you please check OOZIE-3415? It looks very similar to that.
> Can no longer submit a coordinator job which has oozie.libpath and
> oozie.coord.application.path set
> ---
>
> Key: OOZIE-3458
> URL: https://issues.apache.org/jira/browse/OOZIE-3458
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Jerry Chabot
>Priority: Critical
>
> This bug is related to https://issues.apache.org/jira/browse/OOZIE-3457. Our
> Production environment is currently running Amazon EMR 5.19. We want to
> upgrade to EMR 5.22. Oozie will be upgraded from 5.0.0 to 5.1.0. Our
> deployment scripts can no longer submit coordinator jobs after upgrading a
> test environment. Here is a sample post from the oozie client.
> POST [http://hbase.qa.local:11000/oozie/v2/jobs?action=start]
>
> nameNodehdfs://hbase.qa.local:8020
> oozie.use.system.libpathtrue
> startTime2018-03-28T00:00Z
> queueNameoozie
> jobTrackerhbase.qa.local:8032
> oozie.libpathhdfs://hbase.qa.local:8020/user/oozie/cmx
> endTime2020-01-01:00Z
> applicationConfigFilehdfs://hbase.qa.local:8020/usr/share/cmx/conf/app.properties,hdfs://hbase.qa.local:8020/usr/share/cmx/conf/crypto.properties
> frequency240
> user.nameoozie
> oozie.coord.application.pathhdfs://hbase.qa.local:8020/user/oozie/reporting/workspaceDimToVertica/
>
> Error: E0307 : E0307: Runtime error [App directory [null] does not exist and
> app definition cannot be created because of missing config value
> [oozie.jobs.api.generated.xml]]
> Note, our code specifies oozie.libpath and oozie.coord.application.path. Is
> this no longer allowed?
> I've narrowed down the problem to the checkAndWriteApplicationXMLToHDFS(...)
> method in V1JobsServlet.java. It is related to the new feature for submitting
> embedded XML within the job submission. This method is check whether the
> workflow.xml or coodinator.xml exists in either oozie.wf.application.path,
> oozie.libpath, oozie.coord.application.path or oozie.bundle.application.path.
> First, why is it checking oozie.libpath? That is not an application path.
> Second, it stops looking as soon as findAppPathsWithFileNames() returns a
> value. Which it will always do when oozie.libpath is set even if an xml
> doesn't exist there. It does not continue looking for an xml file in
> oozie.coord.application.path or oozie.bundle.application.path. I realize
> only one application path should be specified. IIMO, its the oozie.libpath
> check that is the problem.
> Third, the logic is inefficient. Here are two lines. Why call the same method
> again with the same params. You already have the result!
> if (!findAppPathsWithFileNames(appPath, "workflow.xml").isEmpty()) {
> appPathsWithFileNames = findAppPathsWithFileNames(appPath, "workflow.xml");
> }
> It is critical that we upgrade to EMR 5.22. However, we are blocked until we
> figure out a way to get the job submitted and completing successfully. Any
> suggested workarounds would be greatly appreciated. But, I am filing this bug
> to get it fixed so an upgraded Oozie will make it into a future EMR release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (90 issues) Subscriber: ooziedaily Key Summary OOZIE-3455 Inconsistent CoordMaterializeTransitionXCommand logging https://issues.apache.org/jira/browse/OOZIE-3455 OOZIE-3450 Investigate and clean git sharelib https://issues.apache.org/jira/browse/OOZIE-3450 OOZIE-3449 Make spark-2 as the default profile https://issues.apache.org/jira/browse/OOZIE-3449 OOZIE-3447 Run test case in local : It shows oozie-hsqldb-orm.xml exception https://issues.apache.org/jira/browse/OOZIE-3447 OOZIE-3418 Upgrade to Guava 27 https://issues.apache.org/jira/browse/OOZIE-3418 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3335 Cleanup parseFilter methods https://issues.apache.org/jira/browse/OOZIE-3335 OOZIE-3320 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3320 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3265 properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to appear together https://issues.apache.org/jira/browse/OOZIE-3265 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3249 [tools] Instrumentation log parser https://issues.apache.org/jira/browse/OOZIE-3249 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3179 Adding a configurable config-default.xml location to a workflow https://issues.apache.org/jira/browse/OOZIE-3179 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2927 Append new line character for Hive2 query using query tag https://issues.apache.org/jira/browse/OOZIE-2927 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2833 when using uber mode the regex pattern used in the extractHeapSizeMB method does not allow heap sizes specified in bytes. https://issues.apache.org/jira/browse/OOZIE-2833 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator https://issues.apache.org/jira/browse/OOZIE-2784 OOZIE-2779 Mask Hive2 action Beeline JDBC password https://issues.apache.org/jira/browse/OOZIE-2779
