[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (96 issues) Subscriber: ooziedaily Key Summary OOZIE-3407 Cleanup TestPurgeXCommand https://issues.apache.org/jira/browse/OOZIE-3407 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3401 TestPySpark failure https://issues.apache.org/jira/browse/OOZIE-3401 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3326 Sqoop Action should support tez delegation tokens for hive-import https://issues.apache.org/jira/browse/OOZIE-3326 OOZIE-3320 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3320 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3265 properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to appear together https://issues.apache.org/jira/browse/OOZIE-3265 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3249 [tools] Instrumentation log parser https://issues.apache.org/jira/browse/OOZIE-3249 OOZIE-3218 Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space https://issues.apache.org/jira/browse/OOZIE-3218 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3194 Oozie should set proper permissions to sharelib after upload https://issues.apache.org/jira/browse/OOZIE-3194 OOZIE-3186 Oozie is unable to use configuration linked using jceks://file/... https://issues.apache.org/jira/browse/OOZIE-3186 OOZIE-3179 Adding a configurable config-default.xml location to a workflow https://issues.apache.org/jira/browse/OOZIE-3179 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2949 Escape quotes whitespaces in Sqoop field https://issues.apache.org/jira/browse/OOZIE-2949 OOZIE-2927 Append new line character for Hive2 query using query tag https://issues.apache.org/jira/browse/OOZIE-2927 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2833 when using uber mode the regex pattern used in the extractHeapSizeMB method does not allow heap sizes specified in bytes. https://issues.apache.org/jira/browse/OOZIE-2833 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in
[jira] [Commented] (OOZIE-3407) Cleanup TestPurgeXCommand
[ https://issues.apache.org/jira/browse/OOZIE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732171#comment-16732171 ] Andras Salamon commented on OOZIE-3407: --- The tests were already flaky: https://issues.apache.org/jira/browse/OOZIE-3252 this problem was not introduced with this patch. > Cleanup TestPurgeXCommand > - > > Key: OOZIE-3407 > URL: https://issues.apache.org/jira/browse/OOZIE-3407 > Project: Oozie > Issue Type: Sub-task >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3407-01.patch, OOZIE-3407-02.patch > > > [TestPurgeXCommand|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java] > is the longest java source file of Oozie, more than 4000 rows. > As part of the Purge Service rewrite we need to use this file to make sure we > don't introduce new bugs and probably we also need to add a few more test > cases. Before that it would be useful to cleanup this file and extract > redundant code to new methods to make the code shorter and more manageable. > The patch uploaded to OOZIE-3400 introduced three new test cases to this file > and shows a simple way to make it shorter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3407) Cleanup TestPurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732161#comment-16732161
]
Hadoop QA commented on OOZIE-3407:
--
Testing JIRA OOZIE-3407
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3106
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChildWithSubWF1
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChildWithSubWF2
TestPurgeXCommand#testPurgeCoordWithWFChild2
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/956/
> Cleanup TestPurgeXCommand
> -
>
> Key: OOZIE-3407
> URL: https://issues.apache.org/jira/browse/OOZIE-3407
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3407-01.patch, OOZIE-3407-02.patch
>
>
> [TestPurgeXCommand|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java]
> is the longest java source file of Oozie, more than 4000 rows.
> As part of the Purge Service rewrite we need to use this file to make sure we
> don't introduce new bugs and probably we also need to add a few more test
> cases. Before that it would be useful to cleanup this file and extract
> redundant code to new methods to make the code shorter and more manageable.
> The patch uploaded to OOZIE-3400 introduced three new test cases to this file
> and shows a simple way to make it shorter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: OOZIE-3407 PreCommit Build #956
Jira: https://issues.apache.org/jira/browse/OOZIE-3407
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/956/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 1.92 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2560 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Testing JIRA OOZIE-3407
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [fluent-job/fluent-job-api].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3106
Tests failed at first run:
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChildWithSubWF1
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChildWithSubWF2
TestPurgeXCommand#testPurgeCoordWithWFChild2
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch
+1 Overall result, good!, no -1s
There is at least one warning, please check
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/956/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0100 35770 0 100 3577 0 4441 --:--:-- --:--:-- --:--:--
4437{"self":"https://issues.apache.org/jira/rest/api/2/issue/13205346/comment/16732161","id":"16732161","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA
OOZIE-3407\n\nCleaning local git
[jira] [Commented] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation
[
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732123#comment-16732123
]
Andras Salamon commented on OOZIE-3409:
---
We could create a hack solution like:
# Extending {{ExpressionEvaluatorImpl}} and not using the cache
# Counting EL expression evaluations and drop {{ExpressionEvaluatorImpl}} and
start over after a while
The best solution is to replace the obsolete {{commons.el}}. I've checked our
code, it seems to be this is the only place we use {{commons.el}} in Oozie.
There are seveal EL evaluators listed here:
[https://stackoverflow.com/a/17051440/21348] We might check JEXL (
[http://commons.apache.org/proper/commons-jexl/] ) since we already use that
in other parts of Oozie. Of course we have to be sure it does not have the same
bug.
> Oozie Server : Possible memory leak in EL evaluation
>
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers
> have been running out of memory occasionally. It happens more often when we
> have more activity. Upon analysis of the heap dump, it seems that the culprit
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from
> {{commons-el}} which is used in the Oozie server through
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses
> a static {{Map}} [2] to cache the EL evaluations. This map never gets
> cleared, eventually taking up all the memory. So, in an environment with
> heavy usage and long running oozie servers, this would lead to an OOM. The
> configurations for workflow actions we have are possibly larger than normal
> which is worsening the situation as the Map has to maintain larger strings.
> #
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
> #
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[
https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated OOZIE-2949:
--
Comment: was deleted
(was: We could create a hack solution like:
# Extending {{ExpressionEvaluatorImpl}} and not using the cache
# Counting EL expression evaluations and drop {{ExpressionEvaluatorImpl}} and
start over after a while
The best solution (just like [~gezapeti] suggested) is to replace commons el.
I've checked our code, it seems to be this is the only place we use
{{commons.el}} in Oozie. There are seveal EL evaluators listed here:
[https://stackoverflow.com/a/17051440/21348] We might check JEXL (
[http://commons.apache.org/proper/commons-jexl/] ) since we already use that
in other parts of Oozie.
)
> Escape quotes whitespaces in Sqoop field
> --
>
> Key: OOZIE-2949
> URL: https://issues.apache.org/jira/browse/OOZIE-2949
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Peter Cseh
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch
>
>
> The current behavior of the Sqoop action is:
> {noformat}
> The Sqoop command can be specified either using the command element or
> multiple arg elements.
> When using the command element, Oozie will split the command on every space
> into multiple arguments.
> When using the arg elements, Oozie will pass each argument value as an
> argument to Sqoop.
> {noformat}
> This prevents the user to simply copy-paste the command worked in the shell
> into the workflow.xml.
> We should split the field by taking quotes into account, similar to
> what OOZIE-2391
> did for the Spark action's field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[
https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732113#comment-16732113
]
Andras Salamon commented on OOZIE-2949:
---
We could create a hack solution like:
# Extending {{ExpressionEvaluatorImpl}} and not using the cache
# Counting EL expression evaluations and drop {{ExpressionEvaluatorImpl}} and
start over after a while
The best solution (just like [~gezapeti] suggested) is to replace commons el.
I've checked our code, it seems to be this is the only place we use
{{commons.el}} in Oozie. There are seveal EL evaluators listed here:
[https://stackoverflow.com/a/17051440/21348] We might check JEXL (
[http://commons.apache.org/proper/commons-jexl/] ) since we already use that
in other parts of Oozie.
> Escape quotes whitespaces in Sqoop field
> --
>
> Key: OOZIE-2949
> URL: https://issues.apache.org/jira/browse/OOZIE-2949
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Peter Cseh
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch
>
>
> The current behavior of the Sqoop action is:
> {noformat}
> The Sqoop command can be specified either using the command element or
> multiple arg elements.
> When using the command element, Oozie will split the command on every space
> into multiple arguments.
> When using the arg elements, Oozie will pass each argument value as an
> argument to Sqoop.
> {noformat}
> This prevents the user to simply copy-paste the command worked in the shell
> into the workflow.xml.
> We should split the field by taking quotes into account, similar to
> what OOZIE-2391
> did for the Spark action's field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[
https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732068#comment-16732068
]
Peter Cseh edited comment on OOZIE-2949 at 1/2/19 2:06 PM:
---
The best solution would be some third party lib that handles this for us and is
well-tested, developed. I could not find anything except for Java code
snippets. (There are some 10+ year old things out there though)
It looks like we'll have to implement our own solution here unfortunately.
I think something similar to [this
gist|https://gist.github.com/raymyers/8077031] should be a good enough to start
with. It handles embedding " and ' and escaping with "\"
was (Author: gezapeti):
The best solution would be some third party lib that handles this for us and is
well-tested, developed. I could not find anything except for Java code
snippets. (There are some 10+ year old things out there though)
It looks like we'll have to implement our own solution here unfortunately.
> Escape quotes whitespaces in Sqoop field
> --
>
> Key: OOZIE-2949
> URL: https://issues.apache.org/jira/browse/OOZIE-2949
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Peter Cseh
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch
>
>
> The current behavior of the Sqoop action is:
> {noformat}
> The Sqoop command can be specified either using the command element or
> multiple arg elements.
> When using the command element, Oozie will split the command on every space
> into multiple arguments.
> When using the arg elements, Oozie will pass each argument value as an
> argument to Sqoop.
> {noformat}
> This prevents the user to simply copy-paste the command worked in the shell
> into the workflow.xml.
> We should split the field by taking quotes into account, similar to
> what OOZIE-2391
> did for the Spark action's field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[
https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732068#comment-16732068
]
Peter Cseh commented on OOZIE-2949:
---
The best solution would be some third party lib that handles this for us and is
well-tested, developed. I could not find anything except for Java code
snippets. (There are some 10+ year old things out there though)
It looks like we'll have to implement our own solution here unfortunately.
> Escape quotes whitespaces in Sqoop field
> --
>
> Key: OOZIE-2949
> URL: https://issues.apache.org/jira/browse/OOZIE-2949
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Peter Cseh
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch
>
>
> The current behavior of the Sqoop action is:
> {noformat}
> The Sqoop command can be specified either using the command element or
> multiple arg elements.
> When using the command element, Oozie will split the command on every space
> into multiple arguments.
> When using the arg elements, Oozie will pass each argument value as an
> argument to Sqoop.
> {noformat}
> This prevents the user to simply copy-paste the command worked in the shell
> into the workflow.xml.
> We should split the field by taking quotes into account, similar to
> what OOZIE-2391
> did for the Spark action's field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3407) Cleanup TestPurgeXCommand
[ https://issues.apache.org/jira/browse/OOZIE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732065#comment-16732065 ] Andras Salamon commented on OOZIE-3407: --- I cannot open a ReviewBoard request: https://issues.apache.org/jira/browse/INFRA-17525 > Cleanup TestPurgeXCommand > - > > Key: OOZIE-3407 > URL: https://issues.apache.org/jira/browse/OOZIE-3407 > Project: Oozie > Issue Type: Sub-task >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3407-01.patch, OOZIE-3407-02.patch > > > [TestPurgeXCommand|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java] > is the longest java source file of Oozie, more than 4000 rows. > As part of the Purge Service rewrite we need to use this file to make sure we > don't introduce new bugs and probably we also need to add a few more test > cases. Before that it would be useful to cleanup this file and extract > redundant code to new methods to make the code shorter and more manageable. > The patch uploaded to OOZIE-3400 introduced three new test cases to this file > and shows a simple way to make it shorter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3407) Cleanup TestPurgeXCommand
[
https://issues.apache.org/jira/browse/OOZIE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732060#comment-16732060
]
Andras Salamon commented on OOZIE-3407:
---
Reuploaded the patch using {{--full-index}} to make the ReviewBoard happy.
> Cleanup TestPurgeXCommand
> -
>
> Key: OOZIE-3407
> URL: https://issues.apache.org/jira/browse/OOZIE-3407
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3407-01.patch, OOZIE-3407-02.patch
>
>
> [TestPurgeXCommand|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java]
> is the longest java source file of Oozie, more than 4000 rows.
> As part of the Purge Service rewrite we need to use this file to make sure we
> don't introduce new bugs and probably we also need to add a few more test
> cases. Before that it would be useful to cleanup this file and extract
> redundant code to new methods to make the code shorter and more manageable.
> The patch uploaded to OOZIE-3400 introduced three new test cases to this file
> and shows a simple way to make it shorter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3404) The env variable of SPARK_HOME needs to be set when running pySpark
[
https://issues.apache.org/jira/browse/OOZIE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732031#comment-16732031
]
Hadoop QA commented on OOZIE-3404:
--
Testing JIRA OOZIE-3404
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3106
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChildWithSubWF2
TestPurgeXCommand#testPurgeCoordWithWFChild1
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/955/
> The env variable of SPARK_HOME needs to be set when running pySpark
> ---
>
> Key: OOZIE-3404
> URL: https://issues.apache.org/jira/browse/OOZIE-3404
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3404-1.patch
>
>
> When we run spark in a cluster, we rely on the spark jars on hdfs. We don't
> deploy Spark on the cluster server. So running pySpark according to the Oozie
> documentation is not successful.
>
> I found that when Hadoop is a 2.0+ version, although Oozie sets the
> {{SPARK_HOME}} variable in {{mapred.child.env}} , the {{mapreduce.map.env}}
> variable is read first in Hadoop ([source
> code|https://github.com/apache/hadoop/blob/f95b390df2ca7d599f0ad82cf6e8d980469e7abb/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java#L45])
> . So when we don't set {{SPARK_HOME}} env in {{mapreduce.map.env}} , pySpark
> doesn't work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: OOZIE-3404 PreCommit Build #955
Jira: https://issues.apache.org/jira/browse/OOZIE-3404
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/955/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 1.91 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2561 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Testing JIRA OOZIE-3404
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [fluent-job/fluent-job-api].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3106
Tests failed at first run:
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChildWithSubWF2
TestPurgeXCommand#testPurgeCoordWithWFChild1
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch
-1 Overall result, please check the reported -1(s)
There is at least one warning, please check
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/955/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 35240 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13205107/comment/16732031","id":"16732031","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA
OOZIE-3404\n\nCleaning local git
[jira] [Commented] (OOZIE-2949) Escape quotes whitespaces in Sqoop field
[
https://issues.apache.org/jira/browse/OOZIE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731989#comment-16731989
]
Andras Salamon commented on OOZIE-2949:
---
I've checked the current version of the patch in more detail.
The plan was to handle commands like {{--query 'SELECT * FROM employee'}} and
use two tokens in this case ({{"--query"}} and {{"SELECT * FROM employee"}}).
The current patch works in this case.
It fails if single and double quotes are mixed like ' ... " " '
because it thinks that the first double quote closes the single quote. It would
be easy to fix the patch and recognise the whole command like one token.
I'm not sure if this is a good enough solution.
It will still fail if we want to use a single quote inside the query (e.g.
search for O'Briens).
I've also talked to sqoop experts. The confirmed that sqoop command parsing is
a rather complex task. If we want to be able to block-copy from shell to oozie,
the best approach is probably mimicking the shell parsing. We would still need
real-life examples to create lots of test cases.
> Escape quotes whitespaces in Sqoop field
> --
>
> Key: OOZIE-2949
> URL: https://issues.apache.org/jira/browse/OOZIE-2949
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Peter Cseh
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-2949-0.patch, OOZIE-2949-01.patch
>
>
> The current behavior of the Sqoop action is:
> {noformat}
> The Sqoop command can be specified either using the command element or
> multiple arg elements.
> When using the command element, Oozie will split the command on every space
> into multiple arguments.
> When using the arg elements, Oozie will pass each argument value as an
> argument to Sqoop.
> {noformat}
> This prevents the user to simply copy-paste the command worked in the shell
> into the workflow.xml.
> We should split the field by taking quotes into account, similar to
> what OOZIE-2391
> did for the Spark action's field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3404) The env variable of SPARK_HOME needs to be set when running pySpark
[
https://issues.apache.org/jira/browse/OOZIE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731971#comment-16731971
]
Andras Piros commented on OOZIE-3404:
-
Thanks for the contribution so far [~zuston].
A couple of questions and remarks:
* new unit tests missing
* are you sure you need exactly these MapReduce settings for all Hadoop
versions 2.0+? This means all the Hadoop versions Oozie currently supports as
this is in the range [2.6.0, 3.1.1]
* please provide tested Oozie and Hadoop versions, as well as relevant pieces
of the PySpark workflow definition / job properties file / Spark version /
PySpark sources. As [~asalamon74] already pointed out, PySpark works under a
wide variety of circumstances
* I don't understand why setting exactly these MapReduce properties would help
the PySpark jobs (and only those) to succeed. Can you please explain?
> The env variable of SPARK_HOME needs to be set when running pySpark
> ---
>
> Key: OOZIE-3404
> URL: https://issues.apache.org/jira/browse/OOZIE-3404
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3404-1.patch
>
>
> When we run spark in a cluster, we rely on the spark jars on hdfs. We don't
> deploy Spark on the cluster server. So running pySpark according to the Oozie
> documentation is not successful.
>
> I found that when Hadoop is a 2.0+ version, although Oozie sets the
> {{SPARK_HOME}} variable in {{mapred.child.env}} , the {{mapreduce.map.env}}
> variable is read first in Hadoop ([source
> code|https://github.com/apache/hadoop/blob/f95b390df2ca7d599f0ad82cf6e8d980469e7abb/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java#L45])
> . So when we don't set {{SPARK_HOME}} env in {{mapreduce.map.env}} , pySpark
> doesn't work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3404) The env variable of SPARK_HOME needs to be set when running pySpark
[
https://issues.apache.org/jira/browse/OOZIE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731948#comment-16731948
]
Hadoop QA commented on OOZIE-3404:
--
PreCommit-OOZIE-Build started
> The env variable of SPARK_HOME needs to be set when running pySpark
> ---
>
> Key: OOZIE-3404
> URL: https://issues.apache.org/jira/browse/OOZIE-3404
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3404-1.patch
>
>
> When we run spark in a cluster, we rely on the spark jars on hdfs. We don't
> deploy Spark on the cluster server. So running pySpark according to the Oozie
> documentation is not successful.
>
> I found that when Hadoop is a 2.0+ version, although Oozie sets the
> {{SPARK_HOME}} variable in {{mapred.child.env}} , the {{mapreduce.map.env}}
> variable is read first in Hadoop ([source
> code|https://github.com/apache/hadoop/blob/f95b390df2ca7d599f0ad82cf6e8d980469e7abb/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java#L45])
> . So when we don't set {{SPARK_HOME}} env in {{mapreduce.map.env}} , pySpark
> doesn't work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3341) [docs] Fix difference between command line help and documentation
[
https://issues.apache.org/jira/browse/OOZIE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3341:
Component/s: docs
> [docs] Fix difference between command line help and documentation
> -
>
> Key: OOZIE-3341
> URL: https://issues.apache.org/jira/browse/OOZIE-3341
> Project: Oozie
> Issue Type: Improvement
> Components: docs
>Affects Versions: trunk, 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3341-01.patch
>
>
> The {{help}} command of the oozie CLI prints out usage information. It is
> also possible to print out only for the specified command, for instance:
> {noformat}
> oozie help job{noformat}
> This information is also available in the
> [Markdown|https://github.com/apache/oozie/blob/master/docs/src/site/markdown/DG_CommandLineTool.md]
> documentation and in the generated HTML.
> There are several differences, the Markdown (HTML) looks outdated. For
> instance for {{job}} command lists the fluent job options as {{apijarcheck}},
> {{apijarsubmit}}, and {{apijarrun}} but these options has been renamed to
> {{validatejar}}, {{submitjar}}, and {{runjar}}. The CLI help shows the
> correct names. Note that Fluent Job related differences are handled by
> OOZIE-3358.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3341) [docs] Fix difference between command line help and documentation
[
https://issues.apache.org/jira/browse/OOZIE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3341:
Summary: [docs] Fix difference between command line help and documentation
(was: [docs] Fix difference between command line help and command line
documentation)
> [docs] Fix difference between command line help and documentation
> -
>
> Key: OOZIE-3341
> URL: https://issues.apache.org/jira/browse/OOZIE-3341
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk, 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3341-01.patch
>
>
> The {{help}} command of the oozie CLI prints out usage information. It is
> also possible to print out only for the specified command, for instance:
> {noformat}
> oozie help job{noformat}
> This information is also available in the
> [Markdown|https://github.com/apache/oozie/blob/master/docs/src/site/markdown/DG_CommandLineTool.md]
> documentation and in the generated HTML.
> There are several differences, the Markdown (HTML) looks outdated. For
> instance for {{job}} command lists the fluent job options as {{apijarcheck}},
> {{apijarsubmit}}, and {{apijarrun}} but these options has been renamed to
> {{validatejar}}, {{submitjar}}, and {{runjar}}. The CLI help shows the
> correct names. Note that Fluent Job related differences are handled by
> OOZIE-3358.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3341) [docs] Fix difference between command line help and command line documentation
[
https://issues.apache.org/jira/browse/OOZIE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3341:
Summary: [docs] Fix difference between command line help and command line
documentation (was: Fix difference between command line help and command line
documentation)
> [docs] Fix difference between command line help and command line documentation
> --
>
> Key: OOZIE-3341
> URL: https://issues.apache.org/jira/browse/OOZIE-3341
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk, 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3341-01.patch
>
>
> The {{help}} command of the oozie CLI prints out usage information. It is
> also possible to print out only for the specified command, for instance:
> {noformat}
> oozie help job{noformat}
> This information is also available in the
> [Markdown|https://github.com/apache/oozie/blob/master/docs/src/site/markdown/DG_CommandLineTool.md]
> documentation and in the generated HTML.
> There are several differences, the Markdown (HTML) looks outdated. For
> instance for {{job}} command lists the fluent job options as {{apijarcheck}},
> {{apijarsubmit}}, and {{apijarrun}} but these options has been renamed to
> {{validatejar}}, {{submitjar}}, and {{runjar}}. The CLI help shows the
> correct names. Note that Fluent Job related differences are handled by
> OOZIE-3358.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3341) Fix difference between command line help and command line documentation
[
https://issues.apache.org/jira/browse/OOZIE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731941#comment-16731941
]
Andras Piros commented on OOZIE-3341:
-
Thanks for the contribution [~asalamon74]! +1
> Fix difference between command line help and command line documentation
> ---
>
> Key: OOZIE-3341
> URL: https://issues.apache.org/jira/browse/OOZIE-3341
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk, 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3341-01.patch
>
>
> The {{help}} command of the oozie CLI prints out usage information. It is
> also possible to print out only for the specified command, for instance:
> {noformat}
> oozie help job{noformat}
> This information is also available in the
> [Markdown|https://github.com/apache/oozie/blob/master/docs/src/site/markdown/DG_CommandLineTool.md]
> documentation and in the generated HTML.
> There are several differences, the Markdown (HTML) looks outdated. For
> instance for {{job}} command lists the fluent job options as {{apijarcheck}},
> {{apijarsubmit}}, and {{apijarrun}} but these options has been renamed to
> {{validatejar}}, {{submitjar}}, and {{runjar}}. The CLI help shows the
> correct names. Note that Fluent Job related differences are handled by
> OOZIE-3358.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3341) Fix difference between command line help and command line documentation
[
https://issues.apache.org/jira/browse/OOZIE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3341:
Description:
The {{help}} command of the oozie CLI prints out usage information. It is also
possible to print out only for the specified command, for instance:
{noformat}
oozie help job{noformat}
This information is also available in the
[Markdown|https://github.com/apache/oozie/blob/master/docs/src/site/markdown/DG_CommandLineTool.md]
documentation and in the generated HTML.
There are several differences, the Markdown (HTML) looks outdated. For instance
for {{job}} command lists the fluent job options as {{apijarcheck}},
{{apijarsubmit}}, and {{apijarrun}} but these options has been renamed to
{{validatejar}}, {{submitjar}}, and {{runjar}}. The CLI help shows the correct
names. Note that Fluent Job related differences are handled by OOZIE-3358.
was:
The {{help}} command of the oozie CLI prints out usage information. It is also
possible to print out only for the specified command, for instance:
{noformat}
oozie help job{noformat}
This information is also available in the
[twiki|https://github.com/apache/oozie/blob/master/docs/src/site/twiki/DG_CommandLineTool.twiki]
documentation and in the generated HTML.
There are several differences, the twiki (HTML) looks outdated. For instance
for {{job}} command lists the fluent job options as {{apijarcheck}},
{{apijarsubmit}}, and {{apijarrun}} but these options has been renamed to
{{validatejar}}, {{submitjar}}, and {{runjar}}. The CLI help shows the correct
names. Note that Fluent Job related differences are handled by OOZIE-3358.
> Fix difference between command line help and command line documentation
> ---
>
> Key: OOZIE-3341
> URL: https://issues.apache.org/jira/browse/OOZIE-3341
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk, 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3341-01.patch
>
>
> The {{help}} command of the oozie CLI prints out usage information. It is
> also possible to print out only for the specified command, for instance:
> {noformat}
> oozie help job{noformat}
> This information is also available in the
> [Markdown|https://github.com/apache/oozie/blob/master/docs/src/site/markdown/DG_CommandLineTool.md]
> documentation and in the generated HTML.
> There are several differences, the Markdown (HTML) looks outdated. For
> instance for {{job}} command lists the fluent job options as {{apijarcheck}},
> {{apijarsubmit}}, and {{apijarrun}} but these options has been renamed to
> {{validatejar}}, {{submitjar}}, and {{runjar}}. The CLI help shows the
> correct names. Note that Fluent Job related differences are handled by
> OOZIE-3358.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3400) [core] Fix PurgeService sub-sub-workflow checking
[ https://issues.apache.org/jira/browse/OOZIE-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros updated OOZIE-3400: Summary: [core] Fix PurgeService sub-sub-workflow checking (was: Fix PurgeService sub-sub-workflow checking) > [core] Fix PurgeService sub-sub-workflow checking > - > > Key: OOZIE-3400 > URL: https://issues.apache.org/jira/browse/OOZIE-3400 > Project: Oozie > Issue Type: Sub-task > Components: core >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3400-00.patch, OOZIE-3400-01.patch, > OOZIE-3400-02.patch, OOZIE-3400-03.patch > > > Purge service > [checks|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java#L222-L223] > sub-workflows before it deletes a workflow. If some of the sub-workflows is > not yet ready to be purged it does not delete the workflow. Several unit > tests also checks this, e. g. > [testPurgeWFWithSubWF1|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java#L2271]. > This check is recursive, it also checks sub-sub-workflows, > sub-sub-sub-workflows > The recursive check is buggy. Let's assume that we have a workflow (A) with a > single sub-workflow (B), which also has a sub-sub-workflow (C). If A is ready > to be purged, B is ready to be purged, and C is not ready to be purged then > the current implementation correctly recognizes that C and B is not > purgeable, but it purges A. Instead of this, it should also recognize that > it's not possible to purge A. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3341) Fix difference between command line help and command line documentation
[
https://issues.apache.org/jira/browse/OOZIE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3341:
Affects Version/s: 5.1.0
> Fix difference between command line help and command line documentation
> ---
>
> Key: OOZIE-3341
> URL: https://issues.apache.org/jira/browse/OOZIE-3341
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk, 5.1.0
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3341-01.patch
>
>
> The {{help}} command of the oozie CLI prints out usage information. It is
> also possible to print out only for the specified command, for instance:
> {noformat}
> oozie help job{noformat}
> This information is also available in the
> [twiki|https://github.com/apache/oozie/blob/master/docs/src/site/twiki/DG_CommandLineTool.twiki]
> documentation and in the generated HTML.
> There are several differences, the twiki (HTML) looks outdated. For instance
> for {{job}} command lists the fluent job options as {{apijarcheck}},
> {{apijarsubmit}}, and {{apijarrun}} but these options has been renamed to
> {{validatejar}}, {{submitjar}}, and {{runjar}}. The CLI help shows the
> correct names. Note that Fluent Job related differences are handled by
> OOZIE-3358.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation
[
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon reassigned OOZIE-3409:
-
Assignee: Andras Salamon
> Oozie Server : Possible memory leak in EL evaluation
>
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers
> have been running out of memory occasionally. It happens more often when we
> have more activity. Upon analysis of the heap dump, it seems that the culprit
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from
> {{commons-el}} which is used in the Oozie server through
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses
> a static {{Map}} [2] to cache the EL evaluations. This map never gets
> cleared, eventually taking up all the memory. So, in an environment with
> heavy usage and long running oozie servers, this would lead to an OOM. The
> configurations for workflow actions we have are possibly larger than normal
> which is worsening the situation as the Map has to maintain larger strings.
> #
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
> #
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3400) Fix PurgeService sub-sub-workflow checking
[ https://issues.apache.org/jira/browse/OOZIE-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros updated OOZIE-3400: Component/s: core > Fix PurgeService sub-sub-workflow checking > -- > > Key: OOZIE-3400 > URL: https://issues.apache.org/jira/browse/OOZIE-3400 > Project: Oozie > Issue Type: Sub-task > Components: core >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3400-00.patch, OOZIE-3400-01.patch, > OOZIE-3400-02.patch, OOZIE-3400-03.patch > > > Purge service > [checks|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java#L222-L223] > sub-workflows before it deletes a workflow. If some of the sub-workflows is > not yet ready to be purged it does not delete the workflow. Several unit > tests also checks this, e. g. > [testPurgeWFWithSubWF1|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java#L2271]. > This check is recursive, it also checks sub-sub-workflows, > sub-sub-sub-workflows > The recursive check is buggy. Let's assume that we have a workflow (A) with a > single sub-workflow (B), which also has a sub-sub-workflow (C). If A is ready > to be purged, B is ready to be purged, and C is not ready to be purged then > the current implementation correctly recognizes that C and B is not > purgeable, but it purges A. Instead of this, it should also recognize that > it's not possible to purge A. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3400) Fix PurgeService sub-sub-workflow checking
[ https://issues.apache.org/jira/browse/OOZIE-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731919#comment-16731919 ] Andras Piros commented on OOZIE-3400: - Thanks for the contribution [~asalamon74]! +1 > Fix PurgeService sub-sub-workflow checking > -- > > Key: OOZIE-3400 > URL: https://issues.apache.org/jira/browse/OOZIE-3400 > Project: Oozie > Issue Type: Sub-task >Affects Versions: trunk >Reporter: Andras Salamon >Assignee: Andras Salamon >Priority: Major > Fix For: 5.2.0 > > Attachments: OOZIE-3400-00.patch, OOZIE-3400-01.patch, > OOZIE-3400-02.patch, OOZIE-3400-03.patch > > > Purge service > [checks|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/command/PurgeXCommand.java#L222-L223] > sub-workflows before it deletes a workflow. If some of the sub-workflows is > not yet ready to be purged it does not delete the workflow. Several unit > tests also checks this, e. g. > [testPurgeWFWithSubWF1|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java#L2271]. > This check is recursive, it also checks sub-sub-workflows, > sub-sub-sub-workflows > The recursive check is buggy. Let's assume that we have a workflow (A) with a > single sub-workflow (B), which also has a sub-sub-workflow (C). If A is ready > to be purged, B is ready to be purged, and C is not ready to be purged then > the current implementation correctly recognizes that C and B is not > purgeable, but it purges A. Instead of this, it should also recognize that > it's not possible to purge A. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 69594: OOZIE-3400: Fix PurgeService sub-sub-workflow checking
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69594/#review211601 --- Ship it! Ship It! - András Piros On Dec. 27, 2018, 3:13 p.m., Andras Salamon wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69594/ > --- > > (Updated Dec. 27, 2018, 3:13 p.m.) > > > Review request for oozie, András Piros and Kinga Marton. > > > Repository: oozie-git > > > Description > --- > > OOZIE-3400: Fix PurgeService sub-sub-workflow checking > > > Diffs > - > > core/src/main/java/org/apache/oozie/ErrorCode.java 9cc153bb0 > core/src/main/java/org/apache/oozie/command/PurgeXCommand.java 42c3b28a6 > core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java > d11fcffbb > core/src/test/java/org/apache/oozie/command/TestSelectorTreeTraverser.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/69594/diff/3/ > > > Testing > --- > > Run TestPurgeXCommand unit tests locally. > > > Thanks, > > Andras Salamon > >
[jira] [Resolved] (OOZIE-3412) Update repository links to gitbox
[ https://issues.apache.org/jira/browse/OOZIE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Cseh resolved OOZIE-3412. --- Resolution: Fixed Thanks for the verification! > Update repository links to gitbox > - > > Key: OOZIE-3412 > URL: https://issues.apache.org/jira/browse/OOZIE-3412 > Project: Oozie > Issue Type: Task >Reporter: Peter Cseh >Assignee: Peter Cseh >Priority: Major > > Once INFRA-17506 is done, we should update the website and the pre-commit > job to refer the new repository. > Things to update: > ||Name||Status|| > |Jenkins precommit|Done| > |[Website repo url|http://oozie.apache.org/source-repository.html]|Done| > |[Branhes > site|https://cwiki.apache.org/confluence/display/OOZIE/Branches+Information]|Done| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3412) Update repository links to gitbox
[ https://issues.apache.org/jira/browse/OOZIE-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731827#comment-16731827 ] Andras Salamon commented on OOZIE-3412: --- [~gezapeti] Links look good to me. +1 (Non-binding). > Update repository links to gitbox > - > > Key: OOZIE-3412 > URL: https://issues.apache.org/jira/browse/OOZIE-3412 > Project: Oozie > Issue Type: Task >Reporter: Peter Cseh >Assignee: Peter Cseh >Priority: Major > > Once INFRA-17506 is done, we should update the website and the pre-commit > job to refer the new repository. > Things to update: > ||Name||Status|| > |Jenkins precommit|Done| > |[Website repo url|http://oozie.apache.org/source-repository.html]|Done| > |[Branhes > site|https://cwiki.apache.org/confluence/display/OOZIE/Branches+Information]|Done| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
