[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (97 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3409  Oozie Server : Possible memory leak in EL evaluation
https://issues.apache.org/jira/browse/OOZIE-3409
OOZIE-3407  Cleanup TestPurgeXCommand
https://issues.apache.org/jira/browse/OOZIE-3407
OOZIE-3404  The env variable of SPARK_HOME needs to be set when running pySpark
https://issues.apache.org/jira/browse/OOZIE-3404
OOZIE-3401  TestPySpark failure
https://issues.apache.org/jira/browse/OOZIE-3401
OOZIE-3375  Can't use empty  in coordinator
https://issues.apache.org/jira/browse/OOZIE-3375
OOZIE-3367  Using && in EL expressions in oozie bundle.xml files generates 
parse errors
https://issues.apache.org/jira/browse/OOZIE-3367
OOZIE-3366  Update workflow status and subworkflow status on suspend command
https://issues.apache.org/jira/browse/OOZIE-3366
OOZIE-3364  Rerunning Oozie bundle jobs starts the coordinators in 
indeterminate order
https://issues.apache.org/jira/browse/OOZIE-3364
OOZIE-3362  When killed, SSH action should kill the spawned processes on target 
host
https://issues.apache.org/jira/browse/OOZIE-3362
OOZIE-3326  Sqoop Action should support tez delegation tokens for hive-import
https://issues.apache.org/jira/browse/OOZIE-3326
OOZIE-3320  Oozie ShellAction should support absolute bash file path
https://issues.apache.org/jira/browse/OOZIE-3320
OOZIE-3319  Log SSH action callback error output
https://issues.apache.org/jira/browse/OOZIE-3319
OOZIE-3301  Update NOTICE file
https://issues.apache.org/jira/browse/OOZIE-3301
OOZIE-3274  Remove slf4j
https://issues.apache.org/jira/browse/OOZIE-3274
OOZIE-3266  Coord action rerun support RERUN_SKIP_NODES option
https://issues.apache.org/jira/browse/OOZIE-3266
OOZIE-3265  properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to 
appear together
https://issues.apache.org/jira/browse/OOZIE-3265
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3137  Add support for log4j2 in HiveMain
https://issues.apache.org/jira/browse/OOZIE-3137
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3067  Remove duplicated logic from ZKJobsConcurrencyService
https://issues.apache.org/jira/browse/OOZIE-3067
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI 

[jira] [Commented] (OOZIE-3194) [tools] Oozie should set proper permissions to sharelib after upload

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733490#comment-16733490
 ] 

Andras Piros commented on OOZIE-3194:
-

Thanks for the contribution [~dionusos]! +1

> [tools] Oozie should set proper permissions to sharelib after upload
> 
>
> Key: OOZIE-3194
> URL: https://issues.apache.org/jira/browse/OOZIE-3194
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Fix For: 4.3.1
>
> Attachments: OOZIE-3194-v1.patch, OOZIE-3194-v2.patch, 
> OOZIE-3194-v3.patch
>
>
> When user changes HDFS umask that does not support world readability to newly 
> created files, upgrading sharelib causes failing Oozie jobs.
> I suggest set the required permissions to sharelib after upgrading it. Files 
> to 544 and directories to 755.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3194) [tools] Oozie should set proper permissions to sharelib after upload

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-3194:

Summary: [tools] Oozie should set proper permissions to sharelib after 
upload  (was: Oozie should set proper permissions to sharelib after upload)

> [tools] Oozie should set proper permissions to sharelib after upload
> 
>
> Key: OOZIE-3194
> URL: https://issues.apache.org/jira/browse/OOZIE-3194
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Fix For: 4.3.1
>
> Attachments: OOZIE-3194-v1.patch, OOZIE-3194-v2.patch, 
> OOZIE-3194-v3.patch
>
>
> When user changes HDFS umask that does not support world readability to newly 
> created files, upgrading sharelib causes failing Oozie jobs.
> I suggest set the required permissions to sharelib after upgrading it. Files 
> to 544 and directories to 755.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3407) Cleanup TestPurgeXCommand

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733477#comment-16733477
 ] 

Andras Piros commented on OOZIE-3407:
-

Thanks [~asalamon74] for the contribution! Skimmed through the patch on RB.

> Cleanup TestPurgeXCommand
> -
>
> Key: OOZIE-3407
> URL: https://issues.apache.org/jira/browse/OOZIE-3407
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3407-01.patch, OOZIE-3407-02.patch
>
>
> [TestPurgeXCommand|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java]
>  is the longest java source file of Oozie, more than 4000 rows. 
> As part of the Purge Service rewrite we need to use this file to make sure we 
> don't introduce new bugs and probably we also need to add a few more test 
> cases. Before that it would be useful to cleanup this file and extract 
> redundant code to new methods to make the code shorter and more manageable.
> The patch uploaded to OOZIE-3400 introduced three new test cases to this file 
> and shows a simple way to make it shorter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69657: OOZIE-3407 - Cleanup TestPurgeXCommand

[András Piros via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69657/#review211648
---



The patch does a great job into the right direction.

Only a few nit-like comments:

* test method naming
* separate out test classes like `TestWorkflowPurgeXCommand`, 
`TestCoordinatorPurgeXCommand`, `TestBundlePurgeXCommand`
* apply e.g. the builder pattern to set up test bed at the beginning of each 
test method / at the end when it comes to assertions
* separate out calls to `new PurgeXCommand(...).call()` with descriptive method 
names


core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 82 (original), 90 (patched)


Is it absolutely necessary to have yet another `Services` reference? Wasn't 
the existing `protected Services services` enough / working properly?



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 90 (original), 98 (patched)


Is it absolutely necessary to have yet another `Services` reference? Wasn't 
the existing `protected Services services` enough / working properly?



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 103 (original), 111 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 144 (original), 126 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 185 (original), 141 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 226 (original), 156 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 266 (original), 171 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 306 (original), 187 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 346 (original), 203 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 386 (original), 219 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 423 (original), 235 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 476 (original), 257 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 529 (original), 280 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 582 (original), 302 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 631 (original), 324 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 693 (original), 347 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 758 (original), 370 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 818 (original), 391 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 879 (original), 413 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java
Line 1075 (original), 439 (patched)


Please provide a more descriptive name.



core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java

[jira] [Commented] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733463#comment-16733463
 ] 

Hadoop QA commented on OOZIE-3409:
--


Testing JIRA OOZIE-3409

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 3106
.Tests failed : 0
.Tests in error   : 2
.Tests timed out  : 0

{color:red}-1{color} [ERROR] There are [2] test errors in [examples]. Listing 
only the first [5] ones
testLocalOozieExampleEnd:org.apache.oozie.example.TestLocalOozieExample
testLocalOozieExampleKill:org.apache.oozie.example.TestLocalOozieExample

Check console output for the full list of errors/failures
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeCoordWithWFChild1
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/961/



> Oozie Server : Possible memory leak in EL evaluation
> 
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3409-01.patch
>
>
> I have noticed that there is a potential memory leak in the Oozie server due 
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers 
> have been running out of memory occasionally. It happens more often when we 
> have more activity. Upon analysis of the heap dump, it seems that the culprit 
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from 
> {{commons-el}} which is used in the Oozie server through 
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses 
> a static 

Failed: OOZIE-3409 PreCommit Build #961

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3409
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/961/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.93 MB...]
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2558 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
xargs: WARNING: a NUL character occurred in the input.  It cannot be passed 
through in the argument list.  Did you mean to use the --null option?
  Running test-patch task DISTRO


Testing JIRA OOZIE-3409

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
-1 the patch contains 1 line(s) longer than 132 characters
+1 the patch adds/modifies 2 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/git].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 3106
Tests failed : 0
Tests in error   : 2
Tests timed out  : 0

-1 [ERROR] There are [2] test errors in [examples]. Listing only the first [5] 
ones
testLocalOozieExampleEnd:org.apache.oozie.example.TestLocalOozieExample
testLocalOozieExampleKill:org.apache.oozie.example.TestLocalOozieExample

Check console output for the full list of errors/failures
Tests failed at first run:
TestPurgeXCommand#testPurgeCoordWithWFChild1
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/961/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13205493/comment/16733463","id":"16733463","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3409\n\nCleaning local git 
workspace\n\n\n\n{color:green}+1 
PATCH_APPLIES{color}\n{color:green}+1 CLEAN{color}\n{color:red}-1 
RAW_PATCH_ANALYSIS{color}\n.{color:green}+1{color} the patch does not 

[jira] [Commented] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation

[Praneeth (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733459#comment-16733459
 ] 

Praneeth commented on OOZIE-3409:
-

Thanks [~asalamon74]. {{tomcat-jasper}} seems to be a clean approach.

> Oozie Server : Possible memory leak in EL evaluation
> 
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3409-01.patch
>
>
> I have noticed that there is a potential memory leak in the Oozie server due 
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers 
> have been running out of memory occasionally. It happens more often when we 
> have more activity. Upon analysis of the heap dump, it seems that the culprit 
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from 
> {{commons-el}} which is used in the Oozie server through 
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses 
> a static {{Map}} [2] to cache the EL evaluations. This map never gets 
> cleared, eventually taking up all the memory. So, in an environment with 
> heavy usage and long running oozie servers, this would lead to an OOM. The 
> configurations for workflow actions we have are possibly larger than normal 
> which is worsening the situation as the Map has to maintain larger strings.
>  # 
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
>  # 
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733314#comment-16733314
 ] 

Hadoop QA commented on OOZIE-3409:
--

PreCommit-OOZIE-Build started


> Oozie Server : Possible memory leak in EL evaluation
> 
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3409-01.patch
>
>
> I have noticed that there is a potential memory leak in the Oozie server due 
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers 
> have been running out of memory occasionally. It happens more often when we 
> have more activity. Upon analysis of the heap dump, it seems that the culprit 
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from 
> {{commons-el}} which is used in the Oozie server through 
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses 
> a static {{Map}} [2] to cache the EL evaluations. This map never gets 
> cleared, eventually taking up all the memory. So, in an environment with 
> heavy usage and long running oozie servers, this would lead to an OOM. The 
> configurations for workflow actions we have are possibly larger than normal 
> which is worsening the situation as the Map has to maintain larger strings.
>  # 
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
>  # 
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-3067 PreCommit Build #960

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3067
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/960/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.91 MB...]
[DEBUG] There are no new bugs found in [fluent-job/fluent-job-api].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [tools].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2561 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3067

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/git].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3106
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/960/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  32810 0  100  3281  0   5012 --:--:-- --:--:-- --:--:--  
5009{"self":"https://issues.apache.org/jira/rest/api/2/issue/13104292/comment/16733263","id":"16733263","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 

[jira] [Commented] (OOZIE-3067) Remove duplicated logic from ZKJobsConcurrencyService

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733263#comment-16733263
 ] 

Hadoop QA commented on OOZIE-3067:
--


Testing JIRA OOZIE-3067

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3106
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/960/



> Remove duplicated logic from ZKJobsConcurrencyService
> -
>
> Key: OOZIE-3067
> URL: https://issues.apache.org/jira/browse/OOZIE-3067
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.2.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk
>
> Attachments: 
> 0001-OOZIE-3067-Remove-duplicated-logic-from-ZKJobsConcur.patch, 
> OOZIE-3067.002.patch
>
>
> In ZKJobsConcurrencyService getServerUrls and getOtherServerUrls methods 
> basically the same. However, the logic is written inside both methods.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3186) Oozie is unable to use configuration linked using jceks://file/...

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733175#comment-16733175
 ] 

Hadoop QA commented on OOZIE-3186:
--


Testing JIRA OOZIE-3186

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3110
.{color:orange}Tests failed at first run:{color}
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/959/



> Oozie is unable to use configuration linked using jceks://file/...
> --
>
> Key: OOZIE-3186
> URL: https://issues.apache.org/jira/browse/OOZIE-3186
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0b1, 5.0.0, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>  Labels: usability
> Fix For: trunk
>
> Attachments: OOZIE-3186-001.patch, OOZIE-3186-002.patch, 
> OOZIE-3186-003.patch
>
>
> When Oozie is used with Ambari, the next configuration makes Oozie fail to 
> start:
> {noformat}
> 
>     hadoop.security.credential.provider.path
>     jceks://file/.../oozie-site.jceks
> 
> {noformat}
> Value should have *localjceks://* instead of *jceks://*. But Ambari does not 
> let change this value. I propose change the url when Oozie loads it.
>  
> Stacktrace, when the issue occurs:
> {code:java}
> org.apache.oozie.service.ServiceException: E0103: Could not load service 
> classes, Could not load password for [oozie.service.JPAService.jdbc.password] 
> at org.apache.oozie.service.Services.loadServices(Services.java:309) at 
> org.apache.oozie.service.Services.init(Services.java:213) at 
> 

Failed: OOZIE-3186 PreCommit Build #959

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3186
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/959/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.91 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hcatalog].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive2].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/git].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/streaming].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2560 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3186

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/git].
 +1 There are no new bugs found in [sharelib/streaming].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3110
Tests failed at first run:
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/959/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13141469/comment/16733175","id":"16733175","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3186\n\nCleaning local git 

[jira] [Commented] (OOZIE-3067) Remove duplicated logic from ZKJobsConcurrencyService

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733158#comment-16733158
 ] 

Hadoop QA commented on OOZIE-3067:
--

PreCommit-OOZIE-Build started


> Remove duplicated logic from ZKJobsConcurrencyService
> -
>
> Key: OOZIE-3067
> URL: https://issues.apache.org/jira/browse/OOZIE-3067
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 4.2.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Minor
> Fix For: trunk
>
> Attachments: 
> 0001-OOZIE-3067-Remove-duplicated-logic-from-ZKJobsConcur.patch, 
> OOZIE-3067.002.patch
>
>
> In ZKJobsConcurrencyService getServerUrls and getOtherServerUrls methods 
> basically the same. However, the logic is written inside both methods.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3194) Oozie should set proper permissions to sharelib after upload

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733111#comment-16733111
 ] 

Hadoop QA commented on OOZIE-3194:
--


Testing JIRA OOZIE-3194

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3106
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/958/



> Oozie should set proper permissions to sharelib after upload
> 
>
> Key: OOZIE-3194
> URL: https://issues.apache.org/jira/browse/OOZIE-3194
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Fix For: 4.3.1
>
> Attachments: OOZIE-3194-v1.patch, OOZIE-3194-v2.patch, 
> OOZIE-3194-v3.patch
>
>
> When user changes HDFS umask that does not support world readability to newly 
> created files, upgrading sharelib causes failing Oozie jobs.
> I suggest set the required permissions to sharelib after upgrading it. Files 
> to 544 and directories to 755.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-3194 PreCommit Build #958

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3194
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/958/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.91 MB...]
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [tools].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [fluent-job/fluent-job-api].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2560 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3194

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/git].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3106
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/958/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  32640 0  100  3264  0   3330 --:--:-- --:--:-- --:--:--  
3330{"self":"https://issues.apache.org/jira/rest/api/2/issue/13143504/comment/16733111","id":"16733111","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 

[jira] [Commented] (OOZIE-3186) Oozie is unable to use configuration linked using jceks://file/...

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733078#comment-16733078
 ] 

Hadoop QA commented on OOZIE-3186:
--

PreCommit-OOZIE-Build started


> Oozie is unable to use configuration linked using jceks://file/...
> --
>
> Key: OOZIE-3186
> URL: https://issues.apache.org/jira/browse/OOZIE-3186
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0b1, 5.0.0, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>  Labels: usability
> Fix For: trunk
>
> Attachments: OOZIE-3186-001.patch, OOZIE-3186-002.patch, 
> OOZIE-3186-003.patch
>
>
> When Oozie is used with Ambari, the next configuration makes Oozie fail to 
> start:
> {noformat}
> 
>     hadoop.security.credential.provider.path
>     jceks://file/.../oozie-site.jceks
> 
> {noformat}
> Value should have *localjceks://* instead of *jceks://*. But Ambari does not 
> let change this value. I propose change the url when Oozie loads it.
>  
> Stacktrace, when the issue occurs:
> {code:java}
> org.apache.oozie.service.ServiceException: E0103: Could not load service 
> classes, Could not load password for [oozie.service.JPAService.jdbc.password] 
> at org.apache.oozie.service.Services.loadServices(Services.java:309) at 
> org.apache.oozie.service.Services.init(Services.java:213) at 
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
>  at 
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276)
>  at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779) 
> at 
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)
>  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780) 
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) at 
> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) at 
> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) at 
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) at 
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) at 
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) at 
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>  at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069) at 
> org.apache.catalina.core.StandardHost.start(StandardHost.java:822) at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061) at 
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at 
> org.apache.catalina.core.StandardService.start(StandardService.java:525) at 
> org.apache.catalina.core.StandardServer.start(StandardServer.java:761) at 
> org.apache.catalina.startup.Catalina.start(Catalina.java:595) at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at 
> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: 
> java.lang.IllegalArgumentException: Could not load password for 
> [oozie.service.JPAService.jdbc.password] at 
> org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:615)
>  at 
> org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:602)
>  at org.apache.oozie.service.JPAService.init(JPAService.java:147) at 
> org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at 
> org.apache.oozie.service.Services.setService(Services.java:372) at 
> org.apache.oozie.service.Services.loadServices(Services.java:305) ... 26 more 
> Caused by: java.lang.reflect.InvocationTargetException at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:608)
>  ... 31 more Caused by: java.lang.UnsupportedOperationException: Accessing 
> local file system is not allowed at 
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
>  at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47) 
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795) at 
> 

[jira] [Updated] (OOZIE-3186) Oozie is unable to use configuration linked using jceks://file/...

[Denes Bodo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denes Bodo updated OOZIE-3186:
--
Attachment: OOZIE-3186-003.patch

> Oozie is unable to use configuration linked using jceks://file/...
> --
>
> Key: OOZIE-3186
> URL: https://issues.apache.org/jira/browse/OOZIE-3186
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Affects Versions: 5.0.0b1, 5.0.0, 4.3.1
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>  Labels: usability
> Fix For: trunk
>
> Attachments: OOZIE-3186-001.patch, OOZIE-3186-002.patch, 
> OOZIE-3186-003.patch
>
>
> When Oozie is used with Ambari, the next configuration makes Oozie fail to 
> start:
> {noformat}
> 
>     hadoop.security.credential.provider.path
>     jceks://file/.../oozie-site.jceks
> 
> {noformat}
> Value should have *localjceks://* instead of *jceks://*. But Ambari does not 
> let change this value. I propose change the url when Oozie loads it.
>  
> Stacktrace, when the issue occurs:
> {code:java}
> org.apache.oozie.service.ServiceException: E0103: Could not load service 
> classes, Could not load password for [oozie.service.JPAService.jdbc.password] 
> at org.apache.oozie.service.Services.loadServices(Services.java:309) at 
> org.apache.oozie.service.Services.init(Services.java:213) at 
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46)
>  at 
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276)
>  at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779) 
> at 
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803)
>  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780) 
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) at 
> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) at 
> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) at 
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) at 
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) at 
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) at 
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>  at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069) at 
> org.apache.catalina.core.StandardHost.start(StandardHost.java:822) at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061) at 
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at 
> org.apache.catalina.core.StandardService.start(StandardService.java:525) at 
> org.apache.catalina.core.StandardServer.start(StandardServer.java:761) at 
> org.apache.catalina.startup.Catalina.start(Catalina.java:595) at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at 
> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: 
> java.lang.IllegalArgumentException: Could not load password for 
> [oozie.service.JPAService.jdbc.password] at 
> org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:615)
>  at 
> org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:602)
>  at org.apache.oozie.service.JPAService.init(JPAService.java:147) at 
> org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at 
> org.apache.oozie.service.Services.setService(Services.java:372) at 
> org.apache.oozie.service.Services.loadServices(Services.java:305) ... 26 more 
> Caused by: java.lang.reflect.InvocationTargetException at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:608)
>  ... 31 more Caused by: java.lang.UnsupportedOperationException: Accessing 
> local file system is not allowed at 
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
>  at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47) 
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795) at 
> 

[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733051#comment-16733051
 ] 

Julia Kinga Marton commented on OOZIE-3218:
---

+1 for the quote usage. With this approach we can be in sync with the Sqoop 
command line as well.

> Oozie Sqoop action with command splits the select clause into multiple parts 
> due to delimiter being space
> -
>
> Key: OOZIE-3218
> URL: https://issues.apache.org/jira/browse/OOZIE-3218
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Environment: Hortonworks Hadoop HDP-2.6.4.x release 
>  oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91
>Reporter: Mahesh Balakrishnan
>Assignee: Mahesh Balakrishnan
>Priority: Major
> Attachments: OOZIE-3218-2.patch, OOZIE-3218-3.patch, OOZIE-3218.patch
>
>
> When running a Oozie Sqoop action which has command with {{--query}} in place 
> the query is split into multiple parts causing {{"Unrecognized argument:"}} 
> and in-turn fails.
> {code:xml}
> 
> ${resourceManager}
> ${nameNode}
> import --verbose --connect jdbc:mysql://test.openstacklocal/db 
> --query select * from abc where $CONDITIONS --username test --password test 
> --driver com.mysql.jdbc.Driver -m 1 
> 
> 
> {code}
>  
> Oozie Launcher logs:
> {noformat}
>  Sqoop command arguments :
>  import
>  --verbose
>  --connect
>  jdbc:mysql://test.openstacklocal/db
>  --query
>  "select
>  *
>  from
>  abc
>  where
>  $CONDITIONS"
>  --username
>  hive
>  --password
>  
>  --driver
>  com.mysql.jdbc.Driver
>  -m
>  1
>  Fetching child yarn jobs
>  tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3
>  2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to 
> ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050
>  Child yarn jobs are found - 
>  =
> >>> Invoking Sqoop command line now >>>
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --driver
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --driver
>  3288 [main] ERROR 

[jira] [Commented] (OOZIE-3194) Oozie should set proper permissions to sharelib after upload

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732997#comment-16732997
 ] 

Hadoop QA commented on OOZIE-3194:
--

PreCommit-OOZIE-Build started


> Oozie should set proper permissions to sharelib after upload
> 
>
> Key: OOZIE-3194
> URL: https://issues.apache.org/jira/browse/OOZIE-3194
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Fix For: 4.3.1
>
> Attachments: OOZIE-3194-v1.patch, OOZIE-3194-v2.patch, 
> OOZIE-3194-v3.patch
>
>
> When user changes HDFS umask that does not support world readability to newly 
> created files, upgrading sharelib causes failing Oozie jobs.
> I suggest set the required permissions to sharelib after upgrading it. Files 
> to 544 and directories to 755.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3194) Oozie should set proper permissions to sharelib after upload

[Denes Bodo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denes Bodo updated OOZIE-3194:
--
Attachment: OOZIE-3194-v3.patch

> Oozie should set proper permissions to sharelib after upload
> 
>
> Key: OOZIE-3194
> URL: https://issues.apache.org/jira/browse/OOZIE-3194
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
> Fix For: 4.3.1
>
> Attachments: OOZIE-3194-v1.patch, OOZIE-3194-v2.patch, 
> OOZIE-3194-v3.patch
>
>
> When user changes HDFS umask that does not support world readability to newly 
> created files, upgrading sharelib causes failing Oozie jobs.
> I suggest set the required permissions to sharelib after upgrading it. Files 
> to 544 and directories to 755.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732866#comment-16732866
 ] 

Hadoop QA commented on OOZIE-3218:
--


Testing JIRA OOZIE-3218

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 3106
.Tests failed : 1
.Tests in error   : 0
.Tests timed out  : 0

{color:red}-1{color} [ERROR] There are [1] test failures in [sharelib]. Listing 
only the first [5] ones
testSqoopActionWithRedundantPrefix:org.apache.oozie.action.hadoop.TestSqoopActionExecutor

Check console output for the full list of errors/failures
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild2
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/957/



> Oozie Sqoop action with command splits the select clause into multiple parts 
> due to delimiter being space
> -
>
> Key: OOZIE-3218
> URL: https://issues.apache.org/jira/browse/OOZIE-3218
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Environment: Hortonworks Hadoop HDP-2.6.4.x release 
>  oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91
>Reporter: Mahesh Balakrishnan
>Assignee: Mahesh Balakrishnan
>Priority: Major
> Attachments: OOZIE-3218-2.patch, OOZIE-3218-3.patch, OOZIE-3218.patch
>
>
> When running a Oozie Sqoop action which has command with {{--query}} in place 
> the query is split into multiple parts causing {{"Unrecognized argument:"}} 
> and in-turn fails.
> {code:xml}
> 
> 

Failed: OOZIE-3218 PreCommit Build #957

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3218
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/957/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.92 MB...]
[DEBUG] There are no new bugs found in [core].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2560 bytes
[TRACE] Full summary file size is 1471 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
xargs: WARNING: a NUL character occurred in the input.  It cannot be passed 
through in the argument list.  Did you mean to use the --null option?
  Running test-patch task DISTRO


Testing JIRA OOZIE-3218

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
WARNING: the current HEAD has 100 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/git].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 3106
Tests failed : 1
Tests in error   : 0
Tests timed out  : 0

-1 [ERROR] There are [1] test failures in [sharelib]. Listing only the first 
[5] ones
testSqoopActionWithRedundantPrefix:org.apache.oozie.action.hadoop.TestSqoopActionExecutor

Check console output for the full list of errors/failures
Tests failed at first run:
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild2
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/957/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  38140 0  100  3814  0   4381 --:--:-- --:--:-- --:--:--  
4378{"self":"https://issues.apache.org/jira/rest/api/2/issue/13152080/comment/16732866","id":"16732866","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-3218\n\nCleaning local git 
workspace\n\n\n\n{color:green}+1 
PATCH_APPLIES{color}\n{color:green}+1 

[jira] [Commented] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732856#comment-16732856
 ] 

Andras Piros commented on OOZIE-3409:
-

Thanks [~asalamon74] for the investigation. Can you make a try w/ 
{{org.apache.tomcat:jasper-el}} latest version running w/ Oozie's Jetty 9?

> Oozie Server : Possible memory leak in EL evaluation
> 
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due 
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers 
> have been running out of memory occasionally. It happens more often when we 
> have more activity. Upon analysis of the heap dump, it seems that the culprit 
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from 
> {{commons-el}} which is used in the Oozie server through 
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses 
> a static {{Map}} [2] to cache the EL evaluations. This map never gets 
> cleared, eventually taking up all the memory. So, in an environment with 
> heavy usage and long running oozie servers, this would lead to an OOM. The 
> configurations for workflow actions we have are possibly larger than normal 
> which is worsening the situation as the Map has to maintain larger strings.
>  # 
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
>  # 
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation

[Andras Salamon (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732850#comment-16732850
 ] 

Andras Salamon commented on OOZIE-3409:
---

In the EL-1 bugreport they suggest to use tomcat's jasper el instead of commons 
el: [https://markmail.org/message/nm4c3zwzmxhuhplu]

 

> Oozie Server : Possible memory leak in EL evaluation
> 
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due 
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers 
> have been running out of memory occasionally. It happens more often when we 
> have more activity. Upon analysis of the heap dump, it seems that the culprit 
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from 
> {{commons-el}} which is used in the Oozie server through 
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses 
> a static {{Map}} [2] to cache the EL evaluations. This map never gets 
> cleared, eventually taking up all the memory. So, in an environment with 
> heavy usage and long running oozie servers, this would lead to an OOM. The 
> configurations for workflow actions we have are possibly larger than normal 
> which is worsening the situation as the Map has to maintain larger strings.
>  # 
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
>  # 
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation

[Andras Salamon (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732827#comment-16732827
 ] 

Andras Salamon edited comment on OOZIE-3409 at 1/3/19 9:55 AM:
---

The commons.el bug reported in 2005: EL-1


was (Author: asalamon74):
The commons.el bug reported in 2005.

> Oozie Server : Possible memory leak in EL evaluation
> 
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due 
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers 
> have been running out of memory occasionally. It happens more often when we 
> have more activity. Upon analysis of the heap dump, it seems that the culprit 
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from 
> {{commons-el}} which is used in the Oozie server through 
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses 
> a static {{Map}} [2] to cache the EL evaluations. This map never gets 
> cleared, eventually taking up all the memory. So, in an environment with 
> heavy usage and long running oozie servers, this would lead to an OOM. The 
> configurations for workflow actions we have are possibly larger than normal 
> which is worsening the situation as the Map has to maintain larger strings.
>  # 
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
>  # 
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3409) Oozie Server : Possible memory leak in EL evaluation

[Andras Salamon (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732827#comment-16732827
 ] 

Andras Salamon commented on OOZIE-3409:
---

The commons.el bug reported in 2005.

> Oozie Server : Possible memory leak in EL evaluation
> 
>
> Key: OOZIE-3409
> URL: https://issues.apache.org/jira/browse/OOZIE-3409
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.1.0
>Reporter: Praneeth
>Assignee: Andras Salamon
>Priority: Major
>
> I have noticed that there is a potential memory leak in the Oozie server due 
> to one of its dependencies - {{commons-el:commons-el:1.0}}. Our oozie servers 
> have been running out of memory occasionally. It happens more often when we 
> have more activity. Upon analysis of the heap dump, it seems that the culprit 
> here is the {{org.apache.commons.el.ExpressionEvaluatorImpl}} from 
> {{commons-el}} which is used in the Oozie server through 
> {{org.apache.oozie.util.ELEvaluator}} [1] . {{ExpressionEvaluatorImpl}} uses 
> a static {{Map}} [2] to cache the EL evaluations. This map never gets 
> cleared, eventually taking up all the memory. So, in an environment with 
> heavy usage and long running oozie servers, this would lead to an OOM. The 
> configurations for workflow actions we have are possibly larger than normal 
> which is worsening the situation as the Map has to maintain larger strings.
>  # 
> [https://github.com/apache/oozie/blob/release-5.1.0/core/src/main/java/org/apache/oozie/util/ELEvaluator.java#L143]
>  # 
> [http://svn.apache.org/viewvc/commons/dormant/el/trunk/src/java/org/apache/commons/el/ExpressionEvaluatorImpl.java?view=markup#l94]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

[Andras Salamon (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732797#comment-16732797
 ] 

Andras Salamon commented on OOZIE-3218:
---

It seems to me both JIRAs address the same problem, but provide a different 
solution. If sqoop command contains the query string like this: {{--query 
select * from abc}} Oozie incorrectly splits it into multiple parts instead of 
two tokens "--query" and "select * from abc".

OOZIE-3218 suggests to recognize {{--query}} as a tokenstarter and will keep 
the next arguments as a single token till the next token starter (e.g. 
--username). OOZIE-2949 suggests to use single or double quotes to denote the 
tokens and use {{--query 'select * from abc'}} so it will be easier to 
recognize {{select * from abc}} as a single token.

We have to choose which approach is better and implement that one and close the 
other Jira as duplicate.

I think OOZIE-2949 gives a better solution, we should require the usage of 
single or double quotes. Without the quotes bash treats * as a wildcard, hence 
Sqoop itself suggest to use single quotes in the command line: 
[https://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html#_free_form_query_imports]

Allow the users to copy a working sqoop command line to Oozie XML would be the 
most convenient for the users.

[~mbalakrishnan] [~dionusos] [~gezapeti] [~andras.piros] What are your opinions?

> Oozie Sqoop action with command splits the select clause into multiple parts 
> due to delimiter being space
> -
>
> Key: OOZIE-3218
> URL: https://issues.apache.org/jira/browse/OOZIE-3218
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Environment: Hortonworks Hadoop HDP-2.6.4.x release 
>  oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91
>Reporter: Mahesh Balakrishnan
>Assignee: Mahesh Balakrishnan
>Priority: Major
> Attachments: OOZIE-3218-2.patch, OOZIE-3218-3.patch, OOZIE-3218.patch
>
>
> When running a Oozie Sqoop action which has command with {{--query}} in place 
> the query is split into multiple parts causing {{"Unrecognized argument:"}} 
> and in-turn fails.
> {code:xml}
> 
> ${resourceManager}
> ${nameNode}
> import --verbose --connect jdbc:mysql://test.openstacklocal/db 
> --query select * from abc where $CONDITIONS --username test --password test 
> --driver com.mysql.jdbc.Driver -m 1 
> 
> 
> {code}
>  
> Oozie Launcher logs:
> {noformat}
>  Sqoop command arguments :
>  import
>  --verbose
>  --connect
>  jdbc:mysql://test.openstacklocal/db
>  --query
>  "select
>  *
>  from
>  abc
>  where
>  $CONDITIONS"
>  --username
>  hive
>  --password
>  
>  --driver
>  com.mysql.jdbc.Driver
>  -m
>  1
>  Fetching child yarn jobs
>  tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3
>  2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to 
> ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050
>  Child yarn jobs are found - 
>  =
> >>> Invoking Sqoop command line now >>>
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR 

[jira] [Commented] (OOZIE-3405) SSH action shows empty error Message and Error code

[Peter Orova (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732765#comment-16732765
 ] 

Peter Orova commented on OOZIE-3405:


[~asalamon74] Thank you for your insight, so setting the {{errorCode}} is off 
the table. Do you think we could still provide some information via the 
{{wf:errorMessage(String message)}} function? If we stored the return value of 
the script in the {{.error}} file, could we return that via the 
{{errorMessage}} function in the case of SSH actions?

> SSH action shows empty error Message and Error code
> ---
>
> Key: OOZIE-3405
> URL: https://issues.apache.org/jira/browse/OOZIE-3405
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Peter Orova
>Assignee: Peter Orova
>Priority: Minor
>
> Currently, when an SSH action fails the only message that is returned is the 
> Status. Neither the {{error Message}} nor {{Error code}} fields are filled. 
> This makes reporting on the causes of SSH Action failures via Oozie highly 
> impractical: the only meaningful bit of information there is on a failed SSH 
> Action is the Status.
> The Status is filled based on the presence (or lack of) the {{.error file}} 
> that is produced in case the user submitted script returns with any other 
> value than 0.
> {noformat}
> SshActionExecutor#getActionStatus
>  ...
>  String outFile = getRemoteFileName(context, action, "error", false, true);
>  String checkErrorCmd = SSH_COMMAND_BASE + action.getTrackerUri() + " ls " + 
> outFile;
>  int retVal = getReturnValue(checkErrorCmd);
>  ...
> {noformat}
>  
>  User requirement is to provide some more detailed information on the 
> success/failure of the user-submitted script. That could be at a minimum the 
> return value, optionally the last ~1K of the stderr that is drained. This 
> information could then be communicated via {{errorMessage}} and {{ErrorCode}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732759#comment-16732759
 ] 

Hadoop QA commented on OOZIE-3218:
--

PreCommit-OOZIE-Build started


> Oozie Sqoop action with command splits the select clause into multiple parts 
> due to delimiter being space
> -
>
> Key: OOZIE-3218
> URL: https://issues.apache.org/jira/browse/OOZIE-3218
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Environment: Hortonworks Hadoop HDP-2.6.4.x release 
>  oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91
>Reporter: Mahesh Balakrishnan
>Assignee: Mahesh Balakrishnan
>Priority: Major
> Attachments: OOZIE-3218-2.patch, OOZIE-3218-3.patch, OOZIE-3218.patch
>
>
> When running a Oozie Sqoop action which has command with {{--query}} in place 
> the query is split into multiple parts causing {{"Unrecognized argument:"}} 
> and in-turn fails.
> {code:xml}
> 
> ${resourceManager}
> ${nameNode}
> import --verbose --connect jdbc:mysql://test.openstacklocal/db 
> --query select * from abc where $CONDITIONS --username test --password test 
> --driver com.mysql.jdbc.Driver -m 1 
> 
> 
> {code}
>  
> Oozie Launcher logs:
> {noformat}
>  Sqoop command arguments :
>  import
>  --verbose
>  --connect
>  jdbc:mysql://test.openstacklocal/db
>  --query
>  "select
>  *
>  from
>  abc
>  where
>  $CONDITIONS"
>  --username
>  hive
>  --password
>  
>  --driver
>  com.mysql.jdbc.Driver
>  -m
>  1
>  Fetching child yarn jobs
>  tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3
>  2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to 
> ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050
>  Child yarn jobs are found - 
>  =
> >>> Invoking Sqoop command line now >>>
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --driver
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --driver
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: com.mysql.jdbc.Driver
>  3288 [main] 

[jira] [Commented] (OOZIE-3218) Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732753#comment-16732753
 ] 

Andras Piros commented on OOZIE-3218:
-

[~mbalakrishnan] [~asalamon74] whoever is working on either issues OOZIE-3218 
and OOZIE-2949 please make sure both issues are handled by a single patch. The 
other issue can be closed as duplicate.

> Oozie Sqoop action with command splits the select clause into multiple parts 
> due to delimiter being space
> -
>
> Key: OOZIE-3218
> URL: https://issues.apache.org/jira/browse/OOZIE-3218
> Project: Oozie
>  Issue Type: Bug
>  Components: action, workflow
>Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Environment: Hortonworks Hadoop HDP-2.6.4.x release 
>  oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91
>Reporter: Mahesh Balakrishnan
>Assignee: Mahesh Balakrishnan
>Priority: Major
> Attachments: OOZIE-3218-2.patch, OOZIE-3218-3.patch, OOZIE-3218.patch
>
>
> When running a Oozie Sqoop action which has command with {{--query}} in place 
> the query is split into multiple parts causing {{"Unrecognized argument:"}} 
> and in-turn fails.
> {code:xml}
> 
> ${resourceManager}
> ${nameNode}
> import --verbose --connect jdbc:mysql://test.openstacklocal/db 
> --query select * from abc where $CONDITIONS --username test --password test 
> --driver com.mysql.jdbc.Driver -m 1 
> 
> 
> {code}
>  
> Oozie Launcher logs:
> {noformat}
>  Sqoop command arguments :
>  import
>  --verbose
>  --connect
>  jdbc:mysql://test.openstacklocal/db
>  --query
>  "select
>  *
>  from
>  abc
>  where
>  $CONDITIONS"
>  --username
>  hive
>  --password
>  
>  --driver
>  com.mysql.jdbc.Driver
>  -m
>  1
>  Fetching child yarn jobs
>  tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3
>  2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to 
> ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050
>  Child yarn jobs are found - 
>  =
> >>> Invoking Sqoop command line now >>>
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not 
> been set in the environment. Cannot check for additional configuration.
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 
> 1.4.6.2.6.4.0-91
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug 
> logging.
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing 
> arguments for import:
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: *
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: from
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: where
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: $CONDITIONS"
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --username
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --password
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: abc
>  3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized 
> argument: --driver
>  3288 [main] ERROR 

[jira] [Commented] (OOZIE-3407) Cleanup TestPurgeXCommand

[Andras Salamon (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732744#comment-16732744
 ] 

Andras Salamon commented on OOZIE-3407:
---

Review Board has been fixed, created a review request there.

> Cleanup TestPurgeXCommand
> -
>
> Key: OOZIE-3407
> URL: https://issues.apache.org/jira/browse/OOZIE-3407
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: trunk
>Reporter: Andras Salamon
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: OOZIE-3407-01.patch, OOZIE-3407-02.patch
>
>
> [TestPurgeXCommand|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java]
>  is the longest java source file of Oozie, more than 4000 rows. 
> As part of the Purge Service rewrite we need to use this file to make sure we 
> don't introduce new bugs and probably we also need to add a few more test 
> cases. Before that it would be useful to cleanup this file and extract 
> redundant code to new methods to make the code shorter and more manageable.
> The patch uploaded to OOZIE-3400 introduced three new test cases to this file 
> and shows a simple way to make it shorter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69657: OOZIE-3407 - Cleanup TestPurgeXCommand

[Andras Salamon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69657/
---

Review request for oozie, András Piros and Kinga Marton.


Repository: oozie-git


Description
---

OOZIE-3407 - Cleanup TestPurgeXCommand


Diffs
-

  core/src/test/java/org/apache/oozie/command/TestPurgeXCommand.java 
107547d87d294d600986d361ea1a6f92298e97f7 


Diff: https://reviews.apache.org/r/69657/diff/1/


Testing
---

Running unit tests localy


Thanks,

Andras Salamon