[jira] [Commented] (OOZIE-3300) Hadoop 3 Pig (with Tez) action shall not use given distributed cache

2018-07-06 Thread Daniel Dai (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535444#comment-16535444
 ] 

Daniel Dai commented on OOZIE-3300:
---

I will fix it on Pig side (PIG-4373). Please close this ticket.

> Hadoop 3 Pig (with Tez) action shall not use given distributed cache
> 
>
> Key: OOZIE-3300
> URL: https://issues.apache.org/jira/browse/OOZIE-3300
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1
> Environment: Oozie-4.3.1
> Hadoop-3
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>  Labels: usability
> Attachments: OOZIE-3300_001.patch
>
>
> When I run my Pig action and set -x Tez to use Tez as the execution engine, I 
> got the following error:
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "mycluster" port: -1 file: 
> "/tmp/temp257132233/hive-exec-3.0.0.3.0.0.0-1469.jar" conflicts with cache 
> file (mapreduce.job.cache.files)
>  
> hdfs://mycluster/user/oozie/share/lib/lib_20180611041054/pig/hive-exec-3.0.0.3.0.0.0-1469.jar
> {code}
> or
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "ns1" port: -1 file: 
> "/tmp/temp36150863/joda-time-2.9.6.jar" conflicts with cache file 
> (mapreduce.job.cache.files) 
> hdfs://ns1/user/oozie/share/lib/lib_20180612003013/pig/joda-time-2.9.6.jar
> {code}
> When I exclude the conflicting jar, then Pig cannot start due to class not 
> found exception.
> I think a workaround or solution? could be if we exclude the 
> *mapreduce.job.cache.files* and *mapreduce.job.cache.archives* from 
> pig.properties file.
>  
> It is a bit scary change because I am not sure if the user's job shall use 
> specific driver or implementation which needs to be on distributed cache.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-07-06 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534946#comment-16534946
 ] 

Hadoop QA commented on OOZIE-2829:
--


Testing JIRA OOZIE-2829

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:red}-1{color} There are [1] new bugs found below threshold in [tools] 
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
. The most important FindBugs errors are:
. At OozieSharelibCLI.java:[line 182]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
. At OozieSharelibCLI.java:[line 182]: At OozieSharelibCLI.java:[line 174]
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2910
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/659/



> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2829-006.patch, OOZIE-2829-007.patch, 
> OOZIE-2829-008.patch, OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-2829 PreCommit Build #659

2018-07-06 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2829
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/659/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.74 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[ERROR] There are [1] new bugs found below threshold in [tools].
[DEBUG] You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [fluent-job/fluent-job-api].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[ERROR] There are [1] new bugs found below threshold in total that must be 
fixed.
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2880 bytes
[TRACE] Full summary file size is 1829 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-2829

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 5 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
-1 There are [1] new bugs found below threshold in total that must be fixed.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [webapp].
 -1 There are [1] new bugs found below threshold in [tools] that must be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
 The most important FindBugs errors are:
 At OozieSharelibCLI.java:[line 182]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
 At OozieSharelibCLI.java:[line 182]: At OozieSharelibCLI.java:[line 174]
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2910
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/659/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  35380 0  100  3538  0   6034 --:--:-- --:--:-- --:--:--

Re: Review Request 67834: OOZIE-2829 Improve sharelib upload to accept multiple source folders

2018-07-06 Thread András Piros via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67834/#review205796
---




docs/src/site/twiki/AG_Install.twiki
Lines 60 (patched)


Typo: comma.



docs/src/site/twiki/AG_Install.twiki
Lines 61 (patched)


Would be better to use mixed separators as per functionality:
```

sharelib-name-1=path-name-1,path-name-2:sharelib-name-2=path-name-1,path-name-2
```



tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 74-86 (patched)


Extracting `System.lineSeparator()` to a constant field would increase 
readability.



tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 88 (patched)


Problem is we can have ```hdfs://my/jar.jar#myjar.jar``` inside sharelib 
paths.

Best is to go with e.g. `;` that cannot be present as HDFS or local file.



tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 264 (patched)


Shouldn't it be `System.out`?



tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 282-294 (patched)


A log message about how parallel we are running would be nice.



tools/src/test/java/org/apache/oozie/tools/OozieSharelibFileOperations.java
Lines 42-48 (original), 43-51 (patched)


Could be extracted to a nested `static class`, and tested separately.



tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java
Lines 181-182 (original), 112-113 (patched)


Multiple testing scenarios missing:

* empty value to a key present
* multiple values to a key
* value without a key

Please use a separate test file.



tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java
Lines 116-118 (patched)


`message` missing.



tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java
Lines 143 (patched)


`message` missing.



tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java
Lines 152 (patched)


`message` missing.



tools/src/test/java/org/apache/oozie/tools/diag/IntegrationTestOozieSharelibCLI.java
Lines 84 (patched)


`message` missing.



tools/src/test/java/org/apache/oozie/tools/diag/IntegrationTestOozieSharelibCLI.java
Lines 104 (patched)


`message` missing.



tools/src/test/java/org/apache/oozie/tools/diag/IntegrationTestOozieSharelibCLI.java
Lines 130 (patched)


`message` missing.



tools/src/test/java/org/apache/oozie/tools/diag/IntegrationTestOozieSharelibCLI.java
Lines 147 (patched)


Nice `message` :)


- András Piros


On July 6, 2018, 8:36 a.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67834/
> ---
> 
> (Updated July 6, 2018, 8:36 a.m.)
> 
> 
> Review request for oozie, András Piros, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Right now sharelib can be created via sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY where the SHARED_LIBRARY can be a tarbal or a folder.
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/AG_Install.twiki 46363a3b3 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 75e932c02 
>   tools/src/test/java/org/apache/oozie/tools/OozieSharelibFileOperations.java 
> d344300ed 
>   tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
> d77eba69d 
>   tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
> bce0433c6 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 
> 5929e5ca0 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/IntegrationTestOozieSharelibCLI.java
>  PRE-CREATION 
> 
> 
> Diff:

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-07-06 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534816#comment-16534816
 ] 

Hadoop QA commented on OOZIE-2829:
--

PreCommit-OOZIE-Build started


> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2829-006.patch, OOZIE-2829-007.patch, 
> OOZIE-2829-008.patch, OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-07-06 Thread Julia Kinga Marton (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-2829:
--
Attachment: (was: OOZIE-2829-008.patch)

> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2829-006.patch, OOZIE-2829-007.patch, 
> OOZIE-2829-008.patch, OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-07-06 Thread Julia Kinga Marton (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-2829:
--
Attachment: OOZIE-2829-008.patch

> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2829-006.patch, OOZIE-2829-007.patch, 
> OOZIE-2829-008.patch, OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-07-06 Thread Julia Kinga Marton (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-2829:
--
Attachment: OOZIE-2829-008.patch

> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2829-006.patch, OOZIE-2829-007.patch, 
> OOZIE-2829-008.patch, OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-07-06 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534665#comment-16534665
 ] 

Hadoop QA commented on OOZIE-2829:
--


Testing JIRA OOZIE-2829

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 4 line(s) longer than 132 
characters
.{color:green}+1{color} the patch adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [2] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:red}-1{color} There are [2] new bugs found below threshold in [tools] 
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
. The most important FindBugs errors are:
. At OozieSharelibCLI.java:[line 273]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
. At OozieSharelibCLI.java:[line 169]: At OozieSharelibCLI.java:[line 168]
. At OozieSharelibCLI.java:[line 270]: At OozieSharelibCLI.java:[line 216]
. java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input: At OozieSharelibCLI.java:[line 272]
. At OozieSharelibCLI.java:[line 172]: At OozieSharelibCLI.java:[line 180]
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2910
.Tests failed: 0
.Tests errors: 2

.The patch failed the following testcases:



.Tests failing with errors:
testFakeCommand(org.apache.oozie.tools.TestOozieSharelibCLI)
testHelpCommand(org.apache.oozie.tools.TestOozieSharelibCLI)

.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild3
TestActionCheckXCommand#testActionCheck
TestActionStartXCommand#testActionWithEscapedStringAndCDATA
TestJMSJobEventListener#testConnectionDrop
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/658/



> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>

Failed: OOZIE-2829 PreCommit Build #658

2018-07-06 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2829
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/658/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.77 MB...]


Testing JIRA OOZIE-2829

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
-1 the patch contains 4 line(s) longer than 132 characters
+1 the patch adds/modifies 5 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
-1 There are [2] new bugs found below threshold in total that must be fixed.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [webapp].
 -1 There are [2] new bugs found below threshold in [tools] that must be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
 The most important FindBugs errors are:
 At OozieSharelibCLI.java:[line 273]: 
java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input
 At OozieSharelibCLI.java:[line 169]: At OozieSharelibCLI.java:[line 168]
 At OozieSharelibCLI.java:[line 270]: At OozieSharelibCLI.java:[line 216]
 java/io/File.init(Ljava/lang/String;)V reads a file whose location 
might be specified by user input: At OozieSharelibCLI.java:[line 272]
 At OozieSharelibCLI.java:[line 172]: At OozieSharelibCLI.java:[line 180]
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2910
Tests failed: 0
Tests errors: 2

The patch failed the following testcases:



Tests failing with errors:
testFakeCommand(org.apache.oozie.tools.TestOozieSharelibCLI)
testHelpCommand(org.apache.oozie.tools.TestOozieSharelibCLI)

Tests failed at first run:
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild3
TestActionCheckXCommand#testActionCheck
TestActionStartXCommand#testActionWithEscapedStringAndCDATA
TestJMSJobEventListener#testConnectionDrop
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/658/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13056997/comment/16534665","id":"16534665","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-2829\n\nCleaning local git 
workspace\n\n\n\n{color:green}+1 
PATCH_APPLIES{color}\n{color:green}+1 CLEAN{color}\n{color:red}-1 
RAW_PATCH_ANALYSIS{color}\n.{color:green}+1{color} the patch does not 
introduce any @author tags\n.{color:green}+1{color} the patch does not 
introduce any tabs\n.{color:green}+1{color} the patch does

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-07-06 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534576#comment-16534576
 ] 

Hadoop QA commented on OOZIE-2829:
--

PreCommit-OOZIE-Build started


> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2829-006.patch, OOZIE-2829-007.patch, 
> OOZIE-2829-01.patch, OOZIE-2829-02.patch, OOZIE-2829-03.patch, 
> OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67834: OOZIE-2829 Improve sharelib upload to accept multiple source folders

2018-07-06 Thread Kinga Marton via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67834/
---

(Updated July 6, 2018, 8:36 a.m.)


Review request for oozie, András Piros, Peter Cseh, and Peter Bacsko.


Repository: oozie-git


Description
---

Right now sharelib can be created via sharelib create -fs FS_URI -locallib 
SHARED_LIBRARY where the SHARED_LIBRARY can be a tarbal or a folder.
It would be nice to have the possibility to define additional folders to be 
uploaded into the sharelib, so the users don't have to copy or link the files 
together on their machine.
The syntax could be something like -additional-lib 
sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder


Diffs (updated)
-

  docs/src/site/twiki/AG_Install.twiki 46363a3b3 
  tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 75e932c02 
  tools/src/test/java/org/apache/oozie/tools/OozieSharelibFileOperations.java 
d344300ed 
  tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
d77eba69d 
  tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
bce0433c6 
  tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 
5929e5ca0 
  
tools/src/test/java/org/apache/oozie/tools/diag/IntegrationTestOozieSharelibCLI.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/67834/diff/2/

Changes: https://reviews.apache.org/r/67834/diff/1-2/


Testing
---

Tested manually + added integration test


Thanks,

Kinga Marton

[jira] [Updated] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-07-06 Thread Julia Kinga Marton (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-2829:
--
Attachment: OOZIE-2829-007.patch

> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2829-006.patch, OOZIE-2829-007.patch, 
> OOZIE-2829-01.patch, OOZIE-2829-02.patch, OOZIE-2829-03.patch, 
> OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] Subscription: Oozie Patch Available

2018-07-06 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (91 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3301  Update NOTICE file
https://issues.apache.org/jira/browse/OOZIE-3301
OOZIE-3274  Remove slf4j
https://issues.apache.org/jira/browse/OOZIE-3274
OOZIE-3265  properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to 
appear together
https://issues.apache.org/jira/browse/OOZIE-3265
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3208  "It should never happen" error messages should be more specific to 
root cause
https://issues.apache.org/jira/browse/OOZIE-3208
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3194  Oozie should set proper permissions to sharelib after upload
https://issues.apache.org/jira/browse/OOZIE-3194
OOZIE-3193  Applications are not killed when submitted via subworkflow
https://issues.apache.org/jira/browse/OOZIE-3193
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  [examples] Fix Findbugs warnings
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2714  Detect conflicting resources during class loading
https://issues.apache.org/jira/browse/OOZIE-2714
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is