[jira] [Commented] (OOZIE-3270) Upgrade Derby to 10.14.1.0

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515944#comment-16515944
 ] 

Hadoop QA commented on OOZIE-3270:
--


Testing JIRA OOZIE-3270

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 6 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:red}-1{color} the patch seems to introduce 2 new javac warning(s)
{color:red}-1{color} There are [1] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:red}-1{color} There are [1] new bugs found below threshold in 
[sharelib/oozie] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/oozie/findbugs-new.html
. The most important FindBugs errors are:
. At AllowAllPolicy.java:[lines 57-74]: Should 
org.apache.oozie.action.hadoop.security.AllowAllPolicy$AllPermissionCollection 
be a _static_ inner class?
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2148
.{color:orange}Tests failed at first run:{color}
TestCoordActionsKillXCommand#testActionKillCommandDate
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/631/



>  Upgrade Derby to 10.14.1.0
> ---
>
> Key: OOZIE-3270
> URL: https://issues.apache.org/jira/browse/OOZIE-3270
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
>Priority: Major
> Attachments: OOZIE-3270.01.patch, OOZIE-3270.02.patch, 
> OOZIE-3270.03.patch
>
>
> We should upgrade  Derby to 10.14.1.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-3270 PreCommit Build #631

2018-06-18 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-3270
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/631/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.74 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[ERROR] There are [1] new bugs found below threshold in total that must be 
fixed.
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2727 bytes
[TRACE] Full summary file size is 1684 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3270

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 6 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
-1 COMPILE
+1 HEAD compiles
+1 patch compiles
-1 the patch seems to introduce 2 new javac warning(s)
-1 There are [1] new bugs found below threshold in total that must be fixed.
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/sqoop].
 -1 There are [1] new bugs found below threshold in [sharelib/oozie] that must 
be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
sharelib/oozie/findbugs-new.html
 The most important FindBugs errors are:
 At AllowAllPolicy.java:[lines 57-74]: Should 
org.apache.oozie.action.hadoop.security.AllowAllPolicy$AllPermissionCollection 
be a _static_ inner class?
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2148
Tests failed at first run:
TestCoordActionsKillXCommand#testActionKillCommandDate
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/631/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  35660 0  100  3566  0   4262 --:--:-- --:--:-- --:--:--

[jira] [Commented] (OOZIE-3275) [docs] Update AG_Install.twiki with Access Control List documentation

2018-06-18 Thread Peter Orova (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515905#comment-16515905
 ] 

Peter Orova commented on OOZIE-3275:


+1 lgtm

> [docs] Update AG_Install.twiki with Access Control List documentation
> -
>
> Key: OOZIE-3275
> URL: https://issues.apache.org/jira/browse/OOZIE-3275
> Project: Oozie
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
> Attachments: OOZIE-3275.001.patch, OOZIE-3275.002.patch
>
>
> It's hard to find a detailed description about ACL based authorization setup. 
> Let's extend {{AG_Install.twiki}} and have some.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515789#comment-16515789
 ] 

Hadoop QA commented on OOZIE-2339:
--


Testing JIRA OOZIE-2339

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 97 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [2] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [2] new bugs found in 
[fluent-job/fluent-job-api] that would be nice to have fixed.
. You can find the FindBugs diff here: 
fluent-job/fluent-job-api/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2896
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/629/



> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch, 
> OOZIE-2339.020.patch, OOZIE-2339.021.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
>

Success: OOZIE-2339 PreCommit Build #629

2018-06-18 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2339
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/629/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.84 MB...]
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hcatalog].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[WARN] There are [2] new bugs found in total that would be nice to have fixed.
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2602 bytes
[TRACE] Full summary file size is 1550 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-2339

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 97 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
0 There are [2] new bugs found in total that would be nice to have fixed.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 0 There are [2] new bugs found in [fluent-job/fluent-job-api] that would be 
nice to have fixed.
 You can find the FindBugs diff here: 
fluent-job/fluent-job-api/findbugs-new.html
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2896
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/629/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:--

[jira] [Commented] (OOZIE-3270) Upgrade Derby to 10.14.1.0

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515775#comment-16515775
 ] 

Hadoop QA commented on OOZIE-3270:
--

PreCommit-OOZIE-Build started


>  Upgrade Derby to 10.14.1.0
> ---
>
> Key: OOZIE-3270
> URL: https://issues.apache.org/jira/browse/OOZIE-3270
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
>Priority: Major
> Attachments: OOZIE-3270.01.patch, OOZIE-3270.02.patch, 
> OOZIE-3270.03.patch
>
>
> We should upgrade  Derby to 10.14.1.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-3270) Upgrade Derby to 10.14.1.0

2018-06-18 Thread Peter Cseh (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515717#comment-16515717
 ] 

Peter Cseh edited comment on OOZIE-3270 at 6/18/18 1:45 PM:


The last patch seems to be working on real clusters as well.
I also did some housekeeping and consolidated the SecurityManager usage in our 
tests.


was (Author: gezapeti):
The last patch seems to be working on real clusters as well.

>  Upgrade Derby to 10.14.1.0
> ---
>
> Key: OOZIE-3270
> URL: https://issues.apache.org/jira/browse/OOZIE-3270
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
>Priority: Major
> Attachments: OOZIE-3270.01.patch, OOZIE-3270.02.patch, 
> OOZIE-3270.03.patch
>
>
> We should upgrade  Derby to 10.14.1.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3270) Upgrade Derby to 10.14.1.0

2018-06-18 Thread Peter Cseh (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515717#comment-16515717
 ] 

Peter Cseh commented on OOZIE-3270:
---

The last patch seems to be working on real clusters as well.

>  Upgrade Derby to 10.14.1.0
> ---
>
> Key: OOZIE-3270
> URL: https://issues.apache.org/jira/browse/OOZIE-3270
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
>Priority: Major
> Attachments: OOZIE-3270.01.patch, OOZIE-3270.02.patch, 
> OOZIE-3270.03.patch
>
>
> We should upgrade  Derby to 10.14.1.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3270) Upgrade Derby to 10.14.1.0

2018-06-18 Thread Peter Cseh (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-3270:
--
Attachment: OOZIE-3270.03.patch

>  Upgrade Derby to 10.14.1.0
> ---
>
> Key: OOZIE-3270
> URL: https://issues.apache.org/jira/browse/OOZIE-3270
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Peter Cseh
>Assignee: Peter Cseh
>Priority: Major
> Attachments: OOZIE-3270.01.patch, OOZIE-3270.02.patch, 
> OOZIE-3270.03.patch
>
>
> We should upgrade  Derby to 10.14.1.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515676#comment-16515676
 ] 

Hadoop QA commented on OOZIE-2829:
--


Testing JIRA OOZIE-2829

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch




> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-2829 PreCommit Build #630

2018-06-18 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2829
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/630/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 786.11 KB...]
// test files in new folder

for (int i = 0; i < testFiles; i++) {
String fileName = generateFileName(i);
String expectedFileContent = generateFileContent(i);
InputStream in = null;
try {
in = fs.open(new Path(latestLibPath, fileName));
String actualFileContent = IOUtils.toString(in);
assertEquals(fileName, expectedFileContent, actualFileContent);
} finally {
IOUtils.closeQuietly(in);
}
}

}

/**
 * test fake command
 */
public void testFakeCommand() throws Exception {

ByteArrayOutputStream data = new ByteArrayOutputStream();
PrintStream oldPrintStream = System.err;
System.setErr(new PrintStream(data));
try {
String[] argsFake = { "fakeCommand" };
assertEquals(1, execOozieSharelibCLICommands(argsFake));
assertTrue(data.toString().contains("Invalid sub-command: invalid 
sub-command [fakeCommand]"));
assertTrue(data.toString().contains("use 'help [sub-command]' for 
help details"));
}
finally {
System.setErr(oldPrintStream);
}

}

private FileSystem getTargetFileSysyem() throws Exception {
if (fs == null) {
HadoopAccessorService has = 
getServices().get(HadoopAccessorService.class);
URI uri = new Path(outPath).toUri();
Configuration fsConf = has.createJobConf(uri.getAuthority());
fs = has.createFileSystem(System.getProperty("user.name"), uri, 
fsConf);
}
return fs;

}

private Services getServices() throws ServiceException {
if (services == null) {
services = new Services();
services.getConf()

.set(Services.CONF_SERVICE_CLASSES,"org.apache.oozie.service.LiteWorkflowAppService,"
+ "org.apache.oozie.service.SchedulerService,"
+ "org.apache.oozie.service.HadoopAccessorService,"
+ "org.apache.oozie.service.ShareLibService");
services.init();
}
return services;
}

private Path getDistPath() throws Exception {
if (dstPath == null) {
WorkflowAppService lwas = 
getServices().get(WorkflowAppService.class);
dstPath = lwas.getSystemLibPath();
}
return dstPath;
}

private void writeFile(File folder, String filename, String content) throws 
Exception {
File file = new File(folder.getAbsolutePath() + File.separator + 
filename);
Writer writer = new FileWriter(file);
writer.write(content);
writer.flush();
writer.close();

}

private int execOozieSharelibCLICommands(String[] args) throws Exception {

error: patch failed: 
tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java:19
error: tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java: 
patch does not apply
Patch failed to apply to head of branch

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  17740  1575  100   199   1828231 --:--:-- --:--:-- --:--:--  1829
{"self":"https://issues.apache.org/jira/rest/api/2/issue/13056997/comment/16515676","id":"16515676","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-2829\n\nCleaning local git 
workspace\n\n\n\n{color:red}-1{color} Patch failed 
to apply to head of

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515671#comment-16515671
 ] 

Hadoop QA commented on OOZIE-2829:
--

PreCommit-OOZIE-Build started


> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515670#comment-16515670
 ] 

Hadoop QA commented on OOZIE-2339:
--

PreCommit-OOZIE-Build started


> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch, 
> OOZIE-2339.020.patch, OOZIE-2339.021.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are 
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also 
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

2018-06-18 Thread András Piros via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/#review204922
---


Ship it!




Ship It!

- András Piros


On June 18, 2018, 10:03 a.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66930/
> ---
> 
> (Updated June 18, 2018, 10:03 a.m.)
> 
> 
> Review request for oozie, András Piros and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> On a busy Hadoop cluster it can happen that users cannot install properly 
> Oozie ShareLib.
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
>   tools/src/test/java/org/apache/oozie/tools/OozieSharelibFileOperations.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 
> ccad273b 
> 
> 
> Diff: https://reviews.apache.org/r/66930/diff/6/
> 
> 
> Testing
> ---
> 
> Tested manually
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515661#comment-16515661
 ] 

Hadoop QA commented on OOZIE-2829:
--


Testing JIRA OOZIE-2829

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch




> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Failed: OOZIE-2829 PreCommit Build #628

2018-06-18 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2829
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/628/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 777.49 KB...]
// test files in new folder

for (int i = 0; i < testFiles; i++) {
String fileName = generateFileName(i);
String expectedFileContent = generateFileContent(i);
InputStream in = null;
try {
in = fs.open(new Path(latestLibPath, fileName));
String actualFileContent = IOUtils.toString(in);
assertEquals(fileName, expectedFileContent, actualFileContent);
} finally {
IOUtils.closeQuietly(in);
}
}

}

/**
 * test fake command
 */
public void testFakeCommand() throws Exception {

ByteArrayOutputStream data = new ByteArrayOutputStream();
PrintStream oldPrintStream = System.err;
System.setErr(new PrintStream(data));
try {
String[] argsFake = { "fakeCommand" };
assertEquals(1, execOozieSharelibCLICommands(argsFake));
assertTrue(data.toString().contains("Invalid sub-command: invalid 
sub-command [fakeCommand]"));
assertTrue(data.toString().contains("use 'help [sub-command]' for 
help details"));
}
finally {
System.setErr(oldPrintStream);
}

}

private FileSystem getTargetFileSysyem() throws Exception {
if (fs == null) {
HadoopAccessorService has = 
getServices().get(HadoopAccessorService.class);
URI uri = new Path(outPath).toUri();
Configuration fsConf = has.createJobConf(uri.getAuthority());
fs = has.createFileSystem(System.getProperty("user.name"), uri, 
fsConf);
}
return fs;

}

private Services getServices() throws ServiceException {
if (services == null) {
services = new Services();
services.getConf()

.set(Services.CONF_SERVICE_CLASSES,"org.apache.oozie.service.LiteWorkflowAppService,"
+ "org.apache.oozie.service.SchedulerService,"
+ "org.apache.oozie.service.HadoopAccessorService,"
+ "org.apache.oozie.service.ShareLibService");
services.init();
}
return services;
}

private Path getDistPath() throws Exception {
if (dstPath == null) {
WorkflowAppService lwas = 
getServices().get(WorkflowAppService.class);
dstPath = lwas.getSystemLibPath();
}
return dstPath;
}

private void writeFile(File folder, String filename, String content) throws 
Exception {
File file = new File(folder.getAbsolutePath() + File.separator + 
filename);
Writer writer = new FileWriter(file);
writer.write(content);
writer.flush();
writer.close();

}

private int execOozieSharelibCLICommands(String[] args) throws Exception {

error: patch failed: 
tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java:19
error: tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java: 
patch does not apply
Patch failed to apply to head of branch

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
 17740  1575  100   199   1851233 --:--:-- --:--:-- --:--:--  1850
{"self":"https://issues.apache.org/jira/rest/api/2/issue/13056997/comment/16515661","id":"16515661","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 
OOZIE-2829\n\nCleaning local git 
workspace\n\n\n\n{color:red}-1{color} Patch failed 
to apply to head of

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515658#comment-16515658
 ] 

Hadoop QA commented on OOZIE-2829:
--

PreCommit-OOZIE-Build started


> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2018-06-18 Thread Andras Piros (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515657#comment-16515657
 ] 

Andras Piros commented on OOZIE-2829:
-

Let us see whether the latest patch can still be applied to current {{master}}.

> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Major
> Attachments: OOZIE-2829-01.patch, OOZIE-2829-02.patch, 
> OOZIE-2829-03.patch, OOZIE-2829-04.patch, OOZIE-2829-05.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 64029: OOZIE-2339 Minimum Viable Fluent Job API

2018-06-18 Thread András Piros via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/
---

(Updated June 18, 2018, 12:34 p.m.)


Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.


Changes
---

Addressing FindBugs warnings, removing star imports.


Repository: oozie-git


Description
---

OOZIE-2339 Provide an API for writing jobs based on the XSD schemas


Diffs (updated)
-

  client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
08e2b91e6d9209278f9e086e23cfb4e555f7bd52 
  client/src/main/java/org/apache/oozie/client/ApiJarFactory.java PRE-CREATION 
  client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
  client/src/main/java/org/apache/oozie/client/OozieClient.java 
4c8175628ba5ee346d97ad984532662032b88acc 
  core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
  core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
38fb006980c0acb582b4f91175be1ced8a1afa79 
  core/src/main/java/org/apache/oozie/local/LocalOozie.java 
9ab646cd53223d903e387eb92aa103a7b6577d0f 
  core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
  core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
d4b08711223356c8d4f58a3391b6b87efa0a2b73 
  core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
  core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
  core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
  core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
b86fa6ace43aead115302aad3ae15368b15f1923 
  core/src/main/resources/oozie-default.xml 
8d7465c27bfe0a25b07daf556a36c6ecc2569cb6 
  core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
3395cc25d27805444ba0b6463357c4ecff977ac1 
  core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
fb481e367653187a94199069199f1d1b06d1cecf 
  core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
0a030d660f9c545b520afcce458a27ba2a367658 
  docs/src/site/twiki/DG_CommandLineTool.twiki 
9a174596dd78fb15989daf4e88936a9a2b09ef13 
  docs/src/site/twiki/DG_CustomActionExecutor.twiki 
4acbf0d40ae36c441f8510e107243603fa0e001b 
  docs/src/site/twiki/DG_Examples.twiki 
13dfa28c0eecab70321886b2806a791346c46d3b 
  docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
  examples/pom.xml 420b4f8dff8c3a0fc9ab35380a4dfffe0838bd6d 
  
examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java
 PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
PRE-CREATION 
  
examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java
 PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
PRE-CREATION 
  findbugs-filter.xml 03ee4d103f46487493fd725224cf4912ac7bc7dd 
  fluent-job/fluent-job-api/pom.xml PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributesBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Builder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBase.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBaseBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Chgrp.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChgrpBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Chmod.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChmodBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Delete.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/DistcpAction.java

[jira] [Updated] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

2018-06-18 Thread Andras Piros (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2339:

Attachment: OOZIE-2339.021.patch

> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch, 
> OOZIE-2339.020.patch, OOZIE-2339.021.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are 
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also 
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

2018-06-18 Thread Andras Piros (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515646#comment-16515646
 ] 

Andras Piros commented on OOZIE-2339:
-

Addressing FindBugs warnings, removing star imports.

> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch, 
> OOZIE-2339.020.patch, OOZIE-2339.021.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are 
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also 
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515612#comment-16515612
 ] 

Hadoop QA commented on OOZIE-2791:
--


Testing JIRA OOZIE-2791

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2158
.{color:orange}Tests failed at first run:{color}
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/627/



> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-01.patch, OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
>

Success: OOZIE-2791 PreCommit Build #627

2018-06-18 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2791
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/627/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.72 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [tools].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2365 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-2791

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 5 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [server].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2158
Tests failed at first run:
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/627/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13040921/comment/16515612","id":"16515612","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop

[jira] [Commented] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515606#comment-16515606
 ] 

Hadoop QA commented on OOZIE-2339:
--


Testing JIRA OOZIE-2339

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 97 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [2] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:red}-1{color} There are [2] new bugs found below threshold in 
[fluent-job/fluent-job-api] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
fluent-job/fluent-job-api/findbugs-new.html
. The most important FindBugs errors are:
. At NodeBuilderBaseImpl.java:[line 331]: The use of java.util.Random is 
predictable
. At WorkflowBuilder.java:[line 168]: The use of java.util.Random is predictable
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:orange}0{color} There are [2] new bugs found in [client] that would be 
nice to have fixed.
. You can find the FindBugs diff here: client/findbugs-new.html
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2896
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/626/



> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch, 
> OOZIE-2339.020.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I

Failed: OOZIE-2339 PreCommit Build #626

2018-06-18 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2339
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/626/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.84 MB...]
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[WARN] There are [2] new bugs found in [client].
[DEBUG] You can find the FindBugs diff here: client/findbugs-new.html
[ERROR] There are [2] new bugs found below threshold in total that must be 
fixed.
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2943 bytes
[TRACE] Full summary file size is 1891 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-2339

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 97 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
-1 There are [2] new bugs found below threshold in total that must be fixed.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 -1 There are [2] new bugs found below threshold in [fluent-job/fluent-job-api] 
that must be fixed.
 You can find the FindBugs diff here (look for the red and orange ones): 
fluent-job/fluent-job-api/findbugs-new.html
 The most important FindBugs errors are:
 At NodeBuilderBaseImpl.java:[line 331]: The use of java.util.Random is 
predictable
 At WorkflowBuilder.java:[line 168]: The use of java.util.Random is predictable
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/spark].
 0 There are [2] new bugs found in [client] that would be nice to have fixed.
 You can find the FindBugs diff here: client/findbugs-new.html
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2896
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/626/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:--

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

2018-06-18 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515557#comment-16515557
 ] 

Hadoop QA commented on OOZIE-2791:
--

PreCommit-OOZIE-Build started


> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-01.patch, OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

2018-06-18 Thread Julia Kinga Marton (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-2791:
--
Attachment: OOZIE-2791-008.patch

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-01.patch, OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

2018-06-18 Thread Kinga Marton via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/
---

(Updated June 18, 2018, 10:03 a.m.)


Review request for oozie, András Piros and Peter Cseh.


Repository: oozie-git


Description
---

On a busy Hadoop cluster it can happen that users cannot install properly 
Oozie ShareLib.


Diffs (updated)
-

  tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
  tools/src/test/java/org/apache/oozie/tools/OozieSharelibFileOperations.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java ccad273b 


Diff: https://reviews.apache.org/r/66930/diff/6/

Changes: https://reviews.apache.org/r/66930/diff/5-6/


Testing
---

Tested manually


Thanks,

Kinga Marton

Re: Review Request 64029: OOZIE-2339 Minimum Viable Fluent Job API

2018-06-18 Thread András Piros via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/
---

(Updated June 18, 2018, 9:01 a.m.)


Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.


Summary (updated)
-

OOZIE-2339 Minimum Viable Fluent Job API


Repository: oozie-git


Description
---

OOZIE-2339 Provide an API for writing jobs based on the XSD schemas


Diffs
-

  client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
08e2b91e6d9209278f9e086e23cfb4e555f7bd52 
  client/src/main/java/org/apache/oozie/client/ApiJarFactory.java PRE-CREATION 
  client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
  client/src/main/java/org/apache/oozie/client/OozieClient.java 
4c8175628ba5ee346d97ad984532662032b88acc 
  core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
  core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
38fb006980c0acb582b4f91175be1ced8a1afa79 
  core/src/main/java/org/apache/oozie/local/LocalOozie.java 
9ab646cd53223d903e387eb92aa103a7b6577d0f 
  core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
  core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
d4b08711223356c8d4f58a3391b6b87efa0a2b73 
  core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
  core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
  core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
  core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
b86fa6ace43aead115302aad3ae15368b15f1923 
  core/src/main/resources/oozie-default.xml 
8d7465c27bfe0a25b07daf556a36c6ecc2569cb6 
  core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
3395cc25d27805444ba0b6463357c4ecff977ac1 
  core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
fb481e367653187a94199069199f1d1b06d1cecf 
  core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
0a030d660f9c545b520afcce458a27ba2a367658 
  docs/src/site/twiki/DG_CommandLineTool.twiki 
9a174596dd78fb15989daf4e88936a9a2b09ef13 
  docs/src/site/twiki/DG_CustomActionExecutor.twiki 
4acbf0d40ae36c441f8510e107243603fa0e001b 
  docs/src/site/twiki/DG_Examples.twiki 
13dfa28c0eecab70321886b2806a791346c46d3b 
  docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
  examples/pom.xml 420b4f8dff8c3a0fc9ab35380a4dfffe0838bd6d 
  
examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java
 PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
PRE-CREATION 
  
examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java
 PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
PRE-CREATION 
  findbugs-filter.xml 03ee4d103f46487493fd725224cf4912ac7bc7dd 
  fluent-job/fluent-job-api/pom.xml PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributesBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Builder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBase.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBaseBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Chgrp.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChgrpBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Chmod.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChmodBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Delete.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/DistcpAction.java

[jira] [Comment Edited] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

2018-06-18 Thread Andras Piros (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515506#comment-16515506
 ] 

Andras Piros edited comment on OOZIE-2339 at 6/18/18 9:00 AM:
--

[~pbacsko] [~gezapeti] last patch 020 addressing review comments, and 
introduces {{CredentialBuilder}}.

[~rkanter] your remaining comments have also been fixed by that patch.

Thanks all for your review efforts.


was (Author: andras.piros):
[~pbacsko] [~gezapeti] last patch 020 addressing review comments, and 
introduces {{CredentialBuilder}}.

> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch, 
> OOZIE-2339.020.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are 
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also 
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

2018-06-18 Thread Andras Piros (JIRA) 


 [ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2339:

Attachment: OOZIE-2339.020.patch

> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch, 
> OOZIE-2339.020.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are 
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also 
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

2018-06-18 Thread Andras Piros (JIRA) 


[ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515506#comment-16515506
 ] 

Andras Piros commented on OOZIE-2339:
-

[~pbacsko] [~gezapeti] last patch 020 addressing review comments, and 
introduces {{CredentialBuilder}}.

> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch, 
> OOZIE-2339.020.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are 
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also 
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

2018-06-18 Thread András Piros via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/
---

(Updated June 18, 2018, 8:58 a.m.)


Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.


Changes
---

Addressing review comments, introducing `CredentialBuilder`.


Repository: oozie-git


Description
---

OOZIE-2339 Provide an API for writing jobs based on the XSD schemas


Diffs (updated)
-

  client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 
  client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
08e2b91e6d9209278f9e086e23cfb4e555f7bd52 
  client/src/main/java/org/apache/oozie/client/ApiJarFactory.java PRE-CREATION 
  client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
  client/src/main/java/org/apache/oozie/client/OozieClient.java 
4c8175628ba5ee346d97ad984532662032b88acc 
  core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
  core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
38fb006980c0acb582b4f91175be1ced8a1afa79 
  core/src/main/java/org/apache/oozie/local/LocalOozie.java 
9ab646cd53223d903e387eb92aa103a7b6577d0f 
  core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
  core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
d4b08711223356c8d4f58a3391b6b87efa0a2b73 
  core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
  core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
  core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
  core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
b86fa6ace43aead115302aad3ae15368b15f1923 
  core/src/main/resources/oozie-default.xml 
8d7465c27bfe0a25b07daf556a36c6ecc2569cb6 
  core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
3395cc25d27805444ba0b6463357c4ecff977ac1 
  core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
fb481e367653187a94199069199f1d1b06d1cecf 
  core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
0a030d660f9c545b520afcce458a27ba2a367658 
  docs/src/site/twiki/DG_CommandLineTool.twiki 
9a174596dd78fb15989daf4e88936a9a2b09ef13 
  docs/src/site/twiki/DG_CustomActionExecutor.twiki 
4acbf0d40ae36c441f8510e107243603fa0e001b 
  docs/src/site/twiki/DG_Examples.twiki 
13dfa28c0eecab70321886b2806a791346c46d3b 
  docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
  examples/pom.xml 420b4f8dff8c3a0fc9ab35380a4dfffe0838bd6d 
  
examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java
 PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
PRE-CREATION 
  
examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java
 PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
PRE-CREATION 
  examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
PRE-CREATION 
  findbugs-filter.xml 03ee4d103f46487493fd725224cf4912ac7bc7dd 
  fluent-job/fluent-job-api/pom.xml PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributesBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Builder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBase.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBaseBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Chgrp.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChgrpBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Chmod.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChmodBuilder.java
 PRE-CREATION 
  
fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Delete.java
 PRE-CREATION

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

2018-06-18 Thread András Piros via Review Board 


> On June 13, 2018, 3:06 p.m., Peter Bacsko wrote:
> > fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/DistcpConfigurationConverter.java
> > Lines 32 (patched)
> > 
> >
> > Please use parameterized Map or Map

Cannot do because of this:
```
super(Map.class, CONFIGURATION.class);
```


> On June 13, 2018, 3:06 p.m., Peter Bacsko wrote:
> > fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/HiveConfigurationConverter.java
> > Lines 32 (patched)
> > 
> >
> > Please use parameterized Map or Map

Cannot do because of this:
```
super(Map.class, CONFIGURATION.class);
```


> On June 13, 2018, 3:06 p.m., Peter Bacsko wrote:
> > fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/InlineWorkflowConfigurationConverter.java
> > Lines 32 (patched)
> > 
> >
> > Please use parameterized Map or Map

Cannot do because of this:
```
super(Map.class, CONFIGURATION.class);
```


> On June 13, 2018, 3:06 p.m., Peter Bacsko wrote:
> > fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/MapToConfigurationPropertyConverter.java
> > Lines 30 (patched)
> > 
> >
> > Please use parameterized Map or Map

Cannot do because of this:
```
super(Map.class, CONFIGURATION.class);
```


> On June 13, 2018, 3:06 p.m., Peter Bacsko wrote:
> > fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/ShellConfigurationConverter.java
> > Lines 32 (patched)
> > 
> >
> > Please use parameterized Map or Map

Cannot do because of this:
```
super(Map.class, CONFIGURATION.class);
```


> On June 13, 2018, 3:06 p.m., Peter Bacsko wrote:
> > fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/SparkConfigurationConverter.java
> > Lines 32 (patched)
> > 
> >
> > Please use parameterized Map or Map

Cannot do because of this:
```
super(Map.class, CONFIGURATION.class);
```


> On June 13, 2018, 3:06 p.m., Peter Bacsko wrote:
> > fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/SqoopConfigurationConverter.java
> > Lines 32 (patched)
> > 
> >
> > Please use parameterized Map or Map

Cannot do because of this:
```
super(Map.class, CONFIGURATION.class);
```


> On June 13, 2018, 3:06 p.m., Peter Bacsko wrote:
> > fluent-job/fluent-job-api/src/test/java/org/apache/oozie/fluentjob/api/action/TestActionAttributesBuilder.java
> > Lines 33 (patched)
> > 
> >
> > Is it still relevant?

Written a new test covering this TODO, and removed the TODO itself.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/#review204692
---


On June 12, 2018, 1:42 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated June 12, 2018, 1:42 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
> 
> 
> Diffs
> -
> 
>   client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 08e2b91e6d9209278f9e086e23cfb4e555f7bd52 
>   client/src/main/java/org/apache/oozie/client/ApiJarFactory.java 
> PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 4c8175628ba5ee346d97ad984532662032b88acc 
>   core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
> 38fb006980c0acb582b4f91175be1ced8a1afa79 
>   core/src/main/java/org/apache/oozie/local/LocalOozie.java 
> 9ab646cd53223d903e387eb92aa103a7b6577d0f 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
> d4b08711223356c8d4f58a3391b6b87efa0a2b73 
>   core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
>

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

2018-06-18 Thread András Piros via Review Board 


> On June 14, 2018, 7:40 a.m., Peter Cseh wrote:
> > fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/workflow/Credentials.java
> > Lines 39-45 (patched)
> > 
> >
> > This should be a map instead of a List.
> > Credentials then could be referenced by their names.
> > 
> > Or better, this part could be autogenerated based on the credentials 
> > every action has. Basically we only need a CredentialBuilder to create 
> > Credential objects and add those to actions. 
> > The generator part could collect the used credentials in the workflow 
> > and just put in the  section automatically.

Introduced the new `CredentialBuilder` class, and modified 
`WorkflowBuilder#build()` so that a call to 
`WorkflowBuilder#withCredentials(Credentials)` can be omitted by default.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/#review204761
---


On June 12, 2018, 1:42 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated June 12, 2018, 1:42 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
> 
> 
> Diffs
> -
> 
>   client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 08e2b91e6d9209278f9e086e23cfb4e555f7bd52 
>   client/src/main/java/org/apache/oozie/client/ApiJarFactory.java 
> PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 4c8175628ba5ee346d97ad984532662032b88acc 
>   core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
> 38fb006980c0acb582b4f91175be1ced8a1afa79 
>   core/src/main/java/org/apache/oozie/local/LocalOozie.java 
> 9ab646cd53223d903e387eb92aa103a7b6577d0f 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
> d4b08711223356c8d4f58a3391b6b87efa0a2b73 
>   core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
> 3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
>   core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
> 1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
>   core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
> c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
>   core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
> b86fa6ace43aead115302aad3ae15368b15f1923 
>   core/src/main/resources/oozie-default.xml 
> 8d7465c27bfe0a25b07daf556a36c6ecc2569cb6 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
> 3395cc25d27805444ba0b6463357c4ecff977ac1 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
> fb481e367653187a94199069199f1d1b06d1cecf 
>   core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
> 0a030d660f9c545b520afcce458a27ba2a367658 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> 9a174596dd78fb15989daf4e88936a9a2b09ef13 
>   docs/src/site/twiki/DG_CustomActionExecutor.twiki 
> 4acbf0d40ae36c441f8510e107243603fa0e001b 
>   docs/src/site/twiki/DG_Examples.twiki 
> 13dfa28c0eecab70321886b2806a791346c46d3b 
>   docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
>   examples/pom.xml c13febc81cf5b26b4af90885787113e247ae7cb8 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/CredentialsRetrying.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
> PRE-CREATION 
>   
> examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java
>  PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
> PRE-CREATION 
>   findbugs-filter.xml 03ee4d103f46487493fd725224cf4912ac7bc7dd 
>   fluent-job/fluent-job-api/pom.xml PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
>  PRE-CREATION 
>   
>

[jira] Subscription: Oozie Patch Available

2018-06-18 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (99 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3275  [docs] Update AG_Install.twiki with Access Control List 
documentation
https://issues.apache.org/jira/browse/OOZIE-3275
OOZIE-3274  Remove slf4j
https://issues.apache.org/jira/browse/OOZIE-3274
OOZIE-3270   Upgrade Derby to 10.14.1.0
https://issues.apache.org/jira/browse/OOZIE-3270
OOZIE-3265  properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to 
appear together
https://issues.apache.org/jira/browse/OOZIE-3265
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3194  Oozie should set proper permissions to sharelib after upload
https://issues.apache.org/jira/browse/OOZIE-3194
OOZIE-3193  Applications are not killed when submitted via subworkflow
https://issues.apache.org/jira/browse/OOZIE-3193
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3160  PriorityDelayQueue put()/take() can cause significant CPU load due 
to busy waiting
https://issues.apache.org/jira/browse/OOZIE-3160
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3109  Escape log-streaming's HTML-specific characters
https://issues.apache.org/jira/browse/OOZIE-3109
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  [examples] Fix Findbugs warnings
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791