Re: Review Request 52227: OOZIE-2225 Add wild card filter for gathering jobs

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52227/#review152448
---




core/src/main/java/org/apache/oozie/executor/jpa/WorkflowsJobGetJPAExecutor.java
 (line 279)


Can you make this to use 'else if' instead of 'if' conditions. You might 
need to change couple of 'if' conditions to 'else if' in this method.



core/src/main/java/org/apache/oozie/store/StoreStatusFilter.java (line 391)


Contains more than 132 character. Please fix.



core/src/main/java/org/apache/oozie/store/StoreStatusFilter.java (line 415)


Can you remove this extra empty line.



core/src/test/java/org/apache/oozie/executor/jpa/TestCoordJobInfoGetJPAExecutor.java
 (line 28)


Import spcecific classes only instead of complete package.


- Abhishek Bafna


On Oct. 1, 2016, 12:29 a.m., Sai Chirravuri wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52227/
> ---
> 
> (Updated Oct. 1, 2016, 12:29 a.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Bugs: OOZIE-2225
> https://issues.apache.org/jira/browse/OOZIE-2225
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> commit 05c108f61f4c3da437d826b73251e92662a4518b
> Author: krish 
> Date:   Fri Sep 23 13:50:24 2016 -0700
> 
> OOZIE-2225 Add wild card filter for gathering jobs
> 
> :100644 100644 f1d0f2b... f10de36... M  
> client/src/main/java/org/apache/oozie/cli/OozieCLI.java
> :100644 100644 a882cab... 53a6ecf... M  
> client/src/main/java/org/apache/oozie/client/OozieClient.java
> :100644 100644 01aeddf... 7597142... M  
> core/src/main/java/org/apache/oozie/DagEngine.java
> :100644 100644 3c03267... 275aabf... M  
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowsJobGetJPAExecutor.java
> :100644 100644 fb1db29... 80f0756... M  
> core/src/main/java/org/apache/oozie/store/StoreStatusFilter.java
> :00 100644 000... dd1e943... A  
> core/src/main/java/org/apache/oozie/util/db/JPAUtils.java
> :100644 100644 0b1f644... 87fccd6... M  
> core/src/test/java/org/apache/oozie/executor/jpa/TestBundleJobInfoGetJPAExecutor.java
> :100644 100644 7d468f1... 4378286... M  
> core/src/test/java/org/apache/oozie/executor/jpa/TestCoordJobInfoGetJPAExecutor.java
> :100644 100644 ddb5506... 12cb16b... M  
> core/src/test/java/org/apache/oozie/executor/jpa/TestWorkflowsJobGetJPAExecutor.java
> :100644 100644 ea778bd... 2fed1f8... M  
> core/src/test/java/org/apache/oozie/test/XDataTestCase.java
> :100644 100644 567e260... 3daf38a... M  
> docs/src/site/twiki/DG_CommandLineTool.twiki
> :100644 100644 13e1691... a66e564... M  
> docs/src/site/twiki/WebServicesAPI.twiki
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> f1d0f2b7442e04e01bea93622cf378fc92934497 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> a882cab95c4b75c3fc641fb8e5511357d482eeff 
>   core/src/main/java/org/apache/oozie/CoordinatorEngine.java 
> cec6347905233d9a5fe0eb0e2fa27336a6a967a9 
>   core/src/main/java/org/apache/oozie/DagEngine.java 
> 01aeddfc282261d9ad396565a9d3d30efec2dc59 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/WorkflowsJobGetJPAExecutor.java
>  3c032678d06641ae7bb2f2a9c0631948190ea662 
>   core/src/main/java/org/apache/oozie/store/StoreStatusFilter.java 
> fb1db29cdfe1dbe37c8e2176079f39f841232ee3 
>   core/src/main/java/org/apache/oozie/util/JobsFilterUtils.java 
> 18df836713918a2864566b56646f711dd84a5f97 
>   
> core/src/test/java/org/apache/oozie/executor/jpa/TestBundleJobInfoGetJPAExecutor.java
>  0b1f644c1ff74ce24b78eadf0ed7c0cbc87e93a3 
>   
> core/src/test/java/org/apache/oozie/executor/jpa/TestCoordJobInfoGetJPAExecutor.java
>  7d468f16981710d96473155df25b8de0d5ef322b 
>   
> core/src/test/java/org/apache/oozie/executor/jpa/TestWorkflowsJobGetJPAExecutor.java
>  ddb5506de90ae32d760657f8a867ab46599f1a86 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> 567e260dd3e05c7619d957520ed49882b0cf37f2 
>   docs/src/site/twiki/WebServicesAPI.twiki 
> 13e1691c3497bed94957ef959e16e788d1196c67 
> 
> Diff: https://reviews.apache.org/r/52227/diff/
> 
> 
> Testing
> ---
> 
> CLI manual testing looks good and unit tests passed
> 
> 
> Thanks,
> 
> Sai Chirravuri
> 
>

[jira] [Commented] (OOZIE-2225) Add wild card filter for gathering jobs

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570463#comment-15570463
 ] 

Hadoop QA commented on OOZIE-2225:
--

Testing JIRA OOZIE-2225

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1818
.Tests failed: 0
.Tests errors: 1

.The patch failed the following testcases:

.  

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3376/

> Add wild card filter for gathering jobs
> ---
>
> Key: OOZIE-2225
> URL: https://issues.apache.org/jira/browse/OOZIE-2225
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 4.1.0
>Reporter: Sai Chirravuri
>Assignee: Sai Chirravuri
> Attachments: OOZIE-2225-001.patch, OOZIE-2225-002.patch, 
> OOZIE-2225-003.patch, OOZIE-2225-004.patch, OOZIE-2225-005.patch, 
> OOZIE-2225-007.patch
>
>
> Oozie currently supports name, user, group, status, frequency and unit as 
> filter params.
> Add wild card param that matches partially with name/user or complete match 
> with ID



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: oozie-trunk-precommit-build #3376

[Apache Jenkins Server]

See 

--
[...truncated 13709 lines...]
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] 

[jira] [Updated] (OOZIE-2225) Add wild card filter for gathering jobs

[Sai Chirravuri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sai Chirravuri updated OOZIE-2225:
--
Attachment: OOZIE-2225-007.patch

> Add wild card filter for gathering jobs
> ---
>
> Key: OOZIE-2225
> URL: https://issues.apache.org/jira/browse/OOZIE-2225
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 4.1.0
>Reporter: Sai Chirravuri
>Assignee: Sai Chirravuri
> Attachments: OOZIE-2225-001.patch, OOZIE-2225-002.patch, 
> OOZIE-2225-003.patch, OOZIE-2225-004.patch, OOZIE-2225-005.patch, 
> OOZIE-2225-007.patch
>
>
> Oozie currently supports name, user, group, status, frequency and unit as 
> filter params.
> Add wild card param that matches partially with name/user or complete match 
> with ID



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: oozie-trunk-precommit-build #3375

[Apache Jenkins Server]

See 

--
[...truncated 5880 lines...]
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 11 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) 

[jira] [Commented] (OOZIE-1986) Add FindBugs report to pre-commit build

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569566#comment-15569566
 ] 

Hadoop QA commented on OOZIE-1986:
--

Testing JIRA OOZIE-1986

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 5 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1815
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3375/

> Add FindBugs report to pre-commit build
> ---
>
> Key: OOZIE-1986
> URL: https://issues.apache.org/jira/browse/OOZIE-1986
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-1986.001.patch, OOZIE-1986.002.patch, 
> OOZIE-1986.003.patch, OOZIE-1986.004.patch, OOZIE-1986.005.patch, 
> OOZIE-1986.006.patch, OOZIE-1986.007.patch, OOZIE-1986.008.patch, 
> OOZIE-1986.009.patch, summary.txt
>
>
> Now that we have OOZIE-1793, it will be really useful to have the pre-commit 
> build generate a findbugs report.  It should probably before and after the 
> patch and only complain it there are more after.  And it should link to the 
> report.  Ideally, this would do some kind of diff (like Hadoop does) so it's 
> easy to identify the new findbugs items.
> You can generate the findbugs reports by running:
> {noformat}
> mvn verify -DskipTests
> {noformat}
> This also runs checkstyle



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : oozie-trunk-find-patches-available #136076

[Apache Jenkins Server]

See 

Build failed in Jenkins: oozie-trunk-find-patches-available #136075

[Apache Jenkins Server]

See 

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (yahoo-not-h2 Hadoop Mapreduce Hdfs Pig) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision bd54452808335cbadc48d4aacd0bd23374bed928 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bd54452808335cbadc48d4aacd0bd23374bed928
 > git rev-list bd54452808335cbadc48d4aacd0bd23374bed928 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson2622609098068469994.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
curl: (52) Empty reply from server
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

[jira] [Commented] (OOZIE-1986) Add FindBugs report to pre-commit build

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569330#comment-15569330
 ] 

Hadoop QA commented on OOZIE-1986:
--

Testing JIRA OOZIE-1986

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:red}-1{color} the patch contains 1 line(s) with trailing spaces
.{color:red}-1{color} the patch contains 5 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1815
.Tests failed: 5
.Tests errors: 3

.The patch failed the following testcases:

.  testMain(org.apache.oozie.action.hadoop.TestHiveMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMain)
.  testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
.  testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMainWithOldAPI)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3374/

> Add FindBugs report to pre-commit build
> ---
>
> Key: OOZIE-1986
> URL: https://issues.apache.org/jira/browse/OOZIE-1986
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-1986.001.patch, OOZIE-1986.002.patch, 
> OOZIE-1986.003.patch, OOZIE-1986.004.patch, OOZIE-1986.005.patch, 
> OOZIE-1986.006.patch, OOZIE-1986.007.patch, OOZIE-1986.008.patch, 
> OOZIE-1986.009.patch, summary.txt
>
>
> Now that we have OOZIE-1793, it will be really useful to have the pre-commit 
> build generate a findbugs report.  It should probably before and after the 
> patch and only complain it there are more after.  And it should link to the 
> report.  Ideally, this would do some kind of diff (like Hadoop does) so it's 
> easy to identify the new findbugs items.
> You can generate the findbugs reports by running:
> {noformat}
> mvn verify -DskipTests
> {noformat}
> This also runs checkstyle



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: oozie-trunk-precommit-build #3374

[Apache Jenkins Server]

See 

--
[...truncated 5921 lines...]
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (default) @ oozie-distro ---
[INFO] Executing tasks

main:
  [get] Getting: 

Build failed in Jenkins: oozie-trunk-precommit-build #3372

[Apache Jenkins Server]

See 

--
[...truncated 5951 lines...]
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 20 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) 

Build failed in Jenkins: oozie-trunk-precommit-build #3373

[Apache Jenkins Server]

See 

--
[...truncated 13724 lines...]
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] 

[jira] [Commented] (OOZIE-1986) Add FindBugs report to pre-commit build

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569212#comment-15569212
 ] 

Hadoop QA commented on OOZIE-1986:
--

Testing JIRA OOZIE-1986

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 5 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1815
.Tests failed: 1
.Tests errors: 1

.The patch failed the following testcases:

.  
testBundleStatusTransitWithLock(org.apache.oozie.service.TestStatusTransitService)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3373/

> Add FindBugs report to pre-commit build
> ---
>
> Key: OOZIE-1986
> URL: https://issues.apache.org/jira/browse/OOZIE-1986
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-1986.001.patch, OOZIE-1986.002.patch, 
> OOZIE-1986.003.patch, OOZIE-1986.004.patch, OOZIE-1986.005.patch, 
> OOZIE-1986.006.patch, OOZIE-1986.007.patch, OOZIE-1986.008.patch, 
> OOZIE-1986.009.patch, summary.txt
>
>
> Now that we have OOZIE-1793, it will be really useful to have the pre-commit 
> build generate a findbugs report.  It should probably before and after the 
> patch and only complain it there are more after.  And it should link to the 
> report.  Ideally, this would do some kind of diff (like Hadoop does) so it's 
> easy to identify the new findbugs items.
> You can generate the findbugs reports by running:
> {noformat}
> mvn verify -DskipTests
> {noformat}
> This also runs checkstyle



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52429: OOZIE-1986 Add FindBugs report to pre-commit build

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52429/
---

(Updated Oct. 12, 2016, 4:08 p.m.)


Review request for oozie and Robert Kanter.


Repository: oozie-git


Description
---

We go for that solution:

1. create a new script as `bin/test-patch-11-findbugs-diff` so that it comes 
right after compile

2. `mvn clean verify -DskipTests` before and after applying the patch, and 
saving each and every module's `target/findbugs/findbugsXml.xml` to a secure 
folder

3. call the `findbugs-diff` app for all the before / after FindBugs xml pairs

4. analyze all the `findbugs-new.xml` entries for nodes with XPath 
`/BugCollection/BugInstance` entries. Stop on any BugInstance w/ `priority` and 
/ or `rank` lower than threshold

5. have the reports merged and be downloadable in any case in following 
scenarios:

* -1 for bugs that are too serious to apply the patch with them
* 0 for other new bugs
* +1 if there is no new bug


Diffs (updated)
-

  bin/test-patch-11-findbugs-diff PRE-CREATION 
  bin/test-patch-11-findbugs-diff-0.1.0-all.jar.md5 PRE-CREATION 
  pom.xml 83a1d42 

Diff: https://reviews.apache.org/r/52429/diff/


Testing
---


Thanks,

András Piros

[jira] [Updated] (OOZIE-1986) Add FindBugs report to pre-commit build

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-1986:

Attachment: OOZIE-1986.009.patch

> Add FindBugs report to pre-commit build
> ---
>
> Key: OOZIE-1986
> URL: https://issues.apache.org/jira/browse/OOZIE-1986
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-1986.001.patch, OOZIE-1986.002.patch, 
> OOZIE-1986.003.patch, OOZIE-1986.004.patch, OOZIE-1986.005.patch, 
> OOZIE-1986.006.patch, OOZIE-1986.007.patch, OOZIE-1986.008.patch, 
> OOZIE-1986.009.patch, summary.txt
>
>
> Now that we have OOZIE-1793, it will be really useful to have the pre-commit 
> build generate a findbugs report.  It should probably before and after the 
> patch and only complain it there are more after.  And it should link to the 
> report.  Ideally, this would do some kind of diff (like Hadoop does) so it's 
> easy to identify the new findbugs items.
> You can generate the findbugs reports by running:
> {noformat}
> mvn verify -DskipTests
> {noformat}
> This also runs checkstyle



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52737: OOZIE-2698 OYA: Refactor LauncherAM to make it more testable

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52737/#review152319
---




sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java
 (line 41)


Why `synchronized`? I don't see we reach out to global state, or shared 
resources.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java
 (line 59)


Why `synchronized`? I don't see we reach out to global state, or shared 
resources.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java
 (line 68)


Why `synchronized`? I don't see we reach out to global state, or shared 
resources.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ErrorHolder.java 
(line 20)


Consider using the existing `ErrorCode` and `XException` classes instead.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsAndLocalFSOperations.java
 (line 38)


I'd separate concerns to different classes.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsAndLocalFSOperations.java
 (line 43)


Better would be `"seqFileWriterFactory should not be null"`.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsAndLocalFSOperations.java
 (line 44)


Better would be `"ugi should not be null"`.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsAndLocalFSOperations.java
 (line 57)


What about using `Files#walkFileTree` instead?


https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#walkFileTree(java.nio.file.Path,%20java.util.Set,%20int,%20java.nio.file.FileVisitor)



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsAndLocalFSOperations.java
 (line 81)


What about using Guava's `Files#toString()`?


https://google.github.io/guava/releases/19.0/api/docs/com/google/common/io/Files.html#toString(java.io.File,
 java.nio.charset.Charset)



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsAndLocalFSOperations.java
 (line 102)


What about using Guava's `Files#toString()`?


https://google.github.io/guava/releases/19.0/api/docs/com/google/common/io/Files.html#toString(java.io.File,
 java.nio.charset.Charset)



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
(lines 350 - 352)


Dead code, pls. remove.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
(lines 350 - 352)


Dead code, pls. remove.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
(lines 355 - 356)


Dead code, pls. remove.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
(lines 373 - 375)


Dead code, pls. remove.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
(line 425)


Why not use `AtomicReference#compareAndSet()` instead?



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
(lines 475 - 476)


Consider Guava's `Files#toString()`:


https://google.github.io/guava/releases/19.0/api/docs/com/google/common/io/Files.html#toString(java.io.File,
 java.nio.charset.Charset)



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
(lines 512 - 514)


Consider Guava's `Files#toString()`:


https://google.github.io/guava/releases/19.0/api/docs/com/google/common/io/Files.html#toString(java.io.File,
 java.nio.charset.Charset)



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsHandler.java
 (line 67)


Why is `operation` a `String`?



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsHandler.java
 (lines 69 - 74)


`switch case default` would be nicer here.



sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsHandler.java
 (line 89)

Re: Review Request 52429: OOZIE-1986 Add FindBugs report to pre-commit build

[Attila Sasvari]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52429/#review152327
---




bin/test-patch-11-findbugs-diff (line 49)


(nit) remove whitespace


- Attila Sasvari


On Oct. 12, 2016, 3:07 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52429/
> ---
> 
> (Updated Oct. 12, 2016, 3:07 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We go for that solution:
> 
> 1. create a new script as `bin/test-patch-11-findbugs-diff` so that it comes 
> right after compile
> 
> 2. `mvn clean verify -DskipTests` before and after applying the patch, and 
> saving each and every module's `target/findbugs/findbugsXml.xml` to a secure 
> folder
> 
> 3. call the `findbugs-diff` app for all the before / after FindBugs xml pairs
> 
> 4. analyze all the `findbugs-new.xml` entries for nodes with XPath 
> `/BugCollection/BugInstance` entries. Stop on any BugInstance w/ `priority` 
> and / or `rank` lower than threshold
> 
> 5. have the reports merged and be downloadable in any case in following 
> scenarios:
> 
> * -1 for bugs that are too serious to apply the patch with them
> * 0 for other new bugs
> * +1 if there is no new bug
> 
> 
> Diffs
> -
> 
>   bin/test-patch-11-findbugs-diff PRE-CREATION 
>   bin/test-patch-11-findbugs-diff-0.1.0-all.jar.md5 PRE-CREATION 
>   pom.xml 83a1d42 
> 
> Diff: https://reviews.apache.org/r/52429/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Commented] (OOZIE-2662) DB migration fails if DB is too big

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569053#comment-15569053
 ] 

Hadoop QA commented on OOZIE-2662:
--

Testing JIRA OOZIE-2662

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 20 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 15 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1818
.Tests failed: 5
.Tests errors: 6

.The patch failed the following testcases:

.  testMain(org.apache.oozie.action.hadoop.TestHiveMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMain)
.  testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
.  testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMainWithOldAPI)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3371/

> DB migration fails if DB is too big
> ---
>
> Key: OOZIE-2662
> URL: https://issues.apache.org/jira/browse/OOZIE-2662
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2662.001.patch, OOZIE-2662.002.patch, 
> OOZIE-2662.002.wip.patch, OOZIE-2662.003.patch
>
>
> The initial version of the DB import tool commits all the workflows, actions 
> etc. in one huge commit. If it does not fits into the memory, AOOME is thrown.
> We should commit every 1k or 10k elements to prevent this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: oozie-trunk-precommit-build #3371

[Apache Jenkins Server]

See 

--
[...truncated 5996 lines...]
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (default) @ oozie-distro ---
[INFO] Executing tasks

main:
  [get] Getting: 

Jenkins build is back to normal : oozie-trunk-find-patches-available #136068

[Apache Jenkins Server]

See 

[jira] [Updated] (OOZIE-2662) DB migration fails if DB is too big

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2662:

Attachment: OOZIE-2662.003.patch

Some {{JPAService}} modification related to automatic schema generation inside 
unit tests w/ HSQLDB is not needed anymore.

> DB migration fails if DB is too big
> ---
>
> Key: OOZIE-2662
> URL: https://issues.apache.org/jira/browse/OOZIE-2662
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2662.001.patch, OOZIE-2662.002.patch, 
> OOZIE-2662.002.wip.patch, OOZIE-2662.003.patch
>
>
> The initial version of the DB import tool commits all the workflows, actions 
> etc. in one huge commit. If it does not fits into the memory, AOOME is thrown.
> We should commit every 1k or 10k elements to prevent this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/
---

(Updated Oct. 12, 2016, 3:01 p.m.)


Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and Robert 
Kanter.


Changes
---

Some `JPAService` modifications due to needed schema creation is not needed 
anymore.


Repository: oozie-git


Description
---

We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
size. Furthermore, of at least one row inside that tx fails, we retry the whole 
batch into separate `EntityTransaction`s each.

Following error handling is implemented:

1. check if all necessary tables are present and empty
2. rows are imported till the end even if there are skipped rows in the 
meanwhile
3. if at least one row is skipped in the meanwhile for some 
`ConstraintViolationException`, we delete all rows of all necessary tables. 
That enables the user to have the log messages of all the erroneous rows in one 
run, and Oozie database is never in an inconsistent state of some rows present, 
some not present of an import


Diffs (updated)
-

  tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 0e14a30 
  tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java c43223e 
  tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_bna.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_bnj.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_slareg.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_slasum.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
  tools/src/test/resources/dumpData/ooziedb_ac.json 22bbdc2 
  tools/src/test/resources/dumpData/ooziedb_bna.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_bnj.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_ca.json 2715b94 
  tools/src/test/resources/dumpData/ooziedb_cj.json 979c10e 
  tools/src/test/resources/dumpData/ooziedb_slareg.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_slasum.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_sysinfo.json 15de009 
  tools/src/test/resources/dumpData/ooziedb_wf.json 05e7e36 
  tools/src/test/resources/dumpData/valid/ooziedb_ac.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_ca.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_cj.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_sysinfo.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_wf.json PRE-CREATION 

Diff: https://reviews.apache.org/r/52782/diff/


Testing
---

See `TestDBLoadDump` for further reference.


Thanks,

András Piros

Build failed in Jenkins: oozie-trunk-find-patches-available #136067

[Apache Jenkins Server]

See 

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (yahoo-not-h2 Hadoop Mapreduce Hdfs Pig) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision bd54452808335cbadc48d4aacd0bd23374bed928 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bd54452808335cbadc48d4aacd0bd23374bed928
 > git rev-list bd54452808335cbadc48d4aacd0bd23374bed928 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson362730541329045455.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
curl: (52) Empty reply from server
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

Re: Review Request 52429: OOZIE-1986 Add FindBugs report to pre-commit build

[András Piros]

> On Oct. 12, 2016, 2 p.m., Attila Sasvari wrote:
> > bin/test-patch-11-findbugs-diff, line 179
> > 
> >
> > You can move theses two lines to cleanup_and_exit() if possible 
> > rm -f "${DIFF_DIR}/${FINDBUGS_JAR}" "${DIFF_DIR}/${FINDBUGS_JAR}.md5"
> > echo "[TRACE] FindBugs diff JAR removed"
> > (add an if that tests if jar and md5 are there)
> > 
> > call cleanup_and_exit at the end in the report switch case

Done.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52429/#review152302
---


On Oct. 12, 2016, 2:45 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52429/
> ---
> 
> (Updated Oct. 12, 2016, 2:45 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We go for that solution:
> 
> 1. create a new script as `bin/test-patch-11-findbugs-diff` so that it comes 
> right after compile
> 
> 2. `mvn clean verify -DskipTests` before and after applying the patch, and 
> saving each and every module's `target/findbugs/findbugsXml.xml` to a secure 
> folder
> 
> 3. call the `findbugs-diff` app for all the before / after FindBugs xml pairs
> 
> 4. analyze all the `findbugs-new.xml` entries for nodes with XPath 
> `/BugCollection/BugInstance` entries. Stop on any BugInstance w/ `priority` 
> and / or `rank` lower than threshold
> 
> 5. have the reports merged and be downloadable in any case in following 
> scenarios:
> 
> * -1 for bugs that are too serious to apply the patch with them
> * 0 for other new bugs
> * +1 if there is no new bug
> 
> 
> Diffs
> -
> 
>   bin/test-patch-11-findbugs-diff PRE-CREATION 
>   bin/test-patch-11-findbugs-diff-0.1.0-all.jar.md5 PRE-CREATION 
>   pom.xml 83a1d42 
> 
> Diff: https://reviews.apache.org/r/52429/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 52429: OOZIE-1986 Add FindBugs report to pre-commit build

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52429/
---

(Updated Oct. 12, 2016, 2:45 p.m.)


Review request for oozie and Robert Kanter.


Changes
---

Small refactor.


Repository: oozie-git


Description
---

We go for that solution:

1. create a new script as `bin/test-patch-11-findbugs-diff` so that it comes 
right after compile

2. `mvn clean verify -DskipTests` before and after applying the patch, and 
saving each and every module's `target/findbugs/findbugsXml.xml` to a secure 
folder

3. call the `findbugs-diff` app for all the before / after FindBugs xml pairs

4. analyze all the `findbugs-new.xml` entries for nodes with XPath 
`/BugCollection/BugInstance` entries. Stop on any BugInstance w/ `priority` and 
/ or `rank` lower than threshold

5. have the reports merged and be downloadable in any case in following 
scenarios:

* -1 for bugs that are too serious to apply the patch with them
* 0 for other new bugs
* +1 if there is no new bug


Diffs (updated)
-

  bin/test-patch-11-findbugs-diff PRE-CREATION 
  bin/test-patch-11-findbugs-diff-0.1.0-all.jar.md5 PRE-CREATION 
  pom.xml 83a1d42 

Diff: https://reviews.apache.org/r/52429/diff/


Testing
---


Thanks,

András Piros

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/#review152308
---




core/src/main/java/org/apache/oozie/service/JPAService.java (lines 183 - 186)


Can you add this into hsqldb-oozie-site.xml used for the tests rather than 
the JPAService?


- Peter Cseh


On Oct. 12, 2016, 1:51 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52782/
> ---
> 
> (Updated Oct. 12, 2016, 1:51 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
> size. Furthermore, of at least one row inside that tx fails, we retry the 
> whole batch into separate `EntityTransaction`s each.
> 
> Following error handling is implemented:
> 
> 1. check if all necessary tables are present and empty
> 2. rows are imported till the end even if there are skipped rows in the 
> meanwhile
> 3. if at least one row is skipped in the meanwhile for some 
> `ConstraintViolationException`, we delete all rows of all necessary tables. 
> That enables the user to have the log messages of all the erroneous rows in 
> one run, and Oozie database is never in an inconsistent state of some rows 
> present, some not present of an import
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 028381d 
>   tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 0e14a30 
>   tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java c43223e 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_bna.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_bnj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_slareg.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_slasum.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
>   tools/src/test/resources/dumpData/ooziedb_ac.json 22bbdc2 
>   tools/src/test/resources/dumpData/ooziedb_bna.json e69de29 
>   tools/src/test/resources/dumpData/ooziedb_bnj.json e69de29 
>   tools/src/test/resources/dumpData/ooziedb_ca.json 2715b94 
>   tools/src/test/resources/dumpData/ooziedb_cj.json 979c10e 
>   tools/src/test/resources/dumpData/ooziedb_slareg.json e69de29 
>   tools/src/test/resources/dumpData/ooziedb_slasum.json e69de29 
>   tools/src/test/resources/dumpData/ooziedb_sysinfo.json 15de009 
>   tools/src/test/resources/dumpData/ooziedb_wf.json 05e7e36 
>   tools/src/test/resources/dumpData/valid/ooziedb_ac.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_ca.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_cj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_sysinfo.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_wf.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52782/diff/
> 
> 
> Testing
> ---
> 
> See `TestDBLoadDump` for further reference.
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Commented] (OOZIE-2710) Oozie HCatalog example workflow fails

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568853#comment-15568853
 ] 

Hadoop QA commented on OOZIE-2710:
--

Testing JIRA OOZIE-2710

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1815
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/3370/

> Oozie HCatalog example workflow fails
> -
>
> Key: OOZIE-2710
> URL: https://issues.apache.org/jira/browse/OOZIE-2710
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 4.3.0
>
> Attachments: OOZIE-2710-00.patch
>
>
> Oozie HCatalog example workflow fails with the below exception:
> {noformat}
> 2016-10-12 10:33:37,260 [uber-SubtaskRunner] ERROR org.apache.pig.PigServer  
> - exception during parsing: Error during parsing. Could not resolve 
> org.apache.hive.hcatalog.pig.HCatLoader using imports: [, java.lang., 
> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> Failed to parse: Pig script failed to parse: 
>  pig script failed to validate: 
> org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not 
> resolve org.apache.hive.hcatalog.pig.HCatLoader using imports: [, java.lang., 
> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1819)
>   at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1527)
>   at org.apache.pig.PigServer.parseAndBuild(PigServer.java:460)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:485)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:471)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:172)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:235)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
>   at org.apache.pig.Main.run(Main.java:501)
>   at org.apache.pig.PigRunner.run(PigRunner.java:49)
>   at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:290)
>   at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:233)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:60)
>   at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:73)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:380)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:301)
>   at 
> 

Build failed in Jenkins: oozie-trunk-precommit-build #3370

[Apache Jenkins Server]

See 

--
[...truncated 5892 lines...]
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 11 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) 

Re: Review Request 52429: OOZIE-1986 Add FindBugs report to pre-commit build

[Attila Sasvari]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52429/#review152302
---




bin/test-patch-11-findbugs-diff (line 179)


You can move theses two lines to cleanup_and_exit() if possible 
rm -f "${DIFF_DIR}/${FINDBUGS_JAR}" "${DIFF_DIR}/${FINDBUGS_JAR}.md5"
echo "[TRACE] FindBugs diff JAR removed"
(add an if that tests if jar and md5 are there)

call cleanup_and_exit at the end in the report switch case


- Attila Sasvari


On Oct. 10, 2016, 12:28 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52429/
> ---
> 
> (Updated Oct. 10, 2016, 12:28 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We go for that solution:
> 
> 1. create a new script as `bin/test-patch-11-findbugs-diff` so that it comes 
> right after compile
> 
> 2. `mvn clean verify -DskipTests` before and after applying the patch, and 
> saving each and every module's `target/findbugs/findbugsXml.xml` to a secure 
> folder
> 
> 3. call the `findbugs-diff` app for all the before / after FindBugs xml pairs
> 
> 4. analyze all the `findbugs-new.xml` entries for nodes with XPath 
> `/BugCollection/BugInstance` entries. Stop on any BugInstance w/ `priority` 
> and / or `rank` lower than threshold
> 
> 5. have the reports merged and be downloadable in any case in following 
> scenarios:
> 
> * -1 for bugs that are too serious to apply the patch with them
> * 0 for other new bugs
> * +1 if there is no new bug
> 
> 
> Diffs
> -
> 
>   bin/test-patch-11-findbugs-diff PRE-CREATION 
>   bin/test-patch-11-findbugs-diff-0.1.0-all.jar.md5 PRE-CREATION 
>   pom.xml a33d2c5 
> 
> Diff: https://reviews.apache.org/r/52429/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 52429: OOZIE-1986 Add FindBugs report to pre-commit build

[Peter Cseh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52429/#review152303
---


Ship it!




Ship It!

- Peter Cseh


On Oct. 10, 2016, 12:28 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52429/
> ---
> 
> (Updated Oct. 10, 2016, 12:28 p.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We go for that solution:
> 
> 1. create a new script as `bin/test-patch-11-findbugs-diff` so that it comes 
> right after compile
> 
> 2. `mvn clean verify -DskipTests` before and after applying the patch, and 
> saving each and every module's `target/findbugs/findbugsXml.xml` to a secure 
> folder
> 
> 3. call the `findbugs-diff` app for all the before / after FindBugs xml pairs
> 
> 4. analyze all the `findbugs-new.xml` entries for nodes with XPath 
> `/BugCollection/BugInstance` entries. Stop on any BugInstance w/ `priority` 
> and / or `rank` lower than threshold
> 
> 5. have the reports merged and be downloadable in any case in following 
> scenarios:
> 
> * -1 for bugs that are too serious to apply the patch with them
> * 0 for other new bugs
> * +1 if there is no new bug
> 
> 
> Diffs
> -
> 
>   bin/test-patch-11-findbugs-diff PRE-CREATION 
>   bin/test-patch-11-findbugs-diff-0.1.0-all.jar.md5 PRE-CREATION 
>   pom.xml a33d2c5 
> 
> Diff: https://reviews.apache.org/r/52429/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> András Piros
> 
>

Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

[András Piros]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/
---

Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and Robert 
Kanter.


Repository: oozie-git


Description
---

We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
size. Furthermore, of at least one row inside that tx fails, we retry the whole 
batch into separate `EntityTransaction`s each.

Following error handling is implemented:

1. check if all necessary tables are present and empty
2. rows are imported till the end even if there are skipped rows in the 
meanwhile
3. if at least one row is skipped in the meanwhile for some 
`ConstraintViolationException`, we delete all rows of all necessary tables. 
That enables the user to have the log messages of all the erroneous rows in one 
run, and Oozie database is never in an inconsistent state of some rows present, 
some not present of an import


Diffs
-

  core/src/main/java/org/apache/oozie/service/JPAService.java 028381d 
  tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 0e14a30 
  tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java c43223e 
  tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_bna.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_bnj.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_slareg.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_slasum.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
  tools/src/test/resources/dumpData/ooziedb_ac.json 22bbdc2 
  tools/src/test/resources/dumpData/ooziedb_bna.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_bnj.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_ca.json 2715b94 
  tools/src/test/resources/dumpData/ooziedb_cj.json 979c10e 
  tools/src/test/resources/dumpData/ooziedb_slareg.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_slasum.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_sysinfo.json 15de009 
  tools/src/test/resources/dumpData/ooziedb_wf.json 05e7e36 
  tools/src/test/resources/dumpData/valid/ooziedb_ac.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_ca.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_cj.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_sysinfo.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_wf.json PRE-CREATION 

Diff: https://reviews.apache.org/r/52782/diff/


Testing
---

See `TestDBLoadDump` for further reference.


Thanks,

András Piros

[jira] [Comment Edited] (OOZIE-2662) DB migration fails if DB is too big

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568752#comment-15568752
 ] 

Andras Piros edited comment on OOZIE-2662 at 10/12/16 1:43 PM:
---

Following error handling is implemented:
# check if all necessary tables are present and empty
# rows are imported till the end
# if at least one row is skipped in the meanwhile for some 
{{ConstraintViolationException}}, we delete all rows of all necessary tables. 
That enables the user to have the log messages of all the erroneous rows in one 
run, and Oozie database is never in an inconsistent state of some rows present, 
some not present of an import


was (Author: andras.piros):
Following error handling is implemented:
# check if all necessary tables are present and empty
# rows are imported till the end
# if at least one row is skipped in the meanwhile for some 
{{ConstraintViolationException}}, we delete all necessary tables. That enables 
the user to have the log messages of all the erroneous rows in one run, and 
Oozie database is never in an inconsistent state of some rows present, some not 
present of an import

> DB migration fails if DB is too big
> ---
>
> Key: OOZIE-2662
> URL: https://issues.apache.org/jira/browse/OOZIE-2662
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2662.001.patch, OOZIE-2662.002.patch, 
> OOZIE-2662.002.wip.patch
>
>
> The initial version of the DB import tool commits all the workflows, actions 
> etc. in one huge commit. If it does not fits into the memory, AOOME is thrown.
> We should commit every 1k or 10k elements to prevent this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2662) DB migration fails if DB is too big

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2662:

Attachment: OOZIE-2662.002.patch

Following error handling is implemented:
# check if all necessary tables are present and empty
# rows are imported till the end
# if at least one row is skipped in the meanwhile for some 
{{ConstraintViolationException}}, we delete all necessary tables. That enables 
the user to have the log messages of all the erroneous rows in one run, and 
Oozie database is never in an inconsistent state of some rows present, some not 
present of an import

> DB migration fails if DB is too big
> ---
>
> Key: OOZIE-2662
> URL: https://issues.apache.org/jira/browse/OOZIE-2662
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2662.001.patch, OOZIE-2662.002.patch, 
> OOZIE-2662.002.wip.patch
>
>
> The initial version of the DB import tool commits all the workflows, actions 
> etc. in one huge commit. If it does not fits into the memory, AOOME is thrown.
> We should commit every 1k or 10k elements to prevent this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2710) Oozie HCatalog example workflow fails

[Abhishek Bafna (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Bafna updated OOZIE-2710:
--
Attachment: OOZIE-2710-00.patch

> Oozie HCatalog example workflow fails
> -
>
> Key: OOZIE-2710
> URL: https://issues.apache.org/jira/browse/OOZIE-2710
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 4.3.0
>
> Attachments: OOZIE-2710-00.patch
>
>
> Oozie HCatalog example workflow fails with the below exception:
> {noformat}
> 2016-10-12 10:33:37,260 [uber-SubtaskRunner] ERROR org.apache.pig.PigServer  
> - exception during parsing: Error during parsing. Could not resolve 
> org.apache.hive.hcatalog.pig.HCatLoader using imports: [, java.lang., 
> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> Failed to parse: Pig script failed to parse: 
>  pig script failed to validate: 
> org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not 
> resolve org.apache.hive.hcatalog.pig.HCatLoader using imports: [, java.lang., 
> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1819)
>   at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1527)
>   at org.apache.pig.PigServer.parseAndBuild(PigServer.java:460)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:485)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:471)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:172)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:235)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
>   at org.apache.pig.Main.run(Main.java:501)
>   at org.apache.pig.PigRunner.run(PigRunner.java:49)
>   at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:290)
>   at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:233)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:60)
>   at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:73)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:380)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:301)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:187)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:230)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: 
>  pig script failed to validate: 
> org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not 
> resolve org.apache.hive.hcatalog.pig.HCatLoader using imports: [, java.lang., 
> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1339)
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1324)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5184)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3515)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)

[jira] [Commented] (OOZIE-2710) Oozie HCatalog example workflow fails

[Abhishek Bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568570#comment-15568570
 ] 

Abhishek Bafna commented on OOZIE-2710:
---

Checked all the jars in {{pig}}, {{hive}} and {{hcatalog}}, none of them 
contains the {{HCatLoader}} class. Command used is:

{noformat}
for i in *.jar; do jar -tvf "$i" | grep -Hsi HCatLoader && echo "$i"; done
{noformat}

The class is available in the {{hive-hcatalog-pig-adapter}} module. Added the 
same into oozie pig share lib module, excluded some of the hive-shims and 
curator jars.

> Oozie HCatalog example workflow fails
> -
>
> Key: OOZIE-2710
> URL: https://issues.apache.org/jira/browse/OOZIE-2710
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Abhishek Bafna
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 4.3.0
>
>
> Oozie HCatalog example workflow fails with the below exception:
> {noformat}
> 2016-10-12 10:33:37,260 [uber-SubtaskRunner] ERROR org.apache.pig.PigServer  
> - exception during parsing: Error during parsing. Could not resolve 
> org.apache.hive.hcatalog.pig.HCatLoader using imports: [, java.lang., 
> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
> Failed to parse: Pig script failed to parse: 
>  pig script failed to validate: 
> org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not 
> resolve org.apache.hive.hcatalog.pig.HCatLoader using imports: [, java.lang., 
> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
>   at 
> org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
>   at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1819)
>   at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1527)
>   at org.apache.pig.PigServer.parseAndBuild(PigServer.java:460)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:485)
>   at org.apache.pig.PigServer.executeBatch(PigServer.java:471)
>   at 
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:172)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:235)
>   at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
>   at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
>   at org.apache.pig.Main.run(Main.java:501)
>   at org.apache.pig.PigRunner.run(PigRunner.java:49)
>   at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:290)
>   at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:233)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:60)
>   at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:73)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:232)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:380)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:301)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:187)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:230)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: 
>  pig script failed to validate: 
> org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not 
> resolve org.apache.hive.hcatalog.pig.HCatLoader using imports: [, java.lang., 
> org.apache.pig.builtin., org.apache.pig.impl.builtin.]
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1339)
>   at 
> org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1324)
>   at 
> org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5184)
>   at 
> 

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (86 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues-test.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues-test.apache.org/jira/browse/OOZIE-2618
OOZIE-2617  Read properties files in action configurations
https://issues-test.apache.org/jira/browse/OOZIE-2617
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues-test.apache.org/jira/browse/OOZIE-2616
OOZIE-2606  Set spark.yarn.jars to fix Spark 2.0 with Oozie
https://issues-test.apache.org/jira/browse/OOZIE-2606
OOZIE-2589  CompletedActionXCommand priority param is of no effect
https://issues-test.apache.org/jira/browse/OOZIE-2589
OOZIE-2588  Support getting credentials for same cluster hcat when credentials 
config is empty
https://issues-test.apache.org/jira/browse/OOZIE-2588
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues-test.apache.org/jira/browse/OOZIE-2585
OOZIE-2584  Eliminate Thread.sleep() calls in TestMemoryLocks
https://issues-test.apache.org/jira/browse/OOZIE-2584
OOZIE-2583  oozie throws EL Exception when reference variable name containing 
dot
https://issues-test.apache.org/jira/browse/OOZIE-2583
OOZIE-2582  Populating external child Ids for action failures
https://issues-test.apache.org/jira/browse/OOZIE-2582
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues-test.apache.org/jira/browse/OOZIE-2573
OOZIE-2572  SLA DURATION miss not shown when job is running for longer than 
expected time
https://issues-test.apache.org/jira/browse/OOZIE-2572
OOZIE-2571  Add scala.binary.version Maven property so that Scala 2.11 can be 
used
https://issues-test.apache.org/jira/browse/OOZIE-2571
OOZIE-2569  Adding yarn-site, core-site, hdfs-site and mapred-site into spark 
launcher 
https://issues-test.apache.org/jira/browse/OOZIE-2569
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues-test.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues-test.apache.org/jira/browse/OOZIE-2564
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues-test.apache.org/jira/browse/OOZIE-2552
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues-test.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues-test.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues-test.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues-test.apache.org/jira/browse/OOZIE-2528
OOZIE-2521  Filter options are case sensitive for jobtye=bundle
https://issues-test.apache.org/jira/browse/OOZIE-2521
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues-test.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues-test.apache.org/jira/browse/OOZIE-2517
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues-test.apache.org/jira/browse/OOZIE-2513
OOZIE-2512  ShareLibservice returns incorrect path for jar
https://issues-test.apache.org/jira/browse/OOZIE-2512
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues-test.apache.org/jira/browse/OOZIE-2508
OOZIE-2501  ZK reentrant lock doesn't work for few cases
https://issues-test.apache.org/jira/browse/OOZIE-2501
OOZIE-2500  -DtestJarSimple option mentioned in minioozie doc does not work
https://issues-test.apache.org/jira/browse/OOZIE-2500
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues-test.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues-test.apache.org/jira/browse/OOZIE-2498
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues-test.apache.org/jira/browse/OOZIE-2495
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues-test.apache.org/jira/browse/OOZIE-2491

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (76 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2694  Add logging for FsActionExecutor 
https://issues.apache.org/jira/browse/OOZIE-2694
OOZIE-2693  SimpleHCatDependencyCache.removeMissingDependency can throw NPE
https://issues.apache.org/jira/browse/OOZIE-2693
OOZIE-2692  Oozie job submit doesn't report error message to user if there is 
any issue with job conf
https://issues.apache.org/jira/browse/OOZIE-2692
OOZIE-2691  Show workflow action retry count to UI
https://issues.apache.org/jira/browse/OOZIE-2691
OOZIE-2690  OOZIE NPE while executing kill()
https://issues.apache.org/jira/browse/OOZIE-2690
OOZIE-2684  Bad database schema error for WF_ACTIONS table
https://issues.apache.org/jira/browse/OOZIE-2684
OOZIE-2674  Improve oozie commads documentation
https://issues.apache.org/jira/browse/OOZIE-2674
OOZIE-2670  Upgrade Hbase to 1.2
https://issues.apache.org/jira/browse/OOZIE-2670
OOZIE-2668  Status update and recovery problems when coord action and its 
children not in sync
https://issues.apache.org/jira/browse/OOZIE-2668
OOZIE-2666  Support embedding Jetty into Oozie
https://issues.apache.org/jira/browse/OOZIE-2666
OOZIE-2662  DB migration fails if DB is too big
https://issues.apache.org/jira/browse/OOZIE-2662
OOZIE-2654  Zookeeper dependent services should not depend on Connectionstate 
to be valid before cleaning up
https://issues.apache.org/jira/browse/OOZIE-2654
OOZIE-2652  Skip trash while deleting hive table partition
https://issues.apache.org/jira/browse/OOZIE-2652
OOZIE-2650  Retry coord start on database exceptions
https://issues.apache.org/jira/browse/OOZIE-2650
OOZIE-2641  rerunning an oozie coordinator job stucks on waiting
https://issues.apache.org/jira/browse/OOZIE-2641
OOZIE-2630  Oozie Coordinator EL Functions to get first day of the week/month
https://issues.apache.org/jira/browse/OOZIE-2630
OOZIE-2619  Make  Hive action defaults to match hive defaults when running from 
command line
https://issues.apache.org/jira/browse/OOZIE-2619
OOZIE-2618  Include hive-common in hive-beeline as compile component for 
hive-server2 (hive2 action)
https://issues.apache.org/jira/browse/OOZIE-2618
OOZIE-2616  Add Tez profile for Hive and Pig sharelibs
https://issues.apache.org/jira/browse/OOZIE-2616
OOZIE-2585  Disable or remove test case 
TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeedOverflowToDisk and 
testMemoryUsageAndSpeed
https://issues.apache.org/jira/browse/OOZIE-2585
OOZIE-2573  dataset url  which contains spaces can not be handled rightly
https://issues.apache.org/jira/browse/OOZIE-2573
OOZIE-2568  SSH  action pretends to retry automaticly when it failed
https://issues.apache.org/jira/browse/OOZIE-2568
OOZIE-2564  Create new log4j config for unit tests so that logged messages 
contain threads
https://issues.apache.org/jira/browse/OOZIE-2564
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues.apache.org/jira/browse/OOZIE-2534
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues.apache.org/jira/browse/OOZIE-2528
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues.apache.org/jira/browse/OOZIE-2518
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues.apache.org/jira/browse/OOZIE-2513
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues.apache.org/jira/browse/OOZIE-2499
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues.apache.org/jira/browse/OOZIE-2495
OOZIE-2473  Connection pool for SMTP connection
https://issues.apache.org/jira/browse/OOZIE-2473
OOZIE-2461  Workflow, Coordinator and Bundle job querying should have last 
modified filter
https://issues.apache.org/jira/browse/OOZIE-2461
OOZIE-2458  'oozie-setup.sh sharelib create' should ensure uploaded jars are 
world readable
https://issues.apache.org/jira/browse/OOZIE-2458
OOZIE-2457  Oozie log parsing regex consume more than 90% cpu
https://issues.apache.org/jira/browse/OOZIE-2457
OOZIE-2427  With Kerberos enabled, the authentication failure for the 
javascript resources under /ext-2.2 
https://issues.apache.org/jira/browse/OOZIE-2427
OOZIE-2396  oozie compile with hadoop 2.7.1 failed
https://issues.apache.org/jira/browse/OOZIE-2396
OOZIE-2393  Allow table drop in hcat prepare
https://issues.apache.org/jira/browse/OOZIE-2393
OOZIE-2364  Remove deprecated