date:20170704

Jenkins build is back to normal : oozie-trunk-find-patches-available #162959

2017-07-04 Thread Apache Jenkins Server

See

Build failed in Jenkins: oozie-trunk-find-patches-available #162958

2017-07-04 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision c09561ff263f38ff4ecdf81f71359ae2ae4c90d5 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c09561ff263f38ff4ecdf81f71359ae2ae4c90d5
 > git rev-list c09561ff263f38ff4ecdf81f71359ae2ae4c90d5 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson6127680780765249244.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  200k0  200k0 0   217k  0 --:--:-- --:--:-- --:--:--  
217k100  815k0  815k0 0   434k  0 --:--:--  0:00:01 --:--:--  
434k100 1551k0 1551k0 0   537k  0 --:--:--  0:00:02 --:--:--  
537k100 2119k0 2119k0 0   531k  0 --:--:--  0:00:03 --:--:--  
531k100 2119k0 2119k0 0   531k  0 --:--:--  0:00:03 --:--:--  
531k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

[jira] [Commented] (OOZIE-2978) Remove code that handles Pig versions before 0.8

2017-07-04 Thread Attila Sasvari (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073799#comment-16073799
 ] 

Attila Sasvari commented on OOZIE-2978:
---

[~andras.piros], [~pbacsko] Could you take a look? 

In fact, the test  does not add/modify any testcase as per its purpose (i.e. 
deprecated code / test elimination).


> Remove code that handles Pig versions before 0.8 
> -
>
> Key: OOZIE-2978
> URL: https://issues.apache.org/jira/browse/OOZIE-2978
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2978-01.patch
>
>
> Oozie submits Pig jobs according to given Pig version. For example, Pig 
> versions before 0.8 are treated in the following way:
>  
> [Main.main(args)|https://github.com/apache/oozie/blob/678f1b2b2cc9a2456c7e3d9f693d2637f1f03257/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L382]
> However, we dropped support for *Hadoop 1*, *Hadoop 0.23* support via 
> {{OOZIE-2316}}. I wonder whether submitting jobs using Pig version before 0.8 
> will work at all:
> - 
> https://pig.apache.org/releases.html#13+May%2C+2010%3A+release+0.7.0+available
>  says "This release works with Hadoop 0.20". .
> - Looking at [PIG-2125|https://issues.apache.org/jira/browse/PIG-2125], I 
> believe the answer is NO. 
> Therefore we can safely delete corresponding parts from the codebase. There 
> is also another class {{PigMainWithOldAPI}} that is only used by tests in 
> {{TestPigMainWithOldAPI}}. So we can remove these too.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2662) DB migration fails if DB is too big

2017-07-04 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073687#comment-16073687
 ] 

Hadoop QA commented on OOZIE-2662:
--

Testing JIRA OOZIE-2662

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 10 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 6 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1971
.Tests rerun: 87
.Tests failed at first run: 
org.apache.oozie.command.coord.TestCoordKillXCommand,org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.jms.TestJMSJobEventListener,org.apache.oozie.action.hadoop.TestLauncherAM,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3945/

> DB migration fails if DB is too big
> ---
>
> Key: OOZIE-2662
> URL: https://issues.apache.org/jira/browse/OOZIE-2662
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2662.001.patch, OOZIE-2662.002.patch, 
> OOZIE-2662.002.wip.patch, OOZIE-2662.003.patch, OOZIE-2662.004.patch, 
> OOZIE-2662.005.patch
>
>
> The initial version of the DB import tool commits all the workflows, actions 
> etc. in one huge commit. If it does not fits into the memory, AOOME is thrown.
> We should commit every 1k or 10k elements to prevent this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: oozie-trunk-precommit-build #3945

2017-07-04 Thread Apache Jenkins Server

See 


--
[...truncated 359.11 KB...]
[INFO] Copying jackson-core-asl-1.9.13.jar to 

[INFO] Copying commons-jexl-2.1.1.jar to 

[INFO] Copying oozie-sharelib-hcatalog-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-jobclient-2.6.0.jar to 

[INFO] Copying gson-2.7.jar to 

[INFO] Copying oozie-core-5.0.0-SNAPSHOT.jar to 

[INFO] Copying commons-pool-1.5.4.jar to 

[INFO] Copying htrace-core-3.0.4.jar to 

[INFO] Copying jaxb-impl-2.2.3-1.jar to 

[INFO] Copying jets3t-0.9.0.jar to 

[INFO] Copying datanucleus-rdbms-3.2.9.jar to 

[INFO] Copying hive-common-1.2.0.jar to 

[INFO] Copying snappy-java-1.0.4.1.jar to 

[INFO] Copying javax.annotation-api-1.2.jar to 

[INFO] Copying commons-compiler-2.7.6.jar to 

[INFO] Copying ant-1.9.1.jar to 

[INFO] Copying hadoop-annotations-2.6.0.jar to 

[INFO] Copying janino-2.7.6.jar to 

[INFO] Copying activemq-client-5.13.3.jar to 

[INFO] Copying guice-3.0.jar to 

[INFO] Copying httpclient-4.3.6.jar to 

[INFO] Copying xz-1.0.jar to 

[INFO] Copying oozie-client-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-common-2.6.0.jar to 

[INFO] Copying leveldbjni-all-1.8.jar to 

[INFO] Copying jetty-util-6.1.26.jar to 

[INFO] Copying velocity-1.5.jar to 

[INFO] Copying jackson-core-2.4.2.jar to 

[INFO] Copying apache-log4j-extras-1.2.17.jar to 

[INFO] Copying netty-3.7.0.Final.jar to 

[INFO] Copying xmlenc-0.52.jar to 

[INFO]

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

2017-07-04 Thread Peter Bacsko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review179575
---




core/src/main/java/org/apache/oozie/service/JPAService.java
Lines 267 (patched)


Class instead of Class



core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java
Lines 32 (patched)


The name RETRY_ATTEMPT_COUNTER is misleading. It actually counts the number 
of nested retry calls.



core/src/main/java/org/apache/oozie/util/db/RetryAttemptCounter.java
Lines 41 (patched)


Let's think about the naming.. we don't just count with this class, we also 
store state information. If you were not happy with NestedRetryUti, we can call 
it NestedRetryHandler or anyhing.



core/src/main/java/org/apache/oozie/util/db/RetryAttemptCounter.java
Lines 62 (patched)


No need to use synchronized


- Peter Bacsko


On júl. 3, 2017, 3:15 du, András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated júl. 3, 2017, 3:15 du)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptCounter.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> c60a4581a84d4c67a1ac1cf3dfdc252b85ccd01c 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 161927ac8f1132b3080d2924844826fcc7b807a5 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/util/db/TestPersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestRetryAttemptCounter.java 
> PRE-CREATION 
>   minitest/pom.xml 9515284bb5f32c279a93161c10e6571680e4f9fc 
>   
> minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
>  PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/TestWorkflowRetries.java 
> PRE-CREATION 
>   minitest/src/test/java/org/apache/oozie/test/WorkflowTest.java 
> 2845f0af6efb9ef75fdbfcb326115c62e6fb3bdd 
>   minitest/src/test/resources/hsqldb-oozie-site.xml 
> fa5fe9c3185e973e8247d7bf10b126119d9c02c9 
>   minitest/src/test/resources/oozie-log4j.properties 
> c142d725140930bfa89cd2b163d0768a4c3a750a 
>   minitest/src/test/resources/parallel-fs-and-shell.xml PRE-CREATION 
>   minitest/src/test/resources/wf-test.xml 
> 20c4946862039a65c76ed7f49991345e90a694de 
>   pom.xml 16c5137d44d7db891da46f80adb51c85e4c1b214 
> 
> 
> Diff: https://reviews.apache.org/r/60544/diff/2/
> 
> 
> Testing
> ---
> 
> Tests covered in code:
> 
> Unit tests
> ==
> 
> * testing the retry handler, the retry predicate filter, and

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

2017-07-04 Thread Attila Sasvari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review179556
---



Please document the feature and its configuration in the twiki files.


core/pom.xml
Lines 39 (patched)


Do not add this dependency twice:

https://github.com/apache/oozie/blob/378f294c2cd92ac0d505201857f03746ce2e58ac/core/pom.xml#L743



core/src/main/java/org/apache/oozie/service/JPAService.java
Line 273 (original), 330 (patched)


- Extract method
```
if (em.getTransaction().isActive()) {
...
  em.getTransaction().commit();
}
```

- Extract constant "org.apache.oozie.command.SkipCommitFaultInjection"



core/src/main/java/org/apache/oozie/service/JPAService.java
Lines 351 (patched)


remove dead comnent



core/src/main/java/org/apache/oozie/service/JPAService.java
Line 333 (original), 399 (patched)


use ``commitTransaction(em)``



core/src/main/java/org/apache/oozie/service/JPAService.java
Lines 486 (patched)


this comment is not necessary



core/src/main/java/org/apache/oozie/service/JPAService.java
Lines 490 (patched)


I am not sure if ``em.getTransaction().isActive()`` will ever be evaluated 
true.

However, if so and rollback is not successfull, we should return and 
increase actual retry count (rollback is a kind of retry attempt).



core/src/main/java/org/apache/oozie/service/JPAService.java
Line 413 (original), 496 (patched)


Use ``&& !updateQueryList.isEmpty()`` instead of size() >0



core/src/main/java/org/apache/oozie/service/JPAService.java
Line 499 (original), 599 (patched)


Log that no results were returned like in ``executeGet`` to keep consistency



core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java
Lines 71 (patched)


This a new named query that it is only used in ``XTestCase``, and there is 
no check is performed on it in any tests; it is is used in 
``cleanUpDBTablesInternal()``. In other words, no REST API requests are using 
it for any tasks.

As I see there are other named queries that are not used for meaningful 
purposes (i.e. they are only used in tests). It would be nice to go through all 
named queries and eliminate the "dead" ones. For example "GET_ACTIONS" is only 
used in tests.



core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java
Lines 33 (patched)


Could you add DBCP-333 to reveal what the known bug is?



core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java
Lines 41 (patched)


If driverClassName is null, we can return earlier throwing a 
SQLNestedException with appropriate message. This way it will not be needed to 
check a second time driverClassName is null.



core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java
Lines 42 (patched)


1 try with multiple catch block should be enough



core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java
Lines 347 (patched)


nit: OozieDBCLI also uses the tablenames. We could extract these into a 
central place perhaps in a separate Jira?



core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java
Lines 31 (patched)


No need for this annotation; if  RETRY_ATTEMPT_COUNTER is package private, 
then TestOperationRetryHandler will access it. Or you can restrict visibility 
of RETRY_ATTEMPT_COUNTER to private.



core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java
Lines 74 (patched)


As a user, this error message would confuse me. 
``Retry attempts have been exhausted.`` / ``Database operation retry limit 
has been exceeded.`` would be more descriptive in my opinion.



core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java
Lines 86 (patched)


nit: 
"Exception is not on blacklist" or something would be better than 
"Exception does not apply"



core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java
Lines 101 (patched)


nit: sleepBeforeRetry or something would be more

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

2017-07-04 Thread Peter Bacsko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60544/#review179570
---




core/src/main/java/org/apache/oozie/service/JPAService.java
Line 272 (original), 329 (patched)


I really don't like this code! It's been here for a long time, but it's 
very smelly. 

Should we remove this? Could pls check if we have tests that take advantage 
of this fault injection>



core/src/main/java/org/apache/oozie/service/JPAService.java
Line 334 (original), 400 (patched)


Check this injection usage



core/src/main/java/org/apache/oozie/service/JPAService.java
Line 432 (original), 515 (patched)


Just like above - check if this is worth keeping



core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
Lines 27 (patched)


Could you add comments about this predicate class? Why we need it, when it 
applies, etc. 

Something like:

"A predicate which applies when a given exception (or its causes) are NOT 
blacklisted. Blacklisted exceptions in this class do not indicate a network 
failure, therefore no retry should take place"



core/src/main/java/org/apache/oozie/util/db/RetryAttemptCounter.java
Lines 28 (patched)


Let's rewrite this explanation and discuss f2f.



core/src/test/java/org/apache/oozie/test/XTestCase.java
Lines 872 (patched)


Just  instead of 



minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
Lines 54 (patched)


If we modify the log4j setup, don't we have to restore it in teardown?



minitest/src/test/java/org/apache/oozie/test/TestParallelJPAOperationRetries.java
Lines 80 (patched)


This assertion will not be evaluated by JUnit because it's called on a 
different thread. Instead modify a volatile boolean flag that something wasn't 
quite right.


- Peter Bacsko


On júl. 3, 2017, 3:15 du, András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60544/
> ---
> 
> (Updated júl. 3, 2017, 3:15 du)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/OOZIE-2854
> 
> 
> Diffs
> -
> 
>   core/pom.xml acddf349a89cf09a7fc4f384ebcaec56dfd0ab48 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/JsonBeanPersisterExecutor.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/executor/jpa/QueryExecutor.java 
> 8d94c23e40d1281864db40e141b200ca207a6324 
>   core/src/main/java/org/apache/oozie/service/JPAService.java 
> 028381d3b72bcc3b8c2cd27cacb3e0ac6d48d146 
>   core/src/main/java/org/apache/oozie/sla/SLASummaryBean.java 
> cfe1522a4b1f89085eb29e7f1281c2abd631bdc2 
>   core/src/main/java/org/apache/oozie/store/WorkflowStore.java 
> c565e74893b863caef6c93015cfe38fe520d04ec 
>   core/src/main/java/org/apache/oozie/util/db/BasicDataSourceWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/DatabaseRetryPredicate.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingConnectionWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingHSQLDBDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/FailingMySQLDriverWrapper.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/OperationRetryHandler.java 
> PRE-CREATION 
>   
> core/src/main/java/org/apache/oozie/util/db/PersistenceExceptionSubclassFilterRetryPredicate.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RetryAttemptCounter.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/db/RuntimeExceptionInjector.java 
> PRE-CREATION 
>   core/src/main/resources/META-INF/persistence.xml 
> bad9278597fcd4f93b4cc482afae8af14beaa922 
>   core/src/main/resources/oozie-default.xml 
> c60a4581a84d4c67a1ac1cf3dfdc252b85ccd01c 
>   core/src/main/resources/oozie-log4j.properties 
> c065f3cd4c5a3df1308b69d7c16e8fcfa8796efc 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 161927ac8f1132b3080d2924844826fcc7b807a5 
>   
> core/src/test/java/org/apache/oozie/util/db/TestOozieDmlStatementPredicate.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/util/db/TestOperationRetryHandler.java 
> PRE-CREATION

[jira] [Commented] (OOZIE-2918) Delete LauncherMapper and its test

2017-07-04 Thread Peter Bacsko (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073583#comment-16073583
 ] 

Peter Bacsko commented on OOZIE-2918:
-

+1

Thanks [~asasvari] committed to master

> Delete LauncherMapper and its test
> --
>
> Key: OOZIE-2918
> URL: https://issues.apache.org/jira/browse/OOZIE-2918
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2918-01.patch, OOZIE-2918-02.patch, 
> OOZIE-2918-03.patch, OOZIE-2918-04.patch
>
>
> While LauncherMapper itself is not used anymore it contains a bunch of static 
> methods that are referenced from different places in the code.
> We should refactor out code and related test cases to where they belong.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2662) DB migration fails if DB is too big

2017-07-04 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073554#comment-16073554
 ] 

Hadoop QA commented on OOZIE-2662:
--

Testing JIRA OOZIE-2662

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 10 line(s) longer than 132 
characters
.{color:green}+1{color} the patch does adds/modifies 6 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1971
.Tests failed: 8
.Tests errors: 2

.The patch failed the following testcases:

.  
testImportInvalidDataLeavesTablesEmpty(org.apache.oozie.tools.TestDBLoadDump)
.  
testImportToNonEmptyTablesCausesPrematureExit(org.apache.oozie.tools.TestDBLoadDump)
.  testImportedDBIsExportedCorrectly(org.apache.oozie.tools.TestDBLoadDump)
.  testMain(org.apache.oozie.action.hadoop.TestHiveMain)
.  testPig_withNullExternalID(org.apache.oozie.action.hadoop.TestPigMain)
.  testEmbeddedPigWithinPython(org.apache.oozie.action.hadoop.TestPigMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMain)
.  testPigScript(org.apache.oozie.action.hadoop.TestPigMainWithOldAPI)

.Tests failing with errors:
.  testAddXIncludeFromReader(org.apache.oozie.util.TestXConfiguration)
.  testAddXIncludeFromStream(org.apache.oozie.util.TestXConfiguration)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3944/

> DB migration fails if DB is too big
> ---
>
> Key: OOZIE-2662
> URL: https://issues.apache.org/jira/browse/OOZIE-2662
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2662.001.patch, OOZIE-2662.002.patch, 
> OOZIE-2662.002.wip.patch, OOZIE-2662.003.patch, OOZIE-2662.004.patch, 
> OOZIE-2662.005.patch
>
>
> The initial version of the DB import tool commits all the workflows, actions 
> etc. in one huge commit. If it does not fits into the memory, AOOME is thrown.
> We should commit every 1k or 10k elements to prevent this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: oozie-trunk-precommit-build #3944

2017-07-04 Thread Apache Jenkins Server

See 


--
[...truncated 368.13 KB...]
[INFO] Copying hadoop-annotations-2.6.0.jar to 

[INFO] Copying janino-2.7.6.jar to 

[INFO] Copying activemq-client-5.13.3.jar to 

[INFO] Copying guice-3.0.jar to 

[INFO] Copying httpclient-4.3.6.jar to 

[INFO] Copying xz-1.0.jar to 

[INFO] Copying oozie-client-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-common-2.6.0.jar to 

[INFO] Copying leveldbjni-all-1.8.jar to 

[INFO] Copying jetty-util-6.1.26.jar to 

[INFO] Copying velocity-1.5.jar to 

[INFO] Copying jackson-core-2.4.2.jar to 

[INFO] Copying apache-log4j-extras-1.2.17.jar to 

[INFO] Copying netty-3.7.0.Final.jar to 

[INFO] Copying xmlenc-0.52.jar to 

[INFO] Copying httpcore-4.3.3.jar to 

[INFO] Copying jetty-io-9.2.19.v20160908.jar to 

[INFO] Copying commons-beanutils-core-1.8.0.jar to 

[INFO] Copying antlr-2.7.7.jar to 

[INFO] Copying slf4j-log4j12-1.6.6.jar to 

[INFO] Copying datanucleus-api-jdo-3.2.6.jar to 

[INFO] Copying jung-api-2.0.1.jar to 

[INFO] Copying jta-1.1.jar to 

[INFO] Copying apacheds-kerberos-codec-2.0.0-M15.jar to 

[INFO] Copying kryo-2.22.jar to 

[INFO] Copying jackson-databind-2.4.2.jar to 

[INFO] Copying hive-serde-1.2.0.jar to 

[INFO] Copying jetty-client-9.2.19.v20160908.jar to 

[INFO] Copying hadoop-yarn-server-web-proxy-2.6.0.jar to 

[INFO] Copying hamcrest-core-1.3.jar to 

[INFO] Copying jetty-rewrite-9.2.19.v20160908.jar to

[jira] [Updated] (OOZIE-2662) DB migration fails if DB is too big

2017-07-04 Thread Andras Piros (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2662:

Attachment: OOZIE-2662.005.patch

Addressing new review comments.

> DB migration fails if DB is too big
> ---
>
> Key: OOZIE-2662
> URL: https://issues.apache.org/jira/browse/OOZIE-2662
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2662.001.patch, OOZIE-2662.002.patch, 
> OOZIE-2662.002.wip.patch, OOZIE-2662.003.patch, OOZIE-2662.004.patch, 
> OOZIE-2662.005.patch
>
>
> The initial version of the DB import tool commits all the workflows, actions 
> etc. in one huge commit. If it does not fits into the memory, AOOME is thrown.
> We should commit every 1k or 10k elements to prevent this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

2017-07-04 Thread András Piros


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/
---

(Updated July 4, 2017, noon)


Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and Robert 
Kanter.


Changes
---

Addressing new review comments.


Repository: oozie-git


Description
---

We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
size. Furthermore, of at least one row inside that tx fails, we retry the whole 
batch into separate `EntityTransaction`s each.

Following error handling is implemented:

1. check if all necessary tables are present and empty
2. rows are imported till the end even if there are skipped rows in the 
meanwhile
3. if at least one row is skipped in the meanwhile for some 
`ConstraintViolationException`, we delete all rows of all necessary tables. 
That enables the user to have the log messages of all the erroneous rows in one 
run, and Oozie database is never in an inconsistent state of some rows present, 
some not present of an import


Diffs (updated)
-

  core/src/main/resources/oozie-default.xml 
c60a4581a84d4c67a1ac1cf3dfdc252b85ccd01c 
  tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 
0e14a30693a76b8b2bdc2f7ceaf3f045d69f4155 
  tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java 
c43223ef05aa702be49565ba2626314628e63749 
  tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
  tools/src/test/resources/dumpData/ooziedb_ac.json  
  tools/src/test/resources/dumpData/ooziedb_bna.json  
  tools/src/test/resources/dumpData/ooziedb_bnj.json  
  tools/src/test/resources/dumpData/ooziedb_ca.json  
  tools/src/test/resources/dumpData/ooziedb_cj.json  
  tools/src/test/resources/dumpData/ooziedb_slareg.json  
  tools/src/test/resources/dumpData/ooziedb_slasum.json  
  tools/src/test/resources/dumpData/ooziedb_sysinfo.json  
  tools/src/test/resources/dumpData/ooziedb_wf.json  
  tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION 


Diff: https://reviews.apache.org/r/52782/diff/5/

Changes: https://reviews.apache.org/r/52782/diff/4-5/


Testing
---

See `TestDBLoadDump` for further reference.


Thanks,

András Piros

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

2017-07-04 Thread András Piros



> On July 4, 2017, 11:21 a.m., Peter Bacsko wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
> > Lines 488 (patched)
> > 
> >
> > Is it possible that currentTransaction() is already open at this point? 
> > If I'm not mistaken, calling begin() again results in an exception being 
> > thrown.

`persistAndTryCommit()` should only be called after the previous 
`EntityTransaction` has already been rolled back. So no, `currentTransaction` 
should not be open. Adding a check.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/#review179560
---


On July 4, 2017, 10:09 a.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52782/
> ---
> 
> (Updated July 4, 2017, 10:09 a.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
> size. Furthermore, of at least one row inside that tx fails, we retry the 
> whole batch into separate `EntityTransaction`s each.
> 
> Following error handling is implemented:
> 
> 1. check if all necessary tables are present and empty
> 2. rows are imported till the end even if there are skipped rows in the 
> meanwhile
> 3. if at least one row is skipped in the meanwhile for some 
> `ConstraintViolationException`, we delete all rows of all necessary tables. 
> That enables the user to have the log messages of all the erroneous rows in 
> one run, and Oozie database is never in an inconsistent state of some rows 
> present, some not present of an import
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 
> 0e14a30693a76b8b2bdc2f7ceaf3f045d69f4155 
>   tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java 
> c43223ef05aa702be49565ba2626314628e63749 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
>   tools/src/test/resources/dumpData/ooziedb_ac.json  
>   tools/src/test/resources/dumpData/ooziedb_bna.json  
>   tools/src/test/resources/dumpData/ooziedb_bnj.json  
>   tools/src/test/resources/dumpData/ooziedb_ca.json  
>   tools/src/test/resources/dumpData/ooziedb_cj.json  
>   tools/src/test/resources/dumpData/ooziedb_slareg.json  
>   tools/src/test/resources/dumpData/ooziedb_slasum.json  
>   tools/src/test/resources/dumpData/ooziedb_sysinfo.json  
>   tools/src/test/resources/dumpData/ooziedb_wf.json  
>   tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/52782/diff/4/
> 
> 
> Testing
> ---
> 
> See `TestDBLoadDump` for further reference.
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Commented] (OOZIE-2978) Remove code that handles Pig versions before 0.8

2017-07-04 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073497#comment-16073497
 ] 

Hadoop QA commented on OOZIE-2978:
--

Testing JIRA OOZIE-2978

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 1 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1967
.Tests rerun: 79
.Tests failed at first run: 
org.apache.oozie.action.hadoop.TestJavaActionExecutor,org.apache.oozie.jms.TestJMSJobEventListener,org.apache.oozie.action.hadoop.TestLauncherAM,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3943/

> Remove code that handles Pig versions before 0.8 
> -
>
> Key: OOZIE-2978
> URL: https://issues.apache.org/jira/browse/OOZIE-2978
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2978-01.patch
>
>
> Oozie submits Pig jobs according to given Pig version. For example, Pig 
> versions before 0.8 are treated in the following way:
>  
> [Main.main(args)|https://github.com/apache/oozie/blob/678f1b2b2cc9a2456c7e3d9f693d2637f1f03257/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L382]
> However, we dropped support for *Hadoop 1*, *Hadoop 0.23* support via 
> {{OOZIE-2316}}. I wonder whether submitting jobs using Pig version before 0.8 
> will work at all:
> - 
> https://pig.apache.org/releases.html#13+May%2C+2010%3A+release+0.7.0+available
>  says "This release works with Hadoop 0.20". .
> - Looking at [PIG-2125|https://issues.apache.org/jira/browse/PIG-2125], I 
> believe the answer is NO. 
> Therefore we can safely delete corresponding parts from the codebase. There 
> is also another class {{PigMainWithOldAPI}} that is only used by tests in 
> {{TestPigMainWithOldAPI}}. So we can remove these too.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

2017-07-04 Thread Peter Bacsko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/#review179561
---




tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
Lines 95 (patched)


Should we make this configurable?

Can a bigger value possibly speed up the whole process?


- Peter Bacsko


On júl. 4, 2017, 10:09 de, András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52782/
> ---
> 
> (Updated júl. 4, 2017, 10:09 de)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
> size. Furthermore, of at least one row inside that tx fails, we retry the 
> whole batch into separate `EntityTransaction`s each.
> 
> Following error handling is implemented:
> 
> 1. check if all necessary tables are present and empty
> 2. rows are imported till the end even if there are skipped rows in the 
> meanwhile
> 3. if at least one row is skipped in the meanwhile for some 
> `ConstraintViolationException`, we delete all rows of all necessary tables. 
> That enables the user to have the log messages of all the erroneous rows in 
> one run, and Oozie database is never in an inconsistent state of some rows 
> present, some not present of an import
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 
> 0e14a30693a76b8b2bdc2f7ceaf3f045d69f4155 
>   tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java 
> c43223ef05aa702be49565ba2626314628e63749 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
>   tools/src/test/resources/dumpData/ooziedb_ac.json  
>   tools/src/test/resources/dumpData/ooziedb_bna.json  
>   tools/src/test/resources/dumpData/ooziedb_bnj.json  
>   tools/src/test/resources/dumpData/ooziedb_ca.json  
>   tools/src/test/resources/dumpData/ooziedb_cj.json  
>   tools/src/test/resources/dumpData/ooziedb_slareg.json  
>   tools/src/test/resources/dumpData/ooziedb_slasum.json  
>   tools/src/test/resources/dumpData/ooziedb_sysinfo.json  
>   tools/src/test/resources/dumpData/ooziedb_wf.json  
>   tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/52782/diff/4/
> 
> 
> Testing
> ---
> 
> See `TestDBLoadDump` for further reference.
> 
> 
> Thanks,
> 
> András Piros
> 
>

Build failed in Jenkins: oozie-trunk-precommit-build #3943

2017-07-04 Thread Apache Jenkins Server

See 


--
[...truncated 355.62 KB...]
[INFO] Copying jackson-core-asl-1.9.13.jar to 

[INFO] Copying commons-jexl-2.1.1.jar to 

[INFO] Copying oozie-sharelib-hcatalog-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-jobclient-2.6.0.jar to 

[INFO] Copying gson-2.7.jar to 

[INFO] Copying oozie-core-5.0.0-SNAPSHOT.jar to 

[INFO] Copying commons-pool-1.5.4.jar to 

[INFO] Copying htrace-core-3.0.4.jar to 

[INFO] Copying jaxb-impl-2.2.3-1.jar to 

[INFO] Copying jets3t-0.9.0.jar to 

[INFO] Copying datanucleus-rdbms-3.2.9.jar to 

[INFO] Copying hive-common-1.2.0.jar to 

[INFO] Copying snappy-java-1.0.4.1.jar to 

[INFO] Copying javax.annotation-api-1.2.jar to 

[INFO] Copying commons-compiler-2.7.6.jar to 

[INFO] Copying ant-1.9.1.jar to 

[INFO] Copying hadoop-annotations-2.6.0.jar to 

[INFO] Copying janino-2.7.6.jar to 

[INFO] Copying activemq-client-5.13.3.jar to 

[INFO] Copying guice-3.0.jar to 

[INFO] Copying httpclient-4.3.6.jar to 

[INFO] Copying xz-1.0.jar to 

[INFO] Copying oozie-client-5.0.0-SNAPSHOT.jar to 

[INFO] Copying hadoop-mapreduce-client-common-2.6.0.jar to 

[INFO] Copying leveldbjni-all-1.8.jar to 

[INFO] Copying jetty-util-6.1.26.jar to 

[INFO] Copying velocity-1.5.jar to 

[INFO] Copying jackson-core-2.4.2.jar to 

[INFO] Copying apache-log4j-extras-1.2.17.jar to 

[INFO] Copying netty-3.7.0.Final.jar to 

[INFO] Copying xmlenc-0.52.jar to 

[INFO]

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

2017-07-04 Thread Peter Bacsko



> On jún. 28, 2017, 11:14 de, Peter Bacsko wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
> > Lines 353 (patched)
> > 
> >
> > Do we have to instantiate the batch handling mechanism just for the 
> > sake of a Tx begin/commit?
> 
> András Piros wrote:
> I think `BatchTransactionGuard` is pretty lightweight, and it delivers 
> some statistics that may be of good use. It's also a good practice to have 
> appropriate levels of abstraction, and use them.

To me it's a bit distracting. If a read the code, it's not immediately clear 
why we have a "batch handler" if we don't do any batching. We already have the 
entityManager instance in use in the same scope.


> On jún. 28, 2017, 11:14 de, Peter Bacsko wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
> > Lines 395 (patched)
> > 
> >
> > Define the size of the list?
> 
> András Piros wrote:
> To know the exact size we need to iterate through the file anyway, 
> unfortunately. In practice I didn't encounter measurable performance 
> degradation because of using an auto-growing `ArrayList`.

Was thinking about a practical value like 1024 or 2048, but nevermind.


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/#review179086
---


On júl. 4, 2017, 10:09 de, András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52782/
> ---
> 
> (Updated júl. 4, 2017, 10:09 de)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
> size. Furthermore, of at least one row inside that tx fails, we retry the 
> whole batch into separate `EntityTransaction`s each.
> 
> Following error handling is implemented:
> 
> 1. check if all necessary tables are present and empty
> 2. rows are imported till the end even if there are skipped rows in the 
> meanwhile
> 3. if at least one row is skipped in the meanwhile for some 
> `ConstraintViolationException`, we delete all rows of all necessary tables. 
> That enables the user to have the log messages of all the erroneous rows in 
> one run, and Oozie database is never in an inconsistent state of some rows 
> present, some not present of an import
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 
> 0e14a30693a76b8b2bdc2f7ceaf3f045d69f4155 
>   tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java 
> c43223ef05aa702be49565ba2626314628e63749 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
>   tools/src/test/resources/dumpData/ooziedb_ac.json  
>   tools/src/test/resources/dumpData/ooziedb_bna.json  
>   tools/src/test/resources/dumpData/ooziedb_bnj.json  
>   tools/src/test/resources/dumpData/ooziedb_ca.json  
>   tools/src/test/resources/dumpData/ooziedb_cj.json  
>   tools/src/test/resources/dumpData/ooziedb_slareg.json  
>   tools/src/test/resources/dumpData/ooziedb_slasum.json  
>   tools/src/test/resources/dumpData/ooziedb_sysinfo.json  
>   tools/src/test/resources/dumpData/ooziedb_wf.json  
>   tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/52782/diff/4/
> 
> 
> Testing
> ---
> 
> See `TestDBLoadDump` for further reference.
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

2017-07-04 Thread Peter Bacsko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/#review179560
---




tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
Lines 413 (patched)


Could you rephrese the first sentence pls? It sounds a bit cryptic.

Possible sentence I can think of: "Persists entities in batches, that is, 
the actual commit will be done when the number of persistable entities reach 
the batch limit." Just an example.

Also explain why we need to commit in batches (IIRC it was because of OOME, 
right?)



tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
Lines 467 (patched)


Please check if extra text is necessary after newEntity (JDK8 javadoc is 
very strict)



tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
Lines 484 (patched)


Same here



tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
Lines 488 (patched)


Is it possible that currentTransaction() is already open at this point? If 
I'm not mistaken, calling begin() again results in an exception being thrown.



tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
Lines 505 (patched)


Java8 javadoc checker will whine about the missing text after @return.



tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
Lines 513 (patched)


Same here



tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java
Lines 234 (patched)


Another Class :)



tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java
Lines 241 (patched)


If the class passed here can only be WorkflowActionBean, does it make sense 
to have this argument at all?



tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java
Lines 264 (patched)


Class


- Peter Bacsko


On júl. 4, 2017, 10:09 de, András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52782/
> ---
> 
> (Updated júl. 4, 2017, 10:09 de)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
> size. Furthermore, of at least one row inside that tx fails, we retry the 
> whole batch into separate `EntityTransaction`s each.
> 
> Following error handling is implemented:
> 
> 1. check if all necessary tables are present and empty
> 2. rows are imported till the end even if there are skipped rows in the 
> meanwhile
> 3. if at least one row is skipped in the meanwhile for some 
> `ConstraintViolationException`, we delete all rows of all necessary tables. 
> That enables the user to have the log messages of all the erroneous rows in 
> one run, and Oozie database is never in an inconsistent state of some rows 
> present, some not present of an import
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 
> 0e14a30693a76b8b2bdc2f7ceaf3f045d69f4155 
>   tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java 
> c43223ef05aa702be49565ba2626314628e63749 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
>   tools/src/test/resources/dumpData/ooziedb_ac.json  
>   tools/src/test/resources/dumpData/ooziedb_bna.json  
>   tools/src/test/resources/dumpData/ooziedb_bnj.json  
>   tools/src/test/resources/dumpData/ooziedb_ca.json  
>   tools/src/test/resources/dumpData/ooziedb_cj.json  
>   tools/src/test/resources/dumpData/ooziedb_slareg.json  
>   tools/src/test/resources/dumpData/ooziedb_slasum.json  
>   tools/src/test/resources/dumpData/ooziedb_sysinfo.json  
>   tools/src/test/resources/dumpData/ooziedb_wf.json  
>   tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
>

[jira] [Updated] (OOZIE-2662) DB migration fails if DB is too big

2017-07-04 Thread Andras Piros (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2662:

Attachment: OOZIE-2662.004.patch

Addressing review comments.

> DB migration fails if DB is too big
> ---
>
> Key: OOZIE-2662
> URL: https://issues.apache.org/jira/browse/OOZIE-2662
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2662.001.patch, OOZIE-2662.002.patch, 
> OOZIE-2662.002.wip.patch, OOZIE-2662.003.patch, OOZIE-2662.004.patch
>
>
> The initial version of the DB import tool commits all the workflows, actions 
> etc. in one huge commit. If it does not fits into the memory, AOOME is thrown.
> We should commit every 1k or 10k elements to prevent this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

2017-07-04 Thread András Piros


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/
---

(Updated July 4, 2017, 10:09 a.m.)


Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and Robert 
Kanter.


Changes
---

Addressing review comments.


Repository: oozie-git


Description
---

We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
size. Furthermore, of at least one row inside that tx fails, we retry the whole 
batch into separate `EntityTransaction`s each.

Following error handling is implemented:

1. check if all necessary tables are present and empty
2. rows are imported till the end even if there are skipped rows in the 
meanwhile
3. if at least one row is skipped in the meanwhile for some 
`ConstraintViolationException`, we delete all rows of all necessary tables. 
That enables the user to have the log messages of all the erroneous rows in one 
run, and Oozie database is never in an inconsistent state of some rows present, 
some not present of an import


Diffs (updated)
-

  tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 
0e14a30693a76b8b2bdc2f7ceaf3f045d69f4155 
  tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java 
c43223ef05aa702be49565ba2626314628e63749 
  tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
  tools/src/test/resources/dumpData/ooziedb_ac.json  
  tools/src/test/resources/dumpData/ooziedb_bna.json  
  tools/src/test/resources/dumpData/ooziedb_bnj.json  
  tools/src/test/resources/dumpData/ooziedb_ca.json  
  tools/src/test/resources/dumpData/ooziedb_cj.json  
  tools/src/test/resources/dumpData/ooziedb_slareg.json  
  tools/src/test/resources/dumpData/ooziedb_slasum.json  
  tools/src/test/resources/dumpData/ooziedb_sysinfo.json  
  tools/src/test/resources/dumpData/ooziedb_wf.json  
  tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION 


Diff: https://reviews.apache.org/r/52782/diff/4/

Changes: https://reviews.apache.org/r/52782/diff/3-4/


Testing
---

See `TestDBLoadDump` for further reference.


Thanks,

András Piros

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

2017-07-04 Thread András Piros



> On June 28, 2017, 11:14 a.m., Peter Bacsko wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
> > Line 116 (original), 154 (patched)
> > 
> >
> > Wouldn't it be better if this was a public non-static method? That 
> > would make unit testing easier because we wouldn't have to mess around with 
> > calling main() method with proper args to perform the DB export/import. The 
> > main() could still be tested separately. 
> > 
> > Altough it would require modifying the export part, too (so that the 
> > two classes have the same style).
> > 
> > Anyway we can consider this improvement.

Since `TestDBLoadDump` is more of and end-to-end at the moment, I'd leave like 
it is.


> On June 28, 2017, 11:14 a.m., Peter Bacsko wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
> > Lines 353 (patched)
> > 
> >
> > Do we have to instantiate the batch handling mechanism just for the 
> > sake of a Tx begin/commit?

I think `BatchTransactionGuard` is pretty lightweight, and it delivers some 
statistics that may be of good use. It's also a good practice to have 
appropriate levels of abstraction, and use them.


> On June 28, 2017, 11:14 a.m., Peter Bacsko wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java
> > Lines 395 (patched)
> > 
> >
> > Define the size of the list?

To know the exact size we need to iterate through the file anyway, 
unfortunately. In practice I didn't encounter measurable performance 
degradation because of using an auto-growing `ArrayList`.


> On June 28, 2017, 11:14 a.m., Peter Bacsko wrote:
> > tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java
> > Lines 156 (patched)
> > 
> >
> > Is this extra method call necessary?

IMO it makes code more readable. Zero parameters are always better than one ;)


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/#review179086
---


On June 27, 2017, 10:14 a.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52782/
> ---
> 
> (Updated June 27, 2017, 10:14 a.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and 
> Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
> size. Furthermore, of at least one row inside that tx fails, we retry the 
> whole batch into separate `EntityTransaction`s each.
> 
> Following error handling is implemented:
> 
> 1. check if all necessary tables are present and empty
> 2. rows are imported till the end even if there are skipped rows in the 
> meanwhile
> 3. if at least one row is skipped in the meanwhile for some 
> `ConstraintViolationException`, we delete all rows of all necessary tables. 
> That enables the user to have the log messages of all the erroneous rows in 
> one run, and Oozie database is never in an inconsistent state of some rows 
> present, some not present of an import
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 0e14a30 
>   tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java c43223e 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_bna.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_bnj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_slareg.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_slasum.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
>   tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
>   tools/src/test/resources/dumpData/ooziedb_ac.json 22bbdc2 
>   tools/src/test/resources/dumpData/ooziedb_bna.json e69de29 
>   tools/src/test/resources/dumpData/ooziedb_bnj.json e69de29 
>   tools/src/test/resources/dumpData/ooziedb_ca.json 2715b94 
>   tools/src/test/resources/dumpData/ooziedb_cj.json 979c10e 
>   tools/src/test/resources/dumpData/ooziedb_slareg.json e69de29 
>   tools/src/test/resources/dumpData/ooziedb_slasum.json e69de29 
>

[jira] [Updated] (OOZIE-2978) Remove code that handles Pig versions before 0.8

2017-07-04 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2978:
--
Attachment: OOZIE-2978-01.patch

Tests I performed:
{{mvn  clean test  -Dtest=TestPigMain}}

> Remove code that handles Pig versions before 0.8 
> -
>
> Key: OOZIE-2978
> URL: https://issues.apache.org/jira/browse/OOZIE-2978
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
> Attachments: OOZIE-2978-01.patch
>
>
> Oozie submits Pig jobs according to given Pig version. For example, Pig 
> versions before 0.8 are treated in the following way:
>  
> [Main.main(args)|https://github.com/apache/oozie/blob/678f1b2b2cc9a2456c7e3d9f693d2637f1f03257/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L382]
> However, we dropped support for *Hadoop 1*, *Hadoop 0.23* support via 
> {{OOZIE-2316}}. I wonder whether submitting jobs using Pig version before 0.8 
> will work at all:
> - 
> https://pig.apache.org/releases.html#13+May%2C+2010%3A+release+0.7.0+available
>  says "This release works with Hadoop 0.20". .
> - Looking at [PIG-2125|https://issues.apache.org/jira/browse/PIG-2125], I 
> believe the answer is NO. 
> Therefore we can safely delete corresponding parts from the codebase. There 
> is also another class {{PigMainWithOldAPI}} that is only used by tests in 
> {{TestPigMainWithOldAPI}}. So we can remove these too.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2918) Delete LauncherMapper and its test

2017-07-04 Thread Attila Sasvari (JIRA)


[ 
https://issues.apache.org/jira/browse/OOZIE-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073327#comment-16073327
 ] 

Attila Sasvari commented on OOZIE-2918:
---

[~pbacsko] I uploaded latest patch to reviewboard too, and answered the issue 
you opened regarding PigMainWithOldAPI. I don't think that class is in use 
anymore. At the same time, I believe we can get rid off code that handles old 
pig versions (<0.8) because we dropped support for Hadoop 0.2x and Hadoop 1.To 
this end, I opened OOZIE-2978. 

> Delete LauncherMapper and its test
> --
>
> Key: OOZIE-2918
> URL: https://issues.apache.org/jira/browse/OOZIE-2918
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2918-01.patch, OOZIE-2918-02.patch, 
> OOZIE-2918-03.patch, OOZIE-2918-04.patch
>
>
> While LauncherMapper itself is not used anymore it contains a bunch of static 
> methods that are referenced from different places in the code.
> We should refactor out code and related test cases to where they belong.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-2978) Remove code that handles Pig versions before 0.8

2017-07-04 Thread Attila Sasvari (JIRA)


 [ 
https://issues.apache.org/jira/browse/OOZIE-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari reassigned OOZIE-2978:
-

Assignee: Attila Sasvari

> Remove code that handles Pig versions before 0.8 
> -
>
> Key: OOZIE-2978
> URL: https://issues.apache.org/jira/browse/OOZIE-2978
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>
> Oozie submits Pig jobs according to given Pig version. For example, Pig 
> versions before 0.8 are treated in the following way:
>  
> [Main.main(args)|https://github.com/apache/oozie/blob/678f1b2b2cc9a2456c7e3d9f693d2637f1f03257/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L382]
> However, we dropped support for *Hadoop 1*, *Hadoop 0.23* support via 
> {{OOZIE-2316}}. I wonder whether submitting jobs using Pig version before 0.8 
> will work at all:
> - 
> https://pig.apache.org/releases.html#13+May%2C+2010%3A+release+0.7.0+available
>  says "This release works with Hadoop 0.20". .
> - Looking at [PIG-2125|https://issues.apache.org/jira/browse/PIG-2125], I 
> believe the answer is NO. 
> Therefore we can safely delete corresponding parts from the codebase. There 
> is also another class {{PigMainWithOldAPI}} that is only used by tests in 
> {{TestPigMainWithOldAPI}}. So we can remove these too.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-2978) Remove code that handles Pig versions before 0.8

2017-07-04 Thread Attila Sasvari (JIRA)

Attila Sasvari created OOZIE-2978:
-

 Summary: Remove code that handles Pig versions before 0.8 
 Key: OOZIE-2978
 URL: https://issues.apache.org/jira/browse/OOZIE-2978
 Project: Oozie
  Issue Type: Bug
Reporter: Attila Sasvari


Oozie submits Pig jobs according to given Pig version. For example, Pig 
versions before 0.8 are treated in the following way:
 
[Main.main(args)|https://github.com/apache/oozie/blob/678f1b2b2cc9a2456c7e3d9f693d2637f1f03257/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L382]

However, we dropped support for *Hadoop 1*, *Hadoop 0.23* support via 
{{OOZIE-2316}}. I wonder whether submitting jobs using Pig version before 0.8 
will work at all:
- 
https://pig.apache.org/releases.html#13+May%2C+2010%3A+release+0.7.0+available 
says "This release works with Hadoop 0.20". .
- Looking at [PIG-2125|https://issues.apache.org/jira/browse/PIG-2125], I 
believe the answer is NO. 

Therefore we can safely delete corresponding parts from the codebase. There is 
also another class {{PigMainWithOldAPI}} that is only used by tests in 
{{TestPigMainWithOldAPI}}. So we can remove these too.
 




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Workflow Action states

2017-07-04 Thread Daniel Becker

Hi
I tried to send an e-mail to the list but I think it didn't go through so
I'm sending it again. Sorry if it becomes a duplicate.
Here's the original e-mail:

Hi

I was wondering if somebody knows something about the Oozie workflow action
states (not the workflow job states). These are as follows:

• DONE
• END_MANUAL
• END_RETRY
• ERROR
• FAILED
• KILLED
• OK
• PREP
• RUNNING
• START_MANUAL
• START_RETRY
• USER_RETRY

I'm curious what the individual states mean, especially USER_RETRY, and the
difference between FAILED and ERROR.

I also find it a bit strange that in the test
*org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand.testRetryConsoleUrl()*
 the test action's status changes in the following way:
PREP --> RUNNING --> *DONE* --> USER_RETRY --> RUNNING --> *DONE* ...

Why does it go to DONE? I would think DONE meant the end of the action, but
here it is not that. Could somebody please explain?

Thank you for your help,
Daniel

On Mon, Jul 3, 2017 at 5:24 PM, Daniel Becker 
wrote:

> Hi
>
> I was wondering if somebody knows something about the Oozie workflow
> action states (not the workflow job states). These are as follows:
>
> • DONE
> • END_MANUAL
> • END_RETRY
> • ERROR
> • FAILED
> • KILLED
> • OK
> • PREP
> • RUNNING
> • START_MANUAL
> • START_RETRY
> • USER_RETRY
>
> I'm curious what the individual states mean, especially USER_RETRY, and
> the difference between FAILED and ERROR.
>
> I also find it a bit strange that in the test
> *org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand.testRetryConsoleUrl()*
>  the test action's status changes in the following way:
> PREP --> RUNNING --> *DONE* --> USER_RETRY --> RUNNING --> *DONE* ...
>
> Why does it go to DONE? I would think DONE meant the end of the action,
> but here it is not that. Could somebody please explain?
>
> Thank you for your help,
> Daniel
>
>

[jira] Subscription: Oozie Patch Available

2017-07-04 Thread jira

Issue Subscription
Filter: Oozie Patch Available (109 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2977  error: self-closing element not allowed
https://issues.apache.org/jira/browse/OOZIE-2977
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2974  Change TestLiteWorkflowAppParser so that it works with Hadoop 3
https://issues.apache.org/jira/browse/OOZIE-2974
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  getting at 
org.apache.maven.plugin.javadoc.AbstractJavadocMojo.executeJavadocCommandLine(AbstractJavadocMojo.java:5163)
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2963  getting error in build ArtifactNotFoundException: Could not find 
artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde
https://issues.apache.org/jira/browse/OOZIE-2963
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2961  build contains multiple warnings for is not a parameter name
https://issues.apache.org/jira/browse/OOZIE-2961
OOZIE-2960  many warnings for @return tag has no arguments
https://issues.apache.org/jira/browse/OOZIE-2960
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2931  multiple warnings during build for "no @param for"
https://issues.apache.org/jira/browse/OOZIE-2931
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2918  Delete LauncherMapper and its test
https://issues.apache.org/jira/browse/OOZIE-2918
OOZIE-2916  Set a job name for the MR Action's child job
https://issues.apache.org/jira/browse/OOZIE-2916
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2911  Re-add test testWfActionKillChildJob and adapt it to OYA
https://issues.apache.org/jira/browse/OOZIE-2911
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2854  Oozie should handle transient database problems
https://issues.apache.org/jira/browse/OOZIE-2854
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2784  Include WEEK as a parameter in the

Jenkins build is back to normal : oozie-trunk-find-patches-available #162959

Build failed in Jenkins: oozie-trunk-find-patches-available #162958

[jira] [Commented] (OOZIE-2978) Remove code that handles Pig versions before 0.8

[jira] [Commented] (OOZIE-2662) DB migration fails if DB is too big

Build failed in Jenkins: oozie-trunk-precommit-build #3945

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

Re: Review Request 60544: OOZIE-2854 Oozie should handle transient database problems

[jira] [Commented] (OOZIE-2918) Delete LauncherMapper and its test

[jira] [Commented] (OOZIE-2662) DB migration fails if DB is too big

Build failed in Jenkins: oozie-trunk-precommit-build #3944

[jira] [Updated] (OOZIE-2662) DB migration fails if DB is too big

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

[jira] [Commented] (OOZIE-2978) Remove code that handles Pig versions before 0.8

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

Build failed in Jenkins: oozie-trunk-precommit-build #3943

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

[jira] [Updated] (OOZIE-2662) DB migration fails if DB is too big

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

Re: Review Request 52782: OOZIE-2662 DB migration fails if DB is too big

[jira] [Updated] (OOZIE-2978) Remove code that handles Pig versions before 0.8

[jira] [Commented] (OOZIE-2918) Delete LauncherMapper and its test

[jira] [Assigned] (OOZIE-2978) Remove code that handles Pig versions before 0.8

[jira] [Created] (OOZIE-2978) Remove code that handles Pig versions before 0.8

Re: Workflow Action states

[jira] Subscription: Oozie Patch Available

28 matches

Site Navigation

Mail list logo

Footer information