[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603353#comment-16603353
 ] 

Hadoop QA commented on OOZIE-3160:
--


Testing JIRA OOZIE-3160

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [2] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:red}-1{color} There are [2] new bugs found below threshold in [core] 
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
core/findbugs-new.html
. The most important FindBugs errors are:
. At AsyncXCommandExecutor.java:[line 334]: 
org.apache.oozie.service.AsyncXCommandExecutor$AccessibleRunnableScheduledFuture
 defines compareTo(Object) and uses Object.equals()
. At AsyncXCommandExecutor.java:[lines 326-329]: 
org.apache.oozie.service.AsyncXCommandExecutor$PriorityComparator implements 
Comparator but not Serializable
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [webapp].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2967
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/807/



> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-001.patch, OOZIE-3160-002.patch, OOZIE-3160-003.patch, 
> OOZIE-3160-004.patch, OOZIE-3160-005.patch, OOZIE-3160-006.patch, 
> OOZIE-3160-POC01.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, 
> OOZIE-3160-POC03.patch, OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, 
> PriorityDelayQueue improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public 

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603242#comment-16603242
 ] 

Andras Piros commented on OOZIE-3160:
-

Thanks for the update [~pbacsko]! +1 for patch 006 (pending Jenkins).

Would you please also do some endurance testing, running massive number of 
coordinator / workflow actions for a few days, on a real cluster? Thanks!

> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-001.patch, OOZIE-3160-002.patch, OOZIE-3160-003.patch, 
> OOZIE-3160-004.patch, OOZIE-3160-005.patch, OOZIE-3160-006.patch, 
> OOZIE-3160-POC01.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, 
> OOZIE-3160-POC03.patch, OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, 
> PriorityDelayQueue improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67885: POC: OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[András Piros via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67885/#review208313
---


Ship it!




Ship It!

- András Piros


On Sept. 4, 2018, 3:45 p.m., Peter Bacsko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67885/
> ---
> 
> (Updated Sept. 4, 2018, 3:45 p.m.)
> 
> 
> Review request for oozie, András Piros, Peter Cseh, Kinga Marton, and Robert 
> Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See upstream JIRA for details
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
> ef8d58da5 
>   core/src/main/java/org/apache/oozie/util/PriorityDelayQueue.java 75c20698c 
>   core/src/main/resources/oozie-default.xml b69d2c9aa 
>   core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 
> 9c2a11d6f 
> 
> 
> Diff: https://reviews.apache.org/r/67885/diff/10/
> 
> 
> Testing
> ---
> 
> 1. Executed TestCallableQueueService which passed completely.
> 2. New unit tests for ASyncXCommandExecutor.
> 3. Tried it on a 3-node cluster
> 
> 
> Thanks,
> 
> Peter Bacsko
> 
>

Re: Review Request 67885: POC: OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Peter Bacsko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67885/
---

(Updated szept. 4, 2018, 3:45 du)


Review request for oozie, András Piros, Peter Cseh, Kinga Marton, and Robert 
Kanter.


Changes
---

Final changes


Repository: oozie-git


Description
---

See upstream JIRA for details


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
ef8d58da5 
  core/src/main/java/org/apache/oozie/util/PriorityDelayQueue.java 75c20698c 
  core/src/main/resources/oozie-default.xml b69d2c9aa 
  core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 
9c2a11d6f 


Diff: https://reviews.apache.org/r/67885/diff/10/

Changes: https://reviews.apache.org/r/67885/diff/9-10/


Testing
---

1. Executed TestCallableQueueService which passed completely.
2. New unit tests for ASyncXCommandExecutor.
3. Tried it on a 3-node cluster


Thanks,

Peter Bacsko

Re: Review Request 67885: POC: OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Peter Bacsko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67885/
---

(Updated szept. 4, 2018, 3:42 du)


Review request for oozie, András Piros, Peter Cseh, Kinga Marton, and Robert 
Kanter.


Repository: oozie-git


Description
---

See upstream JIRA for details


Diffs
-

  core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
ef8d58da5 
  core/src/main/java/org/apache/oozie/util/PriorityDelayQueue.java 75c20698c 
  core/src/main/resources/oozie-default.xml b69d2c9aa 
  core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 
9c2a11d6f 


Diff: https://reviews.apache.org/r/67885/diff/9/


Testing
---

1. Executed TestCallableQueueService which passed completely.
2. New unit tests for ASyncXCommandExecutor.
3. Tried it on a 3-node cluster


Thanks,

Peter Bacsko

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603213#comment-16603213
 ] 

Hadoop QA commented on OOZIE-3160:
--

PreCommit-OOZIE-Build started


> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-001.patch, OOZIE-3160-002.patch, OOZIE-3160-003.patch, 
> OOZIE-3160-004.patch, OOZIE-3160-005.patch, OOZIE-3160-006.patch, 
> OOZIE-3160-POC01.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, 
> OOZIE-3160-POC03.patch, OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, 
> PriorityDelayQueue improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Peter Bacsko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-3160:

Attachment: OOZIE-3160-006.patch

> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-001.patch, OOZIE-3160-002.patch, OOZIE-3160-003.patch, 
> OOZIE-3160-004.patch, OOZIE-3160-005.patch, OOZIE-3160-006.patch, 
> OOZIE-3160-POC01.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, 
> OOZIE-3160-POC03.patch, OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, 
> PriorityDelayQueue improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603158#comment-16603158
 ] 

Andras Piros commented on OOZIE-3160:
-

Thanks for the contribution [~pbacsko]!

The last patch 005 looks very promising, a few nits found and left as 
ReviewBoard comments.

> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-001.patch, OOZIE-3160-002.patch, OOZIE-3160-003.patch, 
> OOZIE-3160-004.patch, OOZIE-3160-005.patch, OOZIE-3160-POC01.patch, 
> OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC03.patch, 
> OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, PriorityDelayQueue 
> improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67885: POC: OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[András Piros via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67885/#review208310
---




core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java
Lines 1 (patched)


Apache 2.0 license header missing.



core/src/main/java/org/apache/oozie/service/CallableQueueService.java
Lines 562 (patched)


Typo: `by looking at the`



core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java
Lines 75 (patched)


Remove newline


- András Piros


On Sept. 4, 2018, 12:05 p.m., Peter Bacsko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67885/
> ---
> 
> (Updated Sept. 4, 2018, 12:05 p.m.)
> 
> 
> Review request for oozie, András Piros, Peter Cseh, Kinga Marton, and Robert 
> Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> See upstream JIRA for details
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
> ef8d58da5 
>   core/src/main/java/org/apache/oozie/util/PriorityDelayQueue.java 75c20698c 
>   core/src/main/resources/oozie-default.xml b69d2c9aa 
>   core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java 
> PRE-CREATION 
>   core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 
> 9c2a11d6f 
> 
> 
> Diff: https://reviews.apache.org/r/67885/diff/9/
> 
> 
> Testing
> ---
> 
> 1. Executed TestCallableQueueService which passed completely.
> 2. New unit tests for ASyncXCommandExecutor.
> 3. Tried it on a 3-node cluster
> 
> 
> Thanks,
> 
> Peter Bacsko
> 
>

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603085#comment-16603085
 ] 

Hadoop QA commented on OOZIE-3160:
--


Testing JIRA OOZIE-3160

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 2 testcase(s)
{color:red}-1 RAT{color}
.{color:red}-1{color} the patch seems to introduce 1 new RAT warning(s)
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2958
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/806/



> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-001.patch, OOZIE-3160-002.patch, OOZIE-3160-003.patch, 
> OOZIE-3160-004.patch, OOZIE-3160-005.patch, OOZIE-3160-POC01.patch, 
> OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC03.patch, 
> OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, PriorityDelayQueue 
> improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68505: OOZIE-2877 Git action

[András Piros via Review Board]

> On Aug. 31, 2018, 10:03 a.m., Andras Salamon wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java
> > Line 134 (original), 134 (patched)
> > 
> >
> > Shouldn't we register this error code in registerErrors method?

ActionExecutor#registerError() is used when a specific Exception subclass 
should always be treated given the same errorType and errorCode. This is not 
the case here.


- András


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68505/#review208163
---


On Aug. 31, 2018, 9:50 a.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68505/
> ---
> 
> (Updated Aug. 31, 2018, 9:50 a.m.)
> 
> 
> Review request for oozie, Andras Salamon, Clay B., Peter Cseh, Kinga Marton, 
> and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2877 Git action
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/git-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> b69d2c9aacc1a7f5f903fb80f77bbf5482feb3b5 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 661970d57ac038d7311856a93f5221f081e9ce15 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 
> 76cbe21ecd79eae86885a8db77831bf2c40f2d4f 
>   examples/src/main/apps/git/job.properties PRE-CREATION 
>   examples/src/main/apps/git/workflow.xml PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Git.java 
> PRE-CREATION 
>   fluent-job/fluent-job-api/pom.xml cefd43c143550a24ffba5a1d6922b8241dd5756a 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/GitAction.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/GitActionBuilder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/DistcpConfigurationConverter.java
>  fffb734e7c1b920e94f76e6a2b3062dd5aa73a86 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/ExplicitNodeConverter.java
>  7bb82e5cbbffbee02785361eb2bbde2fc1c31e7a 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/GitConfigurationConverter.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/GitPrepareConverter.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/Hive2ConfigurationConverter.java
>  c67b5aeab41e0c2d963c0ad3cf44fd19fe7c66e3 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/HiveConfigurationConverter.java
>  5f9a2b180623b89c793e8e6bb6e03d445c0ad991 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/InlineWorkflowConfigurationConverter.java
>  b1e17c9b5b8c29a81792ebfc637a53564bdb60fe 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/ShellConfigurationConverter.java
>  ab73dfd2f24cbbfa60a37d75e5057f3ff2a1c0fa 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/SparkConfigurationConverter.java
>  8827769ea84f8f08a79e5e2d0189e2d1ded4bd9d 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/SqoopConfigurationConverter.java
>  1d4f615af583f8b941e570f3e3f239979cc8f825 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/serialization/WorkflowMarshaller.java
>  ec56554ba4cd313a9d5ab8472ab3c62b6bc17380 
>   fluent-job/fluent-job-api/src/main/resources/action_mappings.xml 
> a5f890eef9ccc352a5a852781f4212d96913e6d2 
>   fluent-job/fluent-job-api/src/main/xjb/bindings.xml 
> 48f68905ac2df2818d88907b26e863326d6d2c61 
>   
> fluent-job/fluent-job-client/src/test/java/org/apache/oozie/jobs/client/minitest/TestGitAction.java
>  PRE-CREATION 
>   pom.xml e0dbe85e5238fdc11cbd44cc31f65fc768c7454d 
>   sharelib/git/pom.xml PRE-CREATION 
>   sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitOperations.java 
> PRE-CREATION 
>   sharelib/git/src/test/java/org/apache/oozie/action/hadoop/GitServer.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitMain.java 
> PRE-CREATION 
>   
> 

[jira] [Updated] (OOZIE-2877) [actions] Git action

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2877:

Summary: [actions] Git action  (was: Oozie Git Action)

> [actions] Git action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Affects Versions: 5.0.0
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: 5.1.0
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch, OOZIE-2877.013-1.patch, 
> OOZIE-2877.013.patch, OOZIE-2877.014-1.patch, OOZIE-2877.014-2.patch, 
> OOZIE-2877.014-3.patch, OOZIE-2877.015.patch, OOZIE-2877.016.patch, 
> OOZIE-2877.017.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2877) Git action

[Andras Piros (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Piros updated OOZIE-2877:

Summary: Git action  (was: [actions] Git action)

> Git action
> --
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Affects Versions: 5.0.0
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: 5.1.0
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch, OOZIE-2877.013-1.patch, 
> OOZIE-2877.013.patch, OOZIE-2877.014-1.patch, OOZIE-2877.014-2.patch, 
> OOZIE-2877.014-3.patch, OOZIE-2877.015.patch, OOZIE-2877.016.patch, 
> OOZIE-2877.017.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603050#comment-16603050
 ] 

Andras Piros commented on OOZIE-2877:
-

Thanks for the contribution [~clayb] and for the review rounds [~pbacsko] and 
[~gezapeti]!

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Affects Versions: 5.0.0
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: 5.1.0
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch, OOZIE-2877.013-1.patch, 
> OOZIE-2877.013.patch, OOZIE-2877.014-1.patch, OOZIE-2877.014-2.patch, 
> OOZIE-2877.014-3.patch, OOZIE-2877.015.patch, OOZIE-2877.016.patch, 
> OOZIE-2877.017.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3336) [persistence] Refactor entity classes to feature PK, FK, and UQ constraints

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603046#comment-16603046
 ] 

Andras Piros commented on OOZIE-3336:
-

[~pbacsko] [~asasvari] [~gezapeti] please feel free to comment / modify.

> [persistence] Refactor entity classes to feature PK, FK, and UQ constraints
> ---
>
> Key: OOZIE-3336
> URL: https://issues.apache.org/jira/browse/OOZIE-3336
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Priority: Major
> Fix For: 5.2.0
>
>
> When an Oozie database grows substantial in size, let's say, over a few 
> hundred thousands of {{WorkflowActionBean}}, {{CoordinatorActionBean}} 
> instances, we face a couple of performance issues. Here is an analysis why.
> Current Oozie JPA {{@Entity}} usage, and the resulting database DDL, suffers 
> from a couple of drawback from a performance point of view:
> * {{@Id}} fields are {{String}}:
> ** leaving no space for database primary key indices to work effectively
> ** those values are calculated in case of {{WorkflowActionBean}}, 
> {{CoordinatorActionBean}}, and {{BundleActionBean}} instances
> * no foreign constraint is set from {{WorkflowActionBean}} to 
> {{WorkflowJobBean}}, from {{CoordinatorActionBean}} to 
> {{CoordinatorJobBean}}, or from {{BundleActionBean}} to {{BundleJobBean}} 
> instances:
> ** have to assess JPA queries discovering parent-child relationships by hand
> ** no database indices are created, and hence, those queries that contain any 
> {{JOIN}} instances are slower
> * no use of unique constraints whatsoever
> * JPA queries are created by hand instead of relying on OpenJPA
> * JPA entities are filled by hand instead of relying on OpenJPA
> Following enhancements are necessary:
> # keeping the existing {{String compositeId}} fields, let's break down the 
> contents to following new fields:
> ## {{@Id long id}} - an auto-increment value that is unique across Oozie 
> database
> ## {{long currentSequence}} - the sequence number of the current run since 
> last Oozie server restart. The first part of the {{compositeId}}
> ## {{Timestamp serverStartupTimestamp}} - the timestamp when the Oozie server 
> was last started. The second part of the {{compositeId}}
> ## {{String serverName}} - the third part of the {{compositeId}}
> ## {{String name}} - the fourth and last part of the {{compositeId}}
> ## {{compositeId}} might be calculated when an entity is loaded / persisted, 
> and then stored
> # FK constraints:
> ## {{@OneToMany}} fields where we have a list of child references inside 
> parent
> ## {{@ManyToOne}} fields where we have a parent reference inside child
> ## pay attention to {{FetchType}}, most of the times {{LAZY}} will be needed
> ## the containment fields should not be {{@Transient}} anymore
> # UQ constraints:
> ## on {{currentSequence}} and {{serverStartupTimestamp}}
> ## on {{currentSequence}} and {{name}}
> # new JPQL queries:
> ## to cover changed parent-child relationships
> ## to get use of each disassembled part of {{originalId}} when doing e.g. 
> filtering
> # let JPA fill entities instead performing this by hand
> Following enhancements can be considered as nice-to-have:
> * upgrade to an OpenJPA version that features JPA 2.1's composite indexing 
> capability
> * see whether to have an optimistic locking field using {{@Version}} instead 
> of ZooKeeper based pessimistic locking would increase High Availability 
> characteristics
> * refactor also SLA related entity classes
> It's necessary to have performance benchmarks with some database types like 
> MySQL/MariaDB, and PostgreSQL before and after the changes for following use 
> cases:
> * {{CoordinatorJobBean}} and {{WorkflowJobBean}} instances up to millions
> * {{CoordinatorActionBean}} and {{WorkflowActionBean}} instances up to tens 
> of millions
> * performance for JPQLs that get a list of entities
> * performance of persisting a new entity
> * performance of querying lists of entities based on popular / possible 
> filters like the ones used by {{VxJobsServlet}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3336) [persistence] Refactor entity classes to feature PK, FK, and UQ constraints

[Andras Piros (JIRA)]

Andras Piros created OOZIE-3336:
---

 Summary: [persistence] Refactor entity classes to feature PK, FK, 
and UQ constraints
 Key: OOZIE-3336
 URL: https://issues.apache.org/jira/browse/OOZIE-3336
 Project: Oozie
  Issue Type: Improvement
  Components: core
Affects Versions: 5.0.0
Reporter: Andras Piros
 Fix For: 5.2.0


When an Oozie database grows substantial in size, let's say, over a few hundred 
thousands of {{WorkflowActionBean}}, {{CoordinatorActionBean}} instances, we 
face a couple of performance issues. Here is an analysis why.

Current Oozie JPA {{@Entity}} usage, and the resulting database DDL, suffers 
from a couple of drawback from a performance point of view:
* {{@Id}} fields are {{String}}:
** leaving no space for database primary key indices to work effectively
** those values are calculated in case of {{WorkflowActionBean}}, 
{{CoordinatorActionBean}}, and {{BundleActionBean}} instances
* no foreign constraint is set from {{WorkflowActionBean}} to 
{{WorkflowJobBean}}, from {{CoordinatorActionBean}} to {{CoordinatorJobBean}}, 
or from {{BundleActionBean}} to {{BundleJobBean}} instances:
** have to assess JPA queries discovering parent-child relationships by hand
** no database indices are created, and hence, those queries that contain any 
{{JOIN}} instances are slower
* no use of unique constraints whatsoever
* JPA queries are created by hand instead of relying on OpenJPA
* JPA entities are filled by hand instead of relying on OpenJPA

Following enhancements are necessary:
# keeping the existing {{String compositeId}} fields, let's break down the 
contents to following new fields:
## {{@Id long id}} - an auto-increment value that is unique across Oozie 
database
## {{long currentSequence}} - the sequence number of the current run since last 
Oozie server restart. The first part of the {{compositeId}}
## {{Timestamp serverStartupTimestamp}} - the timestamp when the Oozie server 
was last started. The second part of the {{compositeId}}
## {{String serverName}} - the third part of the {{compositeId}}
## {{String name}} - the fourth and last part of the {{compositeId}}
## {{compositeId}} might be calculated when an entity is loaded / persisted, 
and then stored
# FK constraints:
## {{@OneToMany}} fields where we have a list of child references inside parent
## {{@ManyToOne}} fields where we have a parent reference inside child
## pay attention to {{FetchType}}, most of the times {{LAZY}} will be needed
## the containment fields should not be {{@Transient}} anymore
# UQ constraints:
## on {{currentSequence}} and {{serverStartupTimestamp}}
## on {{currentSequence}} and {{name}}
# new JPQL queries:
## to cover changed parent-child relationships
## to get use of each disassembled part of {{originalId}} when doing e.g. 
filtering
# let JPA fill entities instead performing this by hand

Following enhancements can be considered as nice-to-have:
* upgrade to an OpenJPA version that features JPA 2.1's composite indexing 
capability
* see whether to have an optimistic locking field using {{@Version}} instead of 
ZooKeeper based pessimistic locking would increase High Availability 
characteristics
* refactor also SLA related entity classes

It's necessary to have performance benchmarks with some database types like 
MySQL/MariaDB, and PostgreSQL before and after the changes for following use 
cases:
* {{CoordinatorJobBean}} and {{WorkflowJobBean}} instances up to millions
* {{CoordinatorActionBean}} and {{WorkflowActionBean}} instances up to tens of 
millions
* performance for JPQLs that get a list of entities
* performance of persisting a new entity
* performance of querying lists of entities based on popular / possible filters 
like the ones used by {{VxJobsServlet}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603000#comment-16603000
 ] 

Peter Bacsko commented on OOZIE-2877:
-

+1 for the latest patch. 

 

I think in the current state it can be committed to master.

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Affects Versions: 5.0.0
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: 5.1.0
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch, OOZIE-2877.013-1.patch, 
> OOZIE-2877.013.patch, OOZIE-2877.014-1.patch, OOZIE-2877.014-2.patch, 
> OOZIE-2877.014-3.patch, OOZIE-2877.015.patch, OOZIE-2877.016.patch, 
> OOZIE-2877.017.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602994#comment-16602994
 ] 

Andras Piros commented on OOZIE-3061:
-

Thanks for the contribution [~matijhs]! +1

> Kill only those child jobs which are not already killed
> ---
>
> Key: OOZIE-3061
> URL: https://issues.apache.org/jira/browse/OOZIE-3061
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Satish Subhashrao Saley
>Assignee: Mate Juhasz
>Priority: Trivial
>  Labels: newbie, newbiee
> Fix For: 5.1.0
>
> Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, 
> OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch, 
> OOZIE-3061-006.patch
>
>
> Here we kill all child jobs. 
> https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265
> We should check before killing for already killed application.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68505: OOZIE-2877 Git action

[Peter Bacsko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68505/#review208306
---


Ship it!




Ship It!

- Peter Bacsko


On aug. 31, 2018, 9:50 de, András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68505/
> ---
> 
> (Updated aug. 31, 2018, 9:50 de)
> 
> 
> Review request for oozie, Andras Salamon, Clay B., Peter Cseh, Kinga Marton, 
> and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2877 Git action
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/git-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> b69d2c9aacc1a7f5f903fb80f77bbf5482feb3b5 
>   core/src/test/java/org/apache/oozie/test/XTestCase.java 
> 661970d57ac038d7311856a93f5221f081e9ce15 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 
> 76cbe21ecd79eae86885a8db77831bf2c40f2d4f 
>   examples/src/main/apps/git/job.properties PRE-CREATION 
>   examples/src/main/apps/git/workflow.xml PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Git.java 
> PRE-CREATION 
>   fluent-job/fluent-job-api/pom.xml cefd43c143550a24ffba5a1d6922b8241dd5756a 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/GitAction.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/GitActionBuilder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/DistcpConfigurationConverter.java
>  fffb734e7c1b920e94f76e6a2b3062dd5aa73a86 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/ExplicitNodeConverter.java
>  7bb82e5cbbffbee02785361eb2bbde2fc1c31e7a 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/GitConfigurationConverter.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/GitPrepareConverter.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/Hive2ConfigurationConverter.java
>  c67b5aeab41e0c2d963c0ad3cf44fd19fe7c66e3 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/HiveConfigurationConverter.java
>  5f9a2b180623b89c793e8e6bb6e03d445c0ad991 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/InlineWorkflowConfigurationConverter.java
>  b1e17c9b5b8c29a81792ebfc637a53564bdb60fe 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/ShellConfigurationConverter.java
>  ab73dfd2f24cbbfa60a37d75e5057f3ff2a1c0fa 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/SparkConfigurationConverter.java
>  8827769ea84f8f08a79e5e2d0189e2d1ded4bd9d 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/mapping/SqoopConfigurationConverter.java
>  1d4f615af583f8b941e570f3e3f239979cc8f825 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/serialization/WorkflowMarshaller.java
>  ec56554ba4cd313a9d5ab8472ab3c62b6bc17380 
>   fluent-job/fluent-job-api/src/main/resources/action_mappings.xml 
> a5f890eef9ccc352a5a852781f4212d96913e6d2 
>   fluent-job/fluent-job-api/src/main/xjb/bindings.xml 
> 48f68905ac2df2818d88907b26e863326d6d2c61 
>   
> fluent-job/fluent-job-client/src/test/java/org/apache/oozie/jobs/client/minitest/TestGitAction.java
>  PRE-CREATION 
>   pom.xml e0dbe85e5238fdc11cbd44cc31f65fc768c7454d 
>   sharelib/git/pom.xml PRE-CREATION 
>   sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitOperations.java 
> PRE-CREATION 
>   sharelib/git/src/test/java/org/apache/oozie/action/hadoop/GitServer.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/pom.xml 39cea257e750eb8b3f3c4cb3f137093fccc03016 
>   src/main/assemblies/sharelib.xml 07dc69c8276895b254e5af0cc021cce6ebad18f4 
>   webapp/pom.xml fd3f89feedd76657f1431898a268298549826d6f 
> 
> 
> Diff: https://reviews.apache.org/r/68505/diff/3/
> 
> 
> Testing
> ---
> 
> Multiple unit and integration tests, as well as tests performed on a local 
> Oozie server w/ pseudo-distributed Hadoop.
> 
> Based on the work 

[jira] [Commented] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602988#comment-16602988
 ] 

Andras Piros commented on OOZIE-3307:
-

Thanks [~vladimir.prus] for the code pointers!

[~pbacsko] sure, I'll create a patch in the next few days.

> oozie workflow gets failed throwing error virtual memory limits
> ---
>
> Key: OOZIE-3307
> URL: https://issues.apache.org/jira/browse/OOZIE-3307
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Sabir Naikwadi
>Assignee: Andras Piros
>Priority: Critical
> Fix For: 5.1.0
>
>
> Application application_1531909575787_0039 failed 2 times due to AM Container 
> for appattempt_1531909575787_0039_02 exited with exitCode: -103
>  Failing this attempt.Diagnostics: Container 
> [pid=11516,containerID=container_1531909575787_0039_02_01] is running 
> beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory 
> used; 10.0 GB of 10 GB virtual memory used. Killing container.
>  Dump of the process-tree for container_1531909575787_0039_02_01 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM 
> 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stdout
>  
> 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM
>  Container killed on request. Exit code is 143
>  Container exited with a non-zero exit code 143
>  For more detailed output, check the application tracking page: 
> [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039]
>  Then click on links to logs of each attempt.
>  . Failing the application.|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68140: OOZIE-3229 Improved filtering options in V2SLAServlet

[András Piros via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68140/#review208294
---




core/src/main/java/org/apache/oozie/FilterParser.java
Lines 49 (patched)


Use `PARAMETER_EQUALS`



core/src/main/java/org/apache/oozie/FilterParser.java
Lines 56 (patched)


`Strings.isNullOrEmpty()`



core/src/main/java/org/apache/oozie/FilterParser.java
Lines 74-78 (patched)


Here we could give a bit more context to the caller, we already have that



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Line 57 (original), 85 (patched)


Javadoc comment on this `Pattern`, plus I think the Ascii part 
(`oozie-${USER_NAME_TRUNCATED_TO_$_CHARS}`) cannot be of arbitrary length.



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 118-120 (patched)


Could be `final`



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 122 (patched)


Could be `final`



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 128 (patched)


Can be `final`



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 130 (patched)


`field.type.equals()`



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 133 (patched)


`field.type.equals()`



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 137 (patched)


`field.type.equals()`



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 213-214 (patched)


Would extract to some method like `ensureCriteriaFields()`



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 215-216 (patched)


Extract to `createSelectFrom()`



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 218 (patched)


Always `ORDER BY NOMINAL_TIME`?



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 228-239 (patched)


I'm thinking whether `fieldName LIKE '%fieldValue%'` can / should be 
supported.



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 255 (patched)


`FilterField.PARENT_ID.dbFieldName`



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 273 (patched)


Instead, code would be a bit cleaner if following would be used:
```
if (bundle == null) {
  return;
}
```



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 295 (patched)


Special handling of unparseable `event_status` value



core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
Lines 296-354 (patched)


Isn't it possible to use the Visitor design pattern instead? Or, at the 
very least, extract the `case` contents to well named methods?



core/src/test/java/org/apache/oozie/TestFilterParser.java
Lines 30 (patched)


Needn't and shouldn't `extends XTestCase`



core/src/test/java/org/apache/oozie/TestFilterParser.java
Lines 45 (patched)


Rather use `ExpectedException`



core/src/test/java/org/apache/oozie/TestFilterParser.java
Lines 47-50 (patched)


Instead of this comment, rather use a name like `ignored`



core/src/test/java/org/apache/oozie/TestFilterParser.java
Lines 57 (patched)


Rather use `ExpectedException`



core/src/test/java/org/apache/oozie/TestFilterParser.java
Lines 59-62 (patched)

Re: Review Request 67885: POC: OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Peter Bacsko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67885/
---

(Updated szept. 4, 2018, 12:05 du)


Review request for oozie, András Piros, Peter Cseh, Kinga Marton, and Robert 
Kanter.


Changes
---

Addressed comments of Sala


Repository: oozie-git


Description
---

See upstream JIRA for details


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/service/AsyncXCommandExecutor.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/service/CallableQueueService.java 
ef8d58da5 
  core/src/main/java/org/apache/oozie/util/PriorityDelayQueue.java 75c20698c 
  core/src/main/resources/oozie-default.xml b69d2c9aa 
  core/src/test/java/org/apache/oozie/service/TestAsyncXCommandExecutor.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/service/TestCallableQueueService.java 
9c2a11d6f 


Diff: https://reviews.apache.org/r/67885/diff/9/

Changes: https://reviews.apache.org/r/67885/diff/8-9/


Testing
---

1. Executed TestCallableQueueService which passed completely.
2. New unit tests for ASyncXCommandExecutor.
3. Tried it on a 3-node cluster


Thanks,

Peter Bacsko

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602943#comment-16602943
 ] 

Hadoop QA commented on OOZIE-3160:
--

PreCommit-OOZIE-Build started


> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-001.patch, OOZIE-3160-002.patch, OOZIE-3160-003.patch, 
> OOZIE-3160-004.patch, OOZIE-3160-005.patch, OOZIE-3160-POC01.patch, 
> OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC03.patch, 
> OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, PriorityDelayQueue 
> improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602944#comment-16602944
 ] 

Peter Bacsko commented on OOZIE-3160:
-

Uploaded patch v5 which contains some minor cleanup.

> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-001.patch, OOZIE-3160-002.patch, OOZIE-3160-003.patch, 
> OOZIE-3160-004.patch, OOZIE-3160-005.patch, OOZIE-3160-POC01.patch, 
> OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC03.patch, 
> OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, PriorityDelayQueue 
> improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Peter Bacsko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-3160:

Attachment: OOZIE-3160-005.patch

> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-001.patch, OOZIE-3160-002.patch, OOZIE-3160-003.patch, 
> OOZIE-3160-004.patch, OOZIE-3160-005.patch, OOZIE-3160-POC01.patch, 
> OOZIE-3160-POC02.patch, OOZIE-3160-POC02.patch, OOZIE-3160-POC03.patch, 
> OOZIE-3160-POC04.patch, OOZIE-3160-POC05.patch, PriorityDelayQueue 
> improvement - OOZIE-3160.pdf
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602855#comment-16602855
 ] 

Peter Bacsko edited comment on OOZIE-3307 at 9/4/18 10:17 AM:
--

[~andras.piros] could you take care of this?
 # Set {{-Xmx}} to be 80% of the container memory size
 # If it's already defined as a JVM option by the user, let that setting take 
precedence and don't override it (optional: we can still parse it and print a 
warning if the settings can lead to the situation described in this ticket)

Right now we use 2GB as default in the container request, to me this looks like 
a reasonable value.


was (Author: pbacsko):
[~andras.piros] could you take care of this?
 # Set {{-Xmx}} to be 80% of the container memory size
 # If it's already defined as a java-opts, let that setting take precedence

Right now we use 2GB as default in the container request, to me this looks like 
a reasonable value.

> oozie workflow gets failed throwing error virtual memory limits
> ---
>
> Key: OOZIE-3307
> URL: https://issues.apache.org/jira/browse/OOZIE-3307
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Sabir Naikwadi
>Assignee: Andras Piros
>Priority: Critical
> Fix For: 5.1.0
>
>
> Application application_1531909575787_0039 failed 2 times due to AM Container 
> for appattempt_1531909575787_0039_02 exited with exitCode: -103
>  Failing this attempt.Diagnostics: Container 
> [pid=11516,containerID=container_1531909575787_0039_02_01] is running 
> beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory 
> used; 10.0 GB of 10 GB virtual memory used. Killing container.
>  Dump of the process-tree for container_1531909575787_0039_02_01 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM 
> 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stdout
>  
> 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM
>  Container killed on request. Exit code is 143
>  Container exited with a non-zero exit code 143
>  For more detailed output, check the application tracking page: 
> [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039]
>  Then click on links to logs of each attempt.
>  . Failing the application.|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3229) Improved filtering options in V2SLAServlet

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602859#comment-16602859
 ] 

Hadoop QA commented on OOZIE-3229:
--


Testing JIRA OOZIE-3229

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 2 line(s) longer than 132 
characters
.{color:green}+1{color} the patch adds/modifies 7 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [1] new bugs found in [core] that would be 
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [webapp].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2996
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/805/



> Improved filtering options in V2SLAServlet
> --
>
> Key: OOZIE-3229
> URL: https://issues.apache.org/jira/browse/OOZIE-3229
> Project: Oozie
>  Issue Type: New Feature
>  Components: client
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Andras Piros
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3229-1.patch, OOZIE-3229-2.patch, 
> OOZIE-3229-3.patch, OOZIE-3229-4.patch, OOZIE-3229-5.patch, 
> OOZIE-3229-6.patch, OOZIE-3229-7.patch
>
>
> Currently we can apply a range of filters on top of {{V2SLAServlet}} that can 
> be used in a rich but undocumented set of ways:
> * {{id}}
> * {{parent_id}}
> * {{event_status}}
> * {{app_name}}
> * {{nominal_start}}
> * {{nominal_end}}
> Need to refactor {{V2SLAServlet}} to feature:
> * a richer set of {{SLAEvent}}, {{SLARegistration}}, and {{SLASummary}} 
> filtering based on their attributes
> * filter options will always be {{AND}}-ed, never {{OR}}-ed to each other
> * maintain compatibility with the parameter names and behavior used thus far
> * remove {{SLASummaryFilter}} and refactor 
> {{SLASummaryGetForFilterJPAExecutor}} as just another possibility for 
> confusion
> * document new functionality with rich use case / example library so that 
> users can leverage



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602855#comment-16602855
 ] 

Peter Bacsko edited comment on OOZIE-3307 at 9/4/18 10:16 AM:
--

[~andras.piros] could you take care of this?
 # Set {{-Xmx}} to be 80% of the container memory size
 # If it's already defined as a java-opts, let that setting take precedence

Right now we use 2GB as default in the container request, to me this looks like 
a reasonable value.


was (Author: pbacsko):
[~andras.piros] could you take care of this?
 # Set {{-Xmx}} to be 80% of the container memory size
 # If it's already defined as a java-opts, let that setting take precedence

Right now we use 2GB as default in the container request, to me this is a 
reasonable value.

> oozie workflow gets failed throwing error virtual memory limits
> ---
>
> Key: OOZIE-3307
> URL: https://issues.apache.org/jira/browse/OOZIE-3307
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Sabir Naikwadi
>Assignee: Andras Piros
>Priority: Critical
> Fix For: 5.1.0
>
>
> Application application_1531909575787_0039 failed 2 times due to AM Container 
> for appattempt_1531909575787_0039_02 exited with exitCode: -103
>  Failing this attempt.Diagnostics: Container 
> [pid=11516,containerID=container_1531909575787_0039_02_01] is running 
> beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory 
> used; 10.0 GB of 10 GB virtual memory used. Killing container.
>  Dump of the process-tree for container_1531909575787_0039_02_01 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM 
> 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stdout
>  
> 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM
>  Container killed on request. Exit code is 143
>  Container exited with a non-zero exit code 143
>  For more detailed output, check the application tracking page: 
> [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039]
>  Then click on links to logs of each attempt.
>  . Failing the application.|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602855#comment-16602855
 ] 

Peter Bacsko commented on OOZIE-3307:
-

[~andras.piros] could you take care of this?
 # Set {{-Xmx}} to be 80% of the container memory size
 # If it's already defined as a java-opts, let that setting take precedence

Right now we use 2GB as default in the container request, to me this is a 
reasonable value.

> oozie workflow gets failed throwing error virtual memory limits
> ---
>
> Key: OOZIE-3307
> URL: https://issues.apache.org/jira/browse/OOZIE-3307
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Sabir Naikwadi
>Assignee: Andras Piros
>Priority: Critical
>
> Application application_1531909575787_0039 failed 2 times due to AM Container 
> for appattempt_1531909575787_0039_02 exited with exitCode: -103
>  Failing this attempt.Diagnostics: Container 
> [pid=11516,containerID=container_1531909575787_0039_02_01] is running 
> beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory 
> used; 10.0 GB of 10 GB virtual memory used. Killing container.
>  Dump of the process-tree for container_1531909575787_0039_02_01 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM 
> 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stdout
>  
> 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM
>  Container killed on request. Exit code is 143
>  Container exited with a non-zero exit code 143
>  For more detailed output, check the application tracking page: 
> [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039]
>  Then click on links to logs of each attempt.
>  . Failing the application.|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602847#comment-16602847
 ] 

Peter Bacsko edited comment on OOZIE-3307 at 9/4/18 10:07 AM:
--

That's right, we do set {{-Xmx}} automatically in MR-based Oozie: 
[https://github.com/apache/oozie/blob/branch-4.3/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L382-L415]
 

It's missing from the Oozie-on-YARN based implementation. Perhaps, based on the 
resource request, we could limit {{-Xmx}} as  80-90 % of the requested memory 
to make sure that OOME happens before the container is killed by YARN.


was (Author: pbacsko):
That's right, we do set {{-Xmx}} automatically in MR-based Oozie: 
[https://github.com/apache/oozie/blob/branch-4.3/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L382-L415]

 

It's missing from the Oozie-on-YARN based implementation. Perhaps, based on the 
resource request, we could limit {{-Xmx}} as  80-90 % of the requested memory 
to make sure that OOME happens before the container is killyed by YARN.

> oozie workflow gets failed throwing error virtual memory limits
> ---
>
> Key: OOZIE-3307
> URL: https://issues.apache.org/jira/browse/OOZIE-3307
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Sabir Naikwadi
>Priority: Critical
>
> Application application_1531909575787_0039 failed 2 times due to AM Container 
> for appattempt_1531909575787_0039_02 exited with exitCode: -103
>  Failing this attempt.Diagnostics: Container 
> [pid=11516,containerID=container_1531909575787_0039_02_01] is running 
> beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory 
> used; 10.0 GB of 10 GB virtual memory used. Killing container.
>  Dump of the process-tree for container_1531909575787_0039_02_01 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM 
> 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stdout
>  
> 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM
>  Container killed on request. Exit code is 143
>  Container exited with a non-zero exit code 143
>  For more detailed output, check the application tracking page: 
> [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039]
>  Then click on links to logs of each attempt.
>  . Failing the application.|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits

[Peter Bacsko (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602847#comment-16602847
 ] 

Peter Bacsko commented on OOZIE-3307:
-

That's right, we do set {{-Xmx}} automatically in MR-based Oozie: 
[https://github.com/apache/oozie/blob/branch-4.3/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L382-L415]

 

It's missing from the Oozie-on-YARN based implementation. Perhaps, based on the 
resource request, we could limit {{-Xmx}} as  80-90 % of the requested memory 
to make sure that OOME happens before the container is killyed by YARN.

> oozie workflow gets failed throwing error virtual memory limits
> ---
>
> Key: OOZIE-3307
> URL: https://issues.apache.org/jira/browse/OOZIE-3307
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Sabir Naikwadi
>Priority: Critical
>
> Application application_1531909575787_0039 failed 2 times due to AM Container 
> for appattempt_1531909575787_0039_02 exited with exitCode: -103
>  Failing this attempt.Diagnostics: Container 
> [pid=11516,containerID=container_1531909575787_0039_02_01] is running 
> beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory 
> used; 10.0 GB of 10 GB virtual memory used. Killing container.
>  Dump of the process-tree for container_1531909575787_0039_02_01 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM 
> 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stdout
>  
> 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576 
> /usr/lib/jvm/java-openjdk/bin/java 
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true 
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
>  -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA 
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev 
> org.apache.oozie.action.hadoop.LauncherAM
>  Container killed on request. Exit code is 143
>  Container exited with a non-zero exit code 143
>  For more detailed output, check the application tracking page: 
> [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039]
>  Then click on links to logs of each attempt.
>  . Failing the application.|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3229) Improved filtering options in V2SLAServlet

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602763#comment-16602763
 ] 

Hadoop QA commented on OOZIE-3229:
--

PreCommit-OOZIE-Build started


> Improved filtering options in V2SLAServlet
> --
>
> Key: OOZIE-3229
> URL: https://issues.apache.org/jira/browse/OOZIE-3229
> Project: Oozie
>  Issue Type: New Feature
>  Components: client
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Andras Piros
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3229-1.patch, OOZIE-3229-2.patch, 
> OOZIE-3229-3.patch, OOZIE-3229-4.patch, OOZIE-3229-5.patch, 
> OOZIE-3229-6.patch, OOZIE-3229-7.patch
>
>
> Currently we can apply a range of filters on top of {{V2SLAServlet}} that can 
> be used in a rich but undocumented set of ways:
> * {{id}}
> * {{parent_id}}
> * {{event_status}}
> * {{app_name}}
> * {{nominal_start}}
> * {{nominal_end}}
> Need to refactor {{V2SLAServlet}} to feature:
> * a richer set of {{SLAEvent}}, {{SLARegistration}}, and {{SLASummary}} 
> filtering based on their attributes
> * filter options will always be {{AND}}-ed, never {{OR}}-ed to each other
> * maintain compatibility with the parameter names and behavior used thus far
> * remove {{SLASummaryFilter}} and refactor 
> {{SLASummaryGetForFilterJPAExecutor}} as just another possibility for 
> confusion
> * document new functionality with rich use case / example library so that 
> users can leverage



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68140: OOZIE-3229 Improved filtering options in V2SLAServlet

[Andras Salamon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68140/
---

(Updated Sept. 4, 2018, 8:36 a.m.)


Review request for oozie, András Piros and Kinga Marton.


Changes
---

I also separated TestV2SLAServlet into multiple classes: TestV2SLAServlet, 
TestV2SLAServletBundle, TestV2SLAServletRun


Repository: oozie-git


Description
---

Added lots of new filter fields, also refactored 
SLASummaryGetForFilterJPAExecutor class to eliminate FindBugs errors. I'm not 
sure about the filter field names.


Diffs (updated)
-

  client/src/main/java/org/apache/oozie/client/OozieClient.java 949b4532 
  core/src/main/java/org/apache/oozie/FilterParser.java PRE-CREATION 
  
core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
 b54161e9 
  core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java 3982d1e0 
  core/src/test/java/org/apache/oozie/TestFilterParser.java PRE-CREATION 
  
core/src/test/java/org/apache/oozie/executor/jpa/sla/TestSLASummaryGetForFilterJPAExecutor.java
 PRE-CREATION 
  
core/src/test/java/org/apache/oozie/executor/jpa/sla/TestSLASummaryGetForFilterJPAExecutorFilterCollection.java
 PRE-CREATION 
  core/src/test/java/org/apache/oozie/servlet/TestV2SLAServlet.java aa633225 
  core/src/test/java/org/apache/oozie/servlet/TestV2SLAServletBundle.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/servlet/TestV2SLAServletRun.java 
PRE-CREATION 
  core/src/test/java/org/apache/oozie/servlet/V2SLAServletTestCase.java 
PRE-CREATION 
  docs/src/site/twiki/DG_SLAMonitoring.twiki c91c227d 
  webapp/src/main/webapp/console/sla/css/oozie-sla.css d2f2deee 
  webapp/src/main/webapp/console/sla/js/oozie-sla.js 2ecad228 
  webapp/src/main/webapp/console/sla/oozie-sla.html e5bf6275 


Diff: https://reviews.apache.org/r/68140/diff/5/

Changes: https://reviews.apache.org/r/68140/diff/4-5/


Testing
---


Thanks,

Andras Salamon

[jira] [Updated] (OOZIE-3229) Improved filtering options in V2SLAServlet

[Andras Salamon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Salamon updated OOZIE-3229:
--
Attachment: OOZIE-3229-7.patch

> Improved filtering options in V2SLAServlet
> --
>
> Key: OOZIE-3229
> URL: https://issues.apache.org/jira/browse/OOZIE-3229
> Project: Oozie
>  Issue Type: New Feature
>  Components: client
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Andras Piros
>Assignee: Andras Salamon
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3229-1.patch, OOZIE-3229-2.patch, 
> OOZIE-3229-3.patch, OOZIE-3229-4.patch, OOZIE-3229-5.patch, 
> OOZIE-3229-6.patch, OOZIE-3229-7.patch
>
>
> Currently we can apply a range of filters on top of {{V2SLAServlet}} that can 
> be used in a rich but undocumented set of ways:
> * {{id}}
> * {{parent_id}}
> * {{event_status}}
> * {{app_name}}
> * {{nominal_start}}
> * {{nominal_end}}
> Need to refactor {{V2SLAServlet}} to feature:
> * a richer set of {{SLAEvent}}, {{SLARegistration}}, and {{SLASummary}} 
> filtering based on their attributes
> * filter options will always be {{AND}}-ed, never {{OR}}-ed to each other
> * maintain compatibility with the parameter names and behavior used thus far
> * remove {{SLASummaryFilter}} and refactor 
> {{SLASummaryGetForFilterJPAExecutor}} as just another possibility for 
> confusion
> * document new functionality with rich use case / example library so that 
> users can leverage



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68140: OOZIE-3229 Improved filtering options in V2SLAServlet

[Andras Salamon]

> On Aug. 24, 2018, 2:05 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
> > Lines 68-164 (original), 156-183 (patched)
> > 
> >
> > I feel the need to extract the creation of the `CriteriaQuery` to 
> > another, maybe nested, class - for the sake of better testability.

There are so many references to FilterField enum, that the new class would not 
be too useful here. I've created smaller methods to make the code a little bit 
easier to understand.


> On Aug. 24, 2018, 2:05 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
> > Lines 342-354 (original), 317-326 (patched)
> > 
> >
> > Part of a nested class, maybe?

I created a nested class FilterCollection to store the hashmap and the get and 
set filterfield methods. After our discussion I wanted to put the FilterField 
enum into this class, but I cannot put a static class (enum) into a non-static 
inner class.


> On Aug. 24, 2018, 2:05 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java
> > Lines 222-229 (original), 152-178 (patched)
> > 
> >
> > Part of a nested class, for better testability

I've created a FilterParser class (+unit tests for this class). Later other 
parts of oozie should also use this class (OOZIE-3335).


> On Aug. 24, 2018, 2:05 p.m., András Piros wrote:
> > webapp/src/main/webapp/console/sla/css/oozie-sla.css
> > Line 45 (original), 45 (patched)
> > 
> >
> > Did you test that on a pseudo-distributed Oozie?

Yes, I tested it when I modified the css/html/js. I'd like to test it one more 
time before commit.


> On Aug. 24, 2018, 2:05 p.m., András Piros wrote:
> > webapp/src/main/webapp/console/sla/js/oozie-sla.js
> > Lines 29-31 (original), 25-41 (patched)
> > 
> >
> > Did you test that on a pseudo-distributed Oozie?

Yes, I tested it when I modified the css/html/js. I'd like to test it one more 
time before commit.


- Andras


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68140/#review207874
---


On Aug. 23, 2018, 3:27 p.m., Andras Salamon wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68140/
> ---
> 
> (Updated Aug. 23, 2018, 3:27 p.m.)
> 
> 
> Review request for oozie, András Piros and Kinga Marton.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Added lots of new filter fields, also refactored 
> SLASummaryGetForFilterJPAExecutor class to eliminate FindBugs errors. I'm not 
> sure about the filter field names.
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 949b4532 
>   
> core/src/main/java/org/apache/oozie/executor/jpa/sla/SLASummaryGetForFilterJPAExecutor.java
>  b54161e9 
>   core/src/main/java/org/apache/oozie/servlet/V2SLAServlet.java 3982d1e0 
>   
> core/src/test/java/org/apache/oozie/executor/jpa/sla/TestSLASummaryGetForFilterJPAExecutor.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/servlet/TestV2SLAServlet.java aa633225 
>   docs/src/site/twiki/DG_SLAMonitoring.twiki c91c227d 
>   webapp/src/main/webapp/console/sla/css/oozie-sla.css d2f2deee 
>   webapp/src/main/webapp/console/sla/js/oozie-sla.js 2ecad228 
>   webapp/src/main/webapp/console/sla/oozie-sla.html e5bf6275 
> 
> 
> Diff: https://reviews.apache.org/r/68140/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andras Salamon
> 
>

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (94 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3326  Sqoop Action should support tez delegation tokens for hive-import
https://issues.apache.org/jira/browse/OOZIE-3326
OOZIE-3320  Oozie ShellAction should support absolute bash file path
https://issues.apache.org/jira/browse/OOZIE-3320
OOZIE-3319  Log SSH action callback error output
https://issues.apache.org/jira/browse/OOZIE-3319
OOZIE-3301  Update NOTICE file
https://issues.apache.org/jira/browse/OOZIE-3301
OOZIE-3298  OYA: external ID is not filled properly and failing MR job is 
treated as SUCCEEDED
https://issues.apache.org/jira/browse/OOZIE-3298
OOZIE-3277  [build] Check for star imports in patches in pre-commit
https://issues.apache.org/jira/browse/OOZIE-3277
OOZIE-3274  Remove slf4j
https://issues.apache.org/jira/browse/OOZIE-3274
OOZIE-3266  Coord action rerun support RERUN_SKIP_NODES option
https://issues.apache.org/jira/browse/OOZIE-3266
OOZIE-3265  properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to 
appear together
https://issues.apache.org/jira/browse/OOZIE-3265
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3229  Improved filtering options in V2SLAServlet
https://issues.apache.org/jira/browse/OOZIE-3229
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3194  Oozie should set proper permissions to sharelib after upload
https://issues.apache.org/jira/browse/OOZIE-3194
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3160  PriorityDelayQueue put()/take() can cause significant CPU load due 
to busy waiting
https://issues.apache.org/jira/browse/OOZIE-3160
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736