[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[
https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602262#comment-16602262
]
Hadoop QA commented on OOZIE-3061:
--
Testing JIRA OOZIE-3061
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [3] new bugs found in total that would be nice
to have fixed.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:orange}0{color} There are [3] new bugs found in [core] that would be
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [webapp].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2932
.{color:orange}Tests failed at first run:{color}
TestCoordActionsKillXCommand#testActionKillCommandDate
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/804/
> Kill only those child jobs which are not already killed
> ---
>
> Key: OOZIE-3061
> URL: https://issues.apache.org/jira/browse/OOZIE-3061
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Satish Subhashrao Saley
>Assignee: Mate Juhasz
>Priority: Trivial
> Labels: newbie, newbiee
> Fix For: 5.1.0
>
> Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch,
> OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch,
> OOZIE-3061-006.patch
>
>
> Here we kill all child jobs.
> https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265
> We should check before killing for already killed application.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602164#comment-16602164 ] Hadoop QA commented on OOZIE-3061: -- PreCommit-OOZIE-Build started > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.1.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch, > OOZIE-3061-006.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602159#comment-16602159 ] Mate Juhasz commented on OOZIE-3061: Thanks [~andras.piros], I uploaded a patch according to your comments > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.1.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch, > OOZIE-3061-006.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3061) Kill only those child jobs which are not already killed
[ https://issues.apache.org/jira/browse/OOZIE-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Juhasz updated OOZIE-3061: --- Attachment: OOZIE-3061-006.patch > Kill only those child jobs which are not already killed > --- > > Key: OOZIE-3061 > URL: https://issues.apache.org/jira/browse/OOZIE-3061 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Satish Subhashrao Saley >Assignee: Mate Juhasz >Priority: Trivial > Labels: newbie, newbiee > Fix For: 5.1.0 > > Attachments: OOZIE-3061-001.patch, OOZIE-3061-002.patch, > OOZIE-3061-003.patch, OOZIE-3061-004.patch, OOZIE-3061-005.patch, > OOZIE-3061-006.patch > > > Here we kill all child jobs. > https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L265 > We should check before killing for already killed application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits
[
https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602048#comment-16602048
]
Vladimir Prus commented on OOZIE-3307:
--
I am not sure where Ozzie sets the options, I was looking at the processes with
"ps". E.g. I have Ooozie 4, its starts a launcher as
container_1532455670006_41980_01_01, the first line of the log is "Oozie
Launcher starts", and "ps" output is this:
{code:java}
yarn 17649 17647 0 10:18 ? 00:00:00 /bin/bash -c
/usr/lib/jvm/java-openjdk/bin/java
-Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1532455670006_41980/container_1532455670006_41980_01_01/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1532455670006_41980/container_1532455670006_41980_01_01
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
-Dhadoop.root.logfile=syslog -Xmx4685m -Xmx200m -Xmx2342m -Xmx5197m
-Djava.io.tmpdir=./tmp org.apache.hadoop.mapreduce.v2.app.MRAppMaster
1>/var/log/hadoop-yarn/containers/application_1532455670006_41980/container_1532455670006_41980_01_01/stdout
2>/var/log/hadoop-yarn/containers/application_1532455670006_41980/container_1532455670006_41980_01_01/stderr{code}
Doing the same on cluster with Oozie 5 does not show any -Xmx options.
The code that seems relevant is at
[https://github.com/apache/oozie/blob/3d0843260199799be634dc330d98f1362536fcd5/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L402]
This code appears removed in master. setJavaOpts will only set options provided
in element explicitly, and I see no code that would set Xmx
automatically.
> oozie workflow gets failed throwing error virtual memory limits
> ---
>
> Key: OOZIE-3307
> URL: https://issues.apache.org/jira/browse/OOZIE-3307
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Sabir Naikwadi
>Priority: Critical
>
> Application application_1531909575787_0039 failed 2 times due to AM Container
> for appattempt_1531909575787_0039_02 exited with exitCode: -103
> Failing this attempt.Diagnostics: Container
> [pid=11516,containerID=container_1531909575787_0039_02_01] is running
> beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory
> used; 10.0 GB of 10 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1531909575787_0039_02_01 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c
> /usr/lib/jvm/java-openjdk/bin/java
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
> -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev
> org.apache.oozie.action.hadoop.LauncherAM
> 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stdout
>
> 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576
> /usr/lib/jvm/java-openjdk/bin/java
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
> -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev
> org.apache.oozie.action.hadoop.LauncherAM
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
> For more detailed output, check the application tracking page:
> [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039]
> Then click on links to logs of each attempt.
> . Failing the application.|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits
[
https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601995#comment-16601995
]
Andras Piros commented on OOZIE-3307:
-
[~vladimir.prus] the issue you mention might be the root cause for the original
one.
Can you please point to where Oozie 4.3.0 sets {{-Xmx=1024m}} for
{{LauncherMapper}}? Thanks!
> oozie workflow gets failed throwing error virtual memory limits
> ---
>
> Key: OOZIE-3307
> URL: https://issues.apache.org/jira/browse/OOZIE-3307
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Sabir Naikwadi
>Priority: Critical
>
> Application application_1531909575787_0039 failed 2 times due to AM Container
> for appattempt_1531909575787_0039_02 exited with exitCode: -103
> Failing this attempt.Diagnostics: Container
> [pid=11516,containerID=container_1531909575787_0039_02_01] is running
> beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory
> used; 10.0 GB of 10 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1531909575787_0039_02_01 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c
> /usr/lib/jvm/java-openjdk/bin/java
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
> -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev
> org.apache.oozie.action.hadoop.LauncherAM
> 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stdout
>
> 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576
> /usr/lib/jvm/java-openjdk/bin/java
> -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true
> -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01
> -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA
> -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev
> org.apache.oozie.action.hadoop.LauncherAM
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
> For more detailed output, check the application tracking page:
> [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039]
> Then click on links to logs of each attempt.
> . Failing the application.|
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits
[ https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros updated OOZIE-3307: Affects Version/s: 5.0.0 > oozie workflow gets failed throwing error virtual memory limits > --- > > Key: OOZIE-3307 > URL: https://issues.apache.org/jira/browse/OOZIE-3307 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Sabir Naikwadi >Priority: Critical > > Application application_1531909575787_0039 failed 2 times due to AM Container > for appattempt_1531909575787_0039_02 exited with exitCode: -103 > Failing this attempt.Diagnostics: Container > [pid=11516,containerID=container_1531909575787_0039_02_01] is running > beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory > used; 10.0 GB of 10 GB virtual memory used. Killing container. > Dump of the process-tree for container_1531909575787_0039_02_01 : > | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE| > | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c > /usr/lib/jvm/java-openjdk/bin/java > -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true > -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01 > -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA > -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev > org.apache.oozie.action.hadoop.LauncherAM > 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stdout > > 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01/stderr| > | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576 > /usr/lib/jvm/java-openjdk/bin/java > -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true > -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_01 > -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA > -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev > org.apache.oozie.action.hadoop.LauncherAM > Container killed on request. Exit code is 143 > Container exited with a non-zero exit code 143 > For more detailed output, check the application tracking page: > [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039] > Then click on links to logs of each attempt. > . Failing the application.| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
[
https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601954#comment-16601954
]
Hadoop QA commented on OOZIE-3298:
--
Testing JIRA OOZIE-3298
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 6 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
.{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [1] new bugs found in total that would be nice
to have fixed.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:orange}0{color} There are [1] new bugs found in [core] that would be
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2943
.{color:orange}Tests failed at first run:{color}
TestCoordActionInputCheckXCommand#testNone
TestCoordActionsKillXCommand#testActionKillCommandDate
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/803/
> OYA: external ID is not filled properly and failing MR job is treated as
> SUCCEEDED
> --
>
> Key: OOZIE-3298
> URL: https://issues.apache.org/jira/browse/OOZIE-3298
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch,
> OOZIE-3298.003.patch, OOZIE-3298.004.patch, OOZIE-3298.005.patch,
> OOZIE-3298.007.patch, OOZIE-3298.008.patch, OOZIE-3298.009.patch,
> OOZIE-3298.010.patch, OOZIE-3298.011.patch
>
>
> When a MapReduce action is launched from Oozie (OYA), we don't properly fill
> the external ID field. It gets populated with the YARN id of the LauncherAM,
> not with the id of the actual MR job. If the MR job is succesfully submitted
> then fails, it will be treated as a successfully executed action, which is
> very misleading and can potentially confuse Oozie users.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 68237: OOZIE-3298 OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68237/ --- (Updated Sept. 3, 2018, 9:10 a.m.) Review request for oozie, Andras Salamon, Attila Sasvari, Kinga Marton, and Peter Bacsko. Repository: oozie-git Description --- OOZIE-3298 OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED Diffs - core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 8f0f244013932476d8ae454d224f235948529b34 core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 83a23f5220aa72ba15edc8b98ef80a74213fcee8 core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java f39bba2c691435354dac6da7794e5142b511d937 core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java a31079a41d30677d35a253a4a69505c21aa585f6 core/src/test/java/org/apache/oozie/action/hadoop/TestYarnApplicationIdComparator.java PRE-CREATION core/src/test/java/org/apache/oozie/action/hadoop/TestYarnApplicationIdFinder.java PRE-CREATION core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 893405e38ad71d22460768b76ed03ac4d9a0b95d sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java f460b6bd11f60dfb397c6bba82be1427c2d1b570 Diff: https://reviews.apache.org/r/68237/diff/8/ Testing (updated) --- Tested on a real cluster, plus added test cases to `TestMapReduceActionExecutor` and new test classes. Thanks, András Piros
[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
[ https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601855#comment-16601855 ] Hadoop QA commented on OOZIE-3298: -- PreCommit-OOZIE-Build started > OYA: external ID is not filled properly and failing MR job is treated as > SUCCEEDED > -- > > Key: OOZIE-3298 > URL: https://issues.apache.org/jira/browse/OOZIE-3298 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Peter Bacsko >Assignee: Andras Piros >Priority: Blocker > Fix For: 5.1.0 > > Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch, > OOZIE-3298.003.patch, OOZIE-3298.004.patch, OOZIE-3298.005.patch, > OOZIE-3298.007.patch, OOZIE-3298.008.patch, OOZIE-3298.009.patch, > OOZIE-3298.010.patch, OOZIE-3298.011.patch > > > When a MapReduce action is launched from Oozie (OYA), we don't properly fill > the external ID field. It gets populated with the YARN id of the LauncherAM, > not with the id of the actual MR job. If the MR job is succesfully submitted > then fails, it will be treated as a successfully executed action, which is > very misleading and can potentially confuse Oozie users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
[
https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601854#comment-16601854
]
Andras Piros commented on OOZIE-3298:
-
More comments on {{MapReduceActionExecutor}} and
{{TestMapReduceActionExecutor}} classes.
> OYA: external ID is not filled properly and failing MR job is treated as
> SUCCEEDED
> --
>
> Key: OOZIE-3298
> URL: https://issues.apache.org/jira/browse/OOZIE-3298
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch,
> OOZIE-3298.003.patch, OOZIE-3298.004.patch, OOZIE-3298.005.patch,
> OOZIE-3298.007.patch, OOZIE-3298.008.patch, OOZIE-3298.009.patch,
> OOZIE-3298.010.patch, OOZIE-3298.011.patch
>
>
> When a MapReduce action is launched from Oozie (OYA), we don't properly fill
> the external ID field. It gets populated with the YARN id of the LauncherAM,
> not with the id of the actual MR job. If the MR job is succesfully submitted
> then fails, it will be treated as a successfully executed action, which is
> very misleading and can potentially confuse Oozie users.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 68237: OOZIE-3298 OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68237/ --- (Updated Sept. 3, 2018, 7:59 a.m.) Review request for oozie, Andras Salamon, Attila Sasvari, Kinga Marton, and Peter Bacsko. Changes --- More comments on `MapReduceActionExecutor` and `TestMapReduceActionExecutor` classes. Repository: oozie-git Description --- OOZIE-3298 OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED Diffs (updated) - core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 8f0f244013932476d8ae454d224f235948529b34 core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 83a23f5220aa72ba15edc8b98ef80a74213fcee8 core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java f39bba2c691435354dac6da7794e5142b511d937 core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java a31079a41d30677d35a253a4a69505c21aa585f6 core/src/test/java/org/apache/oozie/action/hadoop/TestYarnApplicationIdComparator.java PRE-CREATION core/src/test/java/org/apache/oozie/action/hadoop/TestYarnApplicationIdFinder.java PRE-CREATION core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 893405e38ad71d22460768b76ed03ac4d9a0b95d sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java f460b6bd11f60dfb397c6bba82be1427c2d1b570 Diff: https://reviews.apache.org/r/68237/diff/8/ Changes: https://reviews.apache.org/r/68237/diff/7-8/ Testing --- Tested on a real cluster, plus added test cases to `TestMapReduceActionExecutor` and new test classes. Please note that I could find no proper way of actually getting MapReduce to start a new job while using `OozieClient#submit()` - apparently no YARN child application is created. Please advise what's the best way to advance, maybe call `JobClient#submitJob()` directly from `ActionExecutorTestCase#startWorkflowAndFailChildMRJob()`. Thanks, András Piros
[jira] [Updated] (OOZIE-3298) OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED
[ https://issues.apache.org/jira/browse/OOZIE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros updated OOZIE-3298: Attachment: OOZIE-3298.011.patch > OYA: external ID is not filled properly and failing MR job is treated as > SUCCEEDED > -- > > Key: OOZIE-3298 > URL: https://issues.apache.org/jira/browse/OOZIE-3298 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0 >Reporter: Peter Bacsko >Assignee: Andras Piros >Priority: Blocker > Fix For: 5.1.0 > > Attachments: OOZIE-3298.001.patch, OOZIE-3298.002.patch, > OOZIE-3298.003.patch, OOZIE-3298.004.patch, OOZIE-3298.005.patch, > OOZIE-3298.007.patch, OOZIE-3298.008.patch, OOZIE-3298.009.patch, > OOZIE-3298.010.patch, OOZIE-3298.011.patch > > > When a MapReduce action is launched from Oozie (OYA), we don't properly fill > the external ID field. It gets populated with the YARN id of the LauncherAM, > not with the id of the actual MR job. If the MR job is succesfully submitted > then fails, it will be treated as a successfully executed action, which is > very misleading and can potentially confuse Oozie users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OOZIE-3335) Cleanup parseFilter methods
Andras Salamon created OOZIE-3335:
-
Summary: Cleanup parseFilter methods
Key: OOZIE-3335
URL: https://issues.apache.org/jira/browse/OOZIE-3335
Project: Oozie
Issue Type: Improvement
Components: core
Affects Versions: trunk
Reporter: Andras Salamon
Assignee: Andras Salamon
The {{parseFilter}} method is implemented several times in oozie:
{noformat}servlet/SLAServlet.java
util/JobsFilterUtils.java
DagEngine.java
{noformat}
The implementations are quite similar. Code duplication should be eliminated.
[OOZIE-3229|https://issues.apache.org/jira/browse/OOZIE-3229] will introduce a
new utility class for filter parsing, all the other classes should use it as
well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (OOZIE-3334) Don't use org.apache.hadoop.hbase.security.User in HDFSCredentials
[ https://issues.apache.org/jira/browse/OOZIE-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon reassigned OOZIE-3334: - Assignee: Peter Cseh (was: Andras Salamon) > Don't use org.apache.hadoop.hbase.security.User in HDFSCredentials > -- > > Key: OOZIE-3334 > URL: https://issues.apache.org/jira/browse/OOZIE-3334 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0b1 >Reporter: Peter Cseh >Assignee: Peter Cseh >Priority: Major > Fix For: trunk > > Attachments: OOZIE-3334.01.patch, OOZIE-3334.02.patch > > > HDFSCredentials is using org.apache.hadoop.hbase.security.User to obtain > delegation tokens. > It introduces a runtime dependency on HBase which is not required in Oozie > core. > It was reported on GitHub: > https://github.com/apache/oozie/commit/c29d9c5fce27f5f5335662ce2649387258f1c7fd#commitcomment-30224511 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OOZIE-3334) Don't use org.apache.hadoop.hbase.security.User in HDFSCredentials
[ https://issues.apache.org/jira/browse/OOZIE-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon reassigned OOZIE-3334: - Assignee: Andras Salamon (was: Peter Cseh) > Don't use org.apache.hadoop.hbase.security.User in HDFSCredentials > -- > > Key: OOZIE-3334 > URL: https://issues.apache.org/jira/browse/OOZIE-3334 > Project: Oozie > Issue Type: Bug >Affects Versions: 5.0.0b1 >Reporter: Peter Cseh >Assignee: Andras Salamon >Priority: Major > Fix For: trunk > > Attachments: OOZIE-3334.01.patch, OOZIE-3334.02.patch > > > HDFSCredentials is using org.apache.hadoop.hbase.security.User to obtain > delegation tokens. > It introduces a runtime dependency on HBase which is not required in Oozie > core. > It was reported on GitHub: > https://github.com/apache/oozie/commit/c29d9c5fce27f5f5335662ce2649387258f1c7fd#commitcomment-30224511 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (94 issues) Subscriber: ooziedaily Key Summary OOZIE-3326 Sqoop Action should support tez delegation tokens for hive-import https://issues.apache.org/jira/browse/OOZIE-3326 OOZIE-3320 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3320 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3298 OYA: external ID is not filled properly and failing MR job is treated as SUCCEEDED https://issues.apache.org/jira/browse/OOZIE-3298 OOZIE-3277 [build] Check for star imports in patches in pre-commit https://issues.apache.org/jira/browse/OOZIE-3277 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3265 properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to appear together https://issues.apache.org/jira/browse/OOZIE-3265 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3249 [tools] Instrumentation log parser https://issues.apache.org/jira/browse/OOZIE-3249 OOZIE-3229 Improved filtering options in V2SLAServlet https://issues.apache.org/jira/browse/OOZIE-3229 OOZIE-3218 Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space https://issues.apache.org/jira/browse/OOZIE-3218 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3194 Oozie should set proper permissions to sharelib after upload https://issues.apache.org/jira/browse/OOZIE-3194 OOZIE-3186 Oozie is unable to use configuration linked using jceks://file/... https://issues.apache.org/jira/browse/OOZIE-3186 OOZIE-3179 Adding a configurable config-default.xml location to a workflow https://issues.apache.org/jira/browse/OOZIE-3179 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting https://issues.apache.org/jira/browse/OOZIE-3160 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-3061 Kill only those child jobs which are not already killed https://issues.apache.org/jira/browse/OOZIE-3061 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2949 Escape quotes whitespaces in Sqoop field https://issues.apache.org/jira/browse/OOZIE-2949 OOZIE-2927 Append new line character for Hive2 query using query tag https://issues.apache.org/jira/browse/OOZIE-2927 OOZIE-2877 Oozie Git Action https://issues.apache.org/jira/browse/OOZIE-2877 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2833 when using uber mode the regex pattern used in the extractHeapSizeMB method does not allow heap sizes specified in bytes. https://issues.apache.org/jira/browse/OOZIE-2833 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator https://issues.apache.org/jira/browse/OOZIE-2784 OOZIE-2779 Mask Hive2 action Beeline JDBC password https://issues.apache.org/jira/browse/OOZIE-2779 OOZIE-2736 Reduce the number of threads during test execution https://issues.apache.org/jira/browse/OOZIE-2736
