[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118333#comment-17118333 ] Tarun Parimi commented on MAPREDUCE-7278: - Thanks [~wilfreds] for the review and commit. I think we should pull it in branch-3.1 as well since it applies cleanly and MAPREDUCE-6485 is present there. > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, MAPREDUCE-7278.004.patch, Screen Shot 2020-04-30 at > 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118275#comment-17118275 ] Hudson commented on MAPREDUCE-7278: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18303 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18303/]) MAPREDUCE-7278. Speculative execution behavior is observed even when (wilfreds: rev 10db97df1c8562a9e29b00e60d5bde1773c09188) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, MAPREDUCE-7278.004.patch, Screen Shot 2020-04-30 at > 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118257#comment-17118257 ] Wilfred Spiegelenburg commented on MAPREDUCE-7278: -- Change is looking good +1, committing to trunk > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, MAPREDUCE-7278.004.patch, Screen Shot 2020-04-30 at > 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113112#comment-17113112
]
Hadoop QA commented on MAPREDUCE-7278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
44s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client:
The patch generated 0 new + 122 unchanged - 1 fixed = 122 total (was 123)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
12s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}128m
28s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
47s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}201m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7785/artifact/out/Dockerfile
|
| JIRA Issue | MAPREDUCE-7278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13003621/MAPREDUCE-7278.004.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112923#comment-17112923
]
Hadoop QA commented on MAPREDUCE-7278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
44s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client:
The patch generated 0 new + 122 unchanged - 1 fixed = 122 total (was 123)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
9s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 19s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.TestLargeSort |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7783/artifact/out/Dockerfile
|
| JIRA Issue | MAPREDUCE-7278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13003602/MAPREDUCE-7278.004.patch
|
| Optional Tests | dupname
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112908#comment-17112908 ] Tarun Parimi commented on MAPREDUCE-7278: - Reattaching patch as jenkins job had problems. > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, MAPREDUCE-7278.004.patch, Screen Shot 2020-04-30 at > 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112815#comment-17112815 ] Tarun Parimi commented on MAPREDUCE-7278: - Thanks [~wilfreds]. Fixed the checkstyle issues in test case related to the patch. > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, MAPREDUCE-7278.004.patch, Screen Shot 2020-04-30 at > 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112750#comment-17112750 ] Wilfred Spiegelenburg commented on MAPREDUCE-7278: -- I agree the asf license is not from this patch. Can you please fix the checkstyle issues? Code looks good no comments on that > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, Screen Shot 2020-04-30 at 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111787#comment-17111787 ] Tarun Parimi commented on MAPREDUCE-7278: - The -1 asflicense seems to be reported in MAPREDUCE-7036 which was reverted. I think we can ignore this. [~wilfreds], [~prabhujoseph] Please take a look at the patch when you get some time. > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, Screen Shot 2020-04-30 at 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1769#comment-1769
]
Hadoop QA commented on MAPREDUCE-7278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
45s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 11 new +
123 unchanged - 0 fixed = 134 total (was 123) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
22s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}129m
37s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
52s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}205m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7782/artifact/out/Dockerfile
|
| JIRA Issue | MAPREDUCE-7278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13003375/MAPREDUCE-7278.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111022#comment-17111022
]
Tarun Parimi commented on MAPREDUCE-7278:
-
The mvn install error seems to be completely unrelated to this patch.
[~wilfreds], [~prabhujoseph] Can you please have a look when you get time?
{code:java}
[INFO] --- exec-maven-plugin:1.3.1:exec (check-jar-contents) @
hadoop-client-check-invariants ---
[ERROR] Found artifact with unexpected contents:
'/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/sourcedir/hadoop-client-modules/hadoop-client-runtime/target/hadoop-client-runtime-3.4.0-SNAPSHOT.jar'
Please check the following and either correct the build or update
the allowed list with reasoning.
microsoft/
microsoft/sql/
microsoft/sql/DateTimeOffset$1.class
microsoft/sql/DateTimeOffset$SerializationProxy.class
microsoft/sql/DateTimeOffset.class
microsoft/sql/Types.class
{code}
> Speculative execution behavior is observed even when
> mapreduce.map.speculative and mapreduce.reduce.speculative are false
> -
>
> Key: MAPREDUCE-7278
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
>Affects Versions: 2.8.0, 3.4.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch,
> MAPREDUCE-7278.003.patch, Screen Shot 2020-04-30 at 8.04.27 PM.png
>
>
> When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER
> state for some time, we observe two task attempts are launched simultaneously
> even when speculative execution is disabled.
> This results in the below message shown in the killed attempts, indicating
> speculation has occurred. This is an issue for jobs which require speculative
> execution to be strictly disabled.
> !Screen Shot 2020-04-30 at 8.04.27 PM.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108239#comment-17108239
]
Hadoop QA commented on MAPREDUCE-7278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 20m
39s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 16m
37s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
46s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 11 new +
123 unchanged - 0 fixed = 134 total (was 123) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 14m
3s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
22s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}130m
30s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
48s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}227m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7781/artifact/out/Dockerfile
|
| JIRA Issue | MAPREDUCE-7278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13003012/MAPREDUCE-7278.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108091#comment-17108091 ] Tarun Parimi commented on MAPREDUCE-7278: - Reattaching same patch to kick in jenkins again > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, Screen Shot 2020-04-30 at 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107930#comment-17107930 ] Tarun Parimi commented on MAPREDUCE-7278: - The mvninstall and asflicense errors are completely unrelated to the patch. > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, Screen Shot 2020-04-30 at 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107699#comment-17107699
]
Hadoop QA commented on MAPREDUCE-7278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 20m
2s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 17m
2s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
43s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 11 new +
123 unchanged - 0 fixed = 134 total (was 123) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 14m
13s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
11s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}131m
38s{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
45s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}206m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7780/artifact/out/Dockerfile
|
| JIRA Issue | MAPREDUCE-7278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13002972/MAPREDUCE-7278.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107529#comment-17107529 ] Tarun Parimi commented on MAPREDUCE-7278: - Thanks a lot [~wilfreds] . Based on your suggestion, I have added a test in TestSpeculativeExecution . I have checked that this test fails when the fix is not present and reproduces the behavior. Please review when you get time. > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, Screen Shot 2020-04-30 at 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105859#comment-17105859
]
Wilfred Spiegelenburg commented on MAPREDUCE-7278:
--
[~tarunparimi] I have added you to the contributor list for the project and
assigned the jira to you.
The test you describe I think can be converted into a test using a
{{MiniMRYarnCluster}} like what is done in the TestSpeculativeExecution. Did
you look at that possibility?
> Speculative execution behavior is observed even when
> mapreduce.map.speculative and mapreduce.reduce.speculative are false
> -
>
> Key: MAPREDUCE-7278
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
>Affects Versions: 2.8.0, 3.4.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch,
> Screen Shot 2020-04-30 at 8.04.27 PM.png
>
>
> When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER
> state for some time, we observe two task attempts are launched simultaneously
> even when speculative execution is disabled.
> This results in the below message shown in the killed attempts, indicating
> speculation has occurred. This is an issue for jobs which require speculative
> execution to be strictly disabled.
> !Screen Shot 2020-04-30 at 8.04.27 PM.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101867#comment-17101867
]
Hadoop QA commented on MAPREDUCE-7278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
3m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
57s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
10s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7778/artifact/out/Dockerfile
|
| JIRA Issue | MAPREDUCE-7278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13002311/MAPREDUCE-7278.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 58050a385ef6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 192cad9ee24 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7778/testReport/ |
| Max. process+thread count | 707 (vs.
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101800#comment-17101800 ] Tarun Parimi commented on MAPREDUCE-7278: - Adding new patch to fix unit test failure. Not sure how to add new unit test to cover this since this patch ensures the original behavior with speculation disabled is achieved > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0 >Reporter: Tarun Parimi >Priority: Major > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > Screen Shot 2020-04-30 at 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096814#comment-17096814
]
Hadoop QA commented on MAPREDUCE-7278:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
58s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 10s{color}
| {color:red} hadoop-mapreduce-client-app in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.app.job.impl.TestTaskImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build//artifact/out/Dockerfile
|
| JIRA Issue | MAPREDUCE-7278 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13001756/MAPREDUCE-7278.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d6bb576d06d8 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6bdab3723ef |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| unit |
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096678#comment-17096678
]
Tarun Parimi commented on MAPREDUCE-7278:
-
I am able to reproduce this consistently in trunk. To trigger the issue, we
need a failing taskattempt to transition to FAIL_CONTAINER_CLEANUP. This occurs
if it is timed out while finishing. To ensure it always times out in a local
cluster, we can set
{{mapreduce.task.exit.timeout}} and
{{mapreduce.task.exit.timeout.check-interval-ms}} to extremely low values.
The below hadoop streaming job always reproduces the issue.
{code:java}
yarn jar
$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.4.0-SNAPSHOT.jar
-Dmapreduce.task.exit.timeout=20
-Dmapreduce.task.exit.timeout.check-interval-ms=10 -mapper /bin/failonce.sh
-reducer /bin/wc -input /sample -output /tmp/output
{code}
The contents of /bin/failonce.sh are as below. It only fails for the 0th
attempt and succeeds for other attempts.
{code:java}
#!/bin/bash
atmpt=`echo $mapred_task_id | awk -F_ '{print $NF}'`
if [[ $atmpt -eq 0 ]]; then exit 1; fi
{code}
> Speculative execution behavior is observed even when
> mapreduce.map.speculative and mapreduce.reduce.speculative are false
> -
>
> Key: MAPREDUCE-7278
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
>Affects Versions: 2.8.0
>Reporter: Tarun Parimi
>Priority: Major
> Attachments: Screen Shot 2020-04-30 at 8.04.27 PM.png
>
>
> When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER
> state for some time, we observe two task attempts are launched simultaneously
> even when speculative execution is disabled.
> This results in the below message shown in the killed attempts, indicating
> speculation has occurred. This is an issue for jobs which require speculative
> execution to be strictly disabled.
> !Screen Shot 2020-04-30 at 8.04.27 PM.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[
https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096650#comment-17096650
]
Tarun Parimi commented on MAPREDUCE-7278:
-
Mutiple task attempts are tried simultaneously for a single attempt failure.
This is because {{TaskAttemptImpl#notifyTaskAttemptFailed()}} is called twice,
when TaskAttempt transitions from,
1. RUNNING -> FAIL_FINISHING_CONTAINER
2. FAIL_CONTAINER_CLEANUP -> FAIL_TASK_CLEANUP -> FAILED
But this issue is seen only after the MAPREDUCE-6485 code change, since it
allows multiple new taskattempts to launch even if there are existing
taskattempts for that task. It checks whether a container is assigned, but in a
fully utilized cluster container assignment can take a few minutes and so we
can have two taskattempts launched when the above transitions occur.
> Speculative execution behavior is observed even when
> mapreduce.map.speculative and mapreduce.reduce.speculative are false
> -
>
> Key: MAPREDUCE-7278
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
>Affects Versions: 2.8.0
>Reporter: Tarun Parimi
>Priority: Major
> Attachments: Screen Shot 2020-04-30 at 8.04.27 PM.png
>
>
> When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER
> state for some time, we observe two task attempts are launched simultaneously
> even when speculative execution is disabled.
> This results in the below message shown in the killed attempts, indicating
> speculation has occurred. This is an issue for jobs which require speculative
> execution to be strictly disabled.
> !Screen Shot 2020-04-30 at 8.04.27 PM.png!
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
