[jira] [Commented] (TEZ-4189) Support fault tolerance in pipelined data transfer for unordered output
[
https://issues.apache.org/jira/browse/TEZ-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116865#comment-17116865
]
Gopal Vijayaraghavan commented on TEZ-4189:
---
bq. because fault tolerance wasn't implemented for pipelined shuffle
A failure tolerance model was working in JDK7 and stopped working with JDK8,
because we routinely use HashMap implementations in our operations to aggregate
things in memory, then read them out into a shuffle output.
http://openjdk.java.net/jeps/180
prevents something like a group-by hash in memory from returning the rows in
the same order.
So the 2nd attempt of the same task might not result in the same rows landing
in spill 1, spill 2 etc.
Requiring us to restart the process, since we cannot restart a reducer from a
fixed spil-id on a second attempt of a mapper & if we've already processed a
spill for the reducer, we can't undo it.
> Support fault tolerance in pipelined data transfer for unordered output
> ---
>
> Key: TEZ-4189
> URL: https://issues.apache.org/jira/browse/TEZ-4189
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: syslog_attempt_1589892856650_0073_1_07_000100_0,
> syslog_attempt_1589892856650_0073_1_07_000100_1,
> syslog_attempt_1589892856650_0073_1_08_000945_0
>
>
> Attached an example:
> Mapper task attempt 0 fails while spilling spill3:
> [^syslog_attempt_1589892856650_0073_1_07_000100_0]
> {code}
> 2020-05-25 13:08:10,702 [INFO] [UnorderedOutSpiller {Reducer_14} #0]
> |writers.UnorderedPartitionedKVWriter|: Writing spill 3 to
> /grid/2/yarn/nm/usercache/root/appcache/application_1589892856650_0073/output/attempt_1589892856650_0073_1_07_000100_0_10003_3/file.out
> 2020-05-25 13:08:10,736 [INFO] [TaskHeartbeatThread] |task.TaskReporter|:
> Received should die response from AM
> {code}
> Mapper task attempt 1 passes, successfully spills out 7 spills:
> [^syslog_attempt_1589892856650_0073_1_07_000100_1]
> {code}
> 2020-05-25 13:09:47,722 [INFO] [TezChild]
> |writers.UnorderedPartitionedKVWriter|: Reducer_14: Adding spill event for
> spill (final update=true), spillId=7
> {code}
> Reducer tasks get killed because they cannot recover from the mapper task
> attempt failure, because fault tolerance wasn't implemented for pipelined
> shuffle I guess: [^syslog_attempt_1589892856650_0073_1_08_000945_0]
> {code}
> 2020-05-25 13:13:14,617 [ERROR] [ShuffleRunner {Map_16}]
> |impl.ShuffleManager|: Killing self as previous attempt data could have been
> consumed
> java.io.IOException: Previous event already got scheduled for
> InputAttemptIdentifier [inputIdentifier=100, attemptNumber=1,
> pathComponent=attempt_1589892856650_0073_1_07_000100_1_10017_0, spillType=1,
> spillId=0]. Previous attempt's data could have been already merged to
> memory/disk outputs. Killing (self) this task early. currentAttemptNum=0,
> eventsProcessed={0, 1}, scheduledForDownload=true, newAttemptNum=1
> at
> org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager.validateInputAttemptForPipelinedShuffle(ShuffleManager.java:503)
> at
> org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager.constructFetcherForHost(ShuffleManager.java:551)
> at
> org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager$RunShuffleCallable.callInternal(ShuffleManager.java:455)
> at
> org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager$RunShuffleCallable.callInternal(ShuffleManager.java:398)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> The interesting part is that the above failing reducer task attempt runs 4
> minutes later than the successful, second mapper task attempt, so there is
> not case of unfortunate scheduling I think, rather the pipelined shuffle is
> not prepared for fetching in case of killed and reattempted inputs. For
> example, in the reducer task, it tries to fetch input from
> attempt_1589892856650_0073_1_07_000100_0
> 3_0, which is the map of the first, failing attempt, but this task start 4
> minutes after
[jira] [Updated] (TEZ-4189) Support fault tolerance in pipelined data transfer for unordered output
[
https://issues.apache.org/jira/browse/TEZ-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4189:
--
Description:
Attached an example:
Mapper task attempt 0 fails while spilling spill3:
[^syslog_attempt_1589892856650_0073_1_07_000100_0]
{code}
2020-05-25 13:08:10,702 [INFO] [UnorderedOutSpiller {Reducer_14} #0]
|writers.UnorderedPartitionedKVWriter|: Writing spill 3 to
/grid/2/yarn/nm/usercache/root/appcache/application_1589892856650_0073/output/attempt_1589892856650_0073_1_07_000100_0_10003_3/file.out
2020-05-25 13:08:10,736 [INFO] [TaskHeartbeatThread] |task.TaskReporter|:
Received should die response from AM
{code}
Mapper task attempt 1 passes, successfully spills out 7 spills:
[^syslog_attempt_1589892856650_0073_1_07_000100_1]
{code}
2020-05-25 13:09:47,722 [INFO] [TezChild]
|writers.UnorderedPartitionedKVWriter|: Reducer_14: Adding spill event for
spill (final update=true), spillId=7
{code}
Reducer tasks get killed because they cannot recover from the mapper task
attempt failure, because fault tolerance wasn't implemented for pipelined
shuffle I guess: [^syslog_attempt_1589892856650_0073_1_08_000945_0]
{code}
2020-05-25 13:13:14,617 [ERROR] [ShuffleRunner {Map_16}] |impl.ShuffleManager|:
Killing self as previous attempt data could have been consumed
java.io.IOException: Previous event already got scheduled for
InputAttemptIdentifier [inputIdentifier=100, attemptNumber=1,
pathComponent=attempt_1589892856650_0073_1_07_000100_1_10017_0, spillType=1,
spillId=0]. Previous attempt's data could have been already merged to
memory/disk outputs. Killing (self) this task early. currentAttemptNum=0,
eventsProcessed={0, 1}, scheduledForDownload=true, newAttemptNum=1
at
org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager.validateInputAttemptForPipelinedShuffle(ShuffleManager.java:503)
at
org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager.constructFetcherForHost(ShuffleManager.java:551)
at
org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager$RunShuffleCallable.callInternal(ShuffleManager.java:455)
at
org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager$RunShuffleCallable.callInternal(ShuffleManager.java:398)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
The interesting part is that the above failing reducer task attempt runs 4
minutes later than the successful, second mapper task attempt, so there is not
case of unfortunate scheduling I think, rather the pipelined shuffle is not
prepared for fetching in case of killed and reattempted inputs. For example, in
the reducer task, it tries to fetch input from
attempt_1589892856650_0073_1_07_000100_0
3_0, which is the map of the first, failing attempt, but this task start 4
minutes after there is already a successful attempt:
attempt_1589892856650_0073_1_07_000100_1.
{code}
2020-05-25 13:13:14,556 [INFO] [Fetcher_B {Map_16} #3] |HttpConnection.url|:
for
url=http://ctr-e141-1563959304486-69251-01-09.hwx.site:13562/mapOutput?job=job_1589892856650_0073=1=94
5=attempt_1589892856650_0073_1_07_28_0_10003_0,attempt_1589892856650_0073_1_07_84_0_10003_0,attempt_1589892856650_0073_1_07_52_0_10002_0,attempt_1589892856650_0073_1_07_000100_0_1000
3_0,attempt_1589892856650_0073_1_07_000108_0_10002_0,attempt_1589892856650_0073_1_07_36_0_10003_0,attempt_1589892856650_0073_1_07_92_0_10002_0,attempt_1589892856650_0073_1_07_76_0_10003_
0,attempt_1589892856650_0073_1_07_12_0_10002_0,attempt_1589892856650_0073_1_07_44_0_10003_0,attempt_1589892856650_0073_1_07_04_0_10003_0,attempt_1589892856650_0073_1_07_68_0_10003_0,
attempt_1589892856650_0073_1_07_60_0_10002_0,attempt_1589892856650_0073_1_07_20_0_10003_0,attempt_1589892856650_0073_1_07_84_0_10003_1,attempt_1589892856650_0073_1_07_000108_0_10002_1,at
tempt_1589892856650_0073_1_07_28_0_10003_1,attempt_1589892856650_0073_1_07_000100_0_10003_1,attempt_1589892856650_0073_1_07_76_0_10003_1,attempt_1589892856650_0073_1_07_52_0_10002_1,atte
mpt_1589892856650_0073_1_07_66_1_10024_0,attempt_1589892856650_0073_1_07_66_1_10024_1=true
sent hash and receievd reply 0 ms
{code}
Fetches an input spill from the first failing task attempt
[jira] [Updated] (TEZ-4189) Support fault tolerance in pipelined data transfer for unordered output
[
https://issues.apache.org/jira/browse/TEZ-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4189:
--
Description:
Attached an example:
Mapper task attempt 0 fails while spilling spill3:
[^syslog_attempt_1589892856650_0073_1_07_000100_0]
{code}
2020-05-25 13:08:10,702 [INFO] [UnorderedOutSpiller {Reducer_14} #0]
|writers.UnorderedPartitionedKVWriter|: Writing spill 3 to
/grid/2/yarn/nm/usercache/root/appcache/application_1589892856650_0073/output/attempt_1589892856650_0073_1_07_000100_0_10003_3/file.out
2020-05-25 13:08:10,736 [INFO] [TaskHeartbeatThread] |task.TaskReporter|:
Received should die response from AM
{code}
Mapper task attempt 1 passes, successfully spills out 7 spills:
[^syslog_attempt_1589892856650_0073_1_07_000100_1]
{code}
2020-05-25 13:09:47,722 [INFO] [TezChild]
|writers.UnorderedPartitionedKVWriter|: Reducer_14: Adding spill event for
spill (final update=true), spillId=7
{code}
Reducer tasks get killed because they cannot recover from the mapper task
attempt failure, because fault tolerance wasn't implemented for pipelined
shuffle: [^syslog_attempt_1589892856650_0073_1_08_000945_0]
{code}
2020-05-25 13:13:14,617 [ERROR] [ShuffleRunner {Map_16}] |impl.ShuffleManager|:
Killing self as previous attempt data could have been consumed
java.io.IOException: Previous event already got scheduled for
InputAttemptIdentifier [inputIdentifier=100, attemptNumber=1,
pathComponent=attempt_1589892856650_0073_1_07_000100_1_10017_0, spillType=1,
spillId=0]. Previous attempt's data could have been already merged to
memory/disk outputs. Killing (self) this task early. currentAttemptNum=0,
eventsProcessed={0, 1}, scheduledForDownload=true, newAttemptNum=1
at
org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager.validateInputAttemptForPipelinedShuffle(ShuffleManager.java:503)
at
org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager.constructFetcherForHost(ShuffleManager.java:551)
at
org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager$RunShuffleCallable.callInternal(ShuffleManager.java:455)
at
org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager$RunShuffleCallable.callInternal(ShuffleManager.java:398)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
The interesting part is that the above failing reducer task attempt runs 4
minutes later than the successful, second mapper task attempt, so there this is
not case of unfortunate scheduling I think, rather the pipelined shuffle is not
prepared for fetching in case of killed and reattempted inputs. For example, in
the reducer task, it tries to fetch input from
attempt_1589892856650_0073_1_07_000100_0
3_0, which is the map of the first, failing attempt, but this task start 4
minutes after there is already a successful attempt:
attempt_1589892856650_0073_1_07_000100_1.
{code}
2020-05-25 13:13:14,556 [INFO] [Fetcher_B {Map_16} #3] |HttpConnection.url|:
for
url=http://ctr-e141-1563959304486-69251-01-09.hwx.site:13562/mapOutput?job=job_1589892856650_0073=1=94
5=attempt_1589892856650_0073_1_07_28_0_10003_0,attempt_1589892856650_0073_1_07_84_0_10003_0,attempt_1589892856650_0073_1_07_52_0_10002_0,attempt_1589892856650_0073_1_07_000100_0_1000
3_0,attempt_1589892856650_0073_1_07_000108_0_10002_0,attempt_1589892856650_0073_1_07_36_0_10003_0,attempt_1589892856650_0073_1_07_92_0_10002_0,attempt_1589892856650_0073_1_07_76_0_10003_
0,attempt_1589892856650_0073_1_07_12_0_10002_0,attempt_1589892856650_0073_1_07_44_0_10003_0,attempt_1589892856650_0073_1_07_04_0_10003_0,attempt_1589892856650_0073_1_07_68_0_10003_0,
attempt_1589892856650_0073_1_07_60_0_10002_0,attempt_1589892856650_0073_1_07_20_0_10003_0,attempt_1589892856650_0073_1_07_84_0_10003_1,attempt_1589892856650_0073_1_07_000108_0_10002_1,at
tempt_1589892856650_0073_1_07_28_0_10003_1,attempt_1589892856650_0073_1_07_000100_0_10003_1,attempt_1589892856650_0073_1_07_76_0_10003_1,attempt_1589892856650_0073_1_07_52_0_10002_1,atte
mpt_1589892856650_0073_1_07_66_1_10024_0,attempt_1589892856650_0073_1_07_66_1_10024_1=true
sent hash and receievd reply 0 ms
{code}
Fetches an input spill from the first failing task attempt
[jira] [Updated] (TEZ-4189) Support fault tolerance in pipelined data transfer for unordered output
[ https://issues.apache.org/jira/browse/TEZ-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4189: -- Attachment: syslog_attempt_1589892856650_0073_1_08_000945_0 syslog_attempt_1589892856650_0073_1_07_000100_1 syslog_attempt_1589892856650_0073_1_07_000100_0 > Support fault tolerance in pipelined data transfer for unordered output > --- > > Key: TEZ-4189 > URL: https://issues.apache.org/jira/browse/TEZ-4189 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: syslog_attempt_1589892856650_0073_1_07_000100_0, > syslog_attempt_1589892856650_0073_1_07_000100_1, > syslog_attempt_1589892856650_0073_1_08_000945_0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4189) Support fault tolerance in pipelined data transfer for unordered output
László Bodor created TEZ-4189: - Summary: Support fault tolerance in pipelined data transfer for unordered output Key: TEZ-4189 URL: https://issues.apache.org/jira/browse/TEZ-4189 Project: Apache Tez Issue Type: Bug Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TEZ-4189) Support fault tolerance in pipelined data transfer for unordered output
[ https://issues.apache.org/jira/browse/TEZ-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned TEZ-4189: - Assignee: László Bodor > Support fault tolerance in pipelined data transfer for unordered output > --- > > Key: TEZ-4189 > URL: https://issues.apache.org/jira/browse/TEZ-4189 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4188) Link to NodeManager Logs of Home and DAG details doesn't consider yarnProtocol
[
https://issues.apache.org/jira/browse/TEZ-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116502#comment-17116502
]
TezQA commented on TEZ-4188:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} jshint {color} | {color:green} 0m
15s{color} | {color:green} There were no new jshint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.9 Server=19.03.9 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/449/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4188 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13004011/TEZ-4188.01.patch |
| Optional Tests | dupname asflicense jshint |
| uname | Linux b50276c4a6f5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / f047d4a |
| Max. process+thread count | 46 (vs. ulimit of 5500) |
| modules | C: tez-ui U: tez-ui |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/449/console |
| versions | git=2.7.4 maven=3.3.9 jshint=2.11.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Link to NodeManager Logs of Home and DAG details doesn't consider yarnProtocol
> --
>
> Key: TEZ-4188
> URL: https://issues.apache.org/jira/browse/TEZ-4188
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eugene Chung
>Priority: Major
> Attachments: DagDetails.png, TEZ-4188.01.patch, TablesOfHome.png
>
>
> Even if yarnProtocol of configs.env is set to 'https', the link to
> NodeManager 'Logs' always use 'http'.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4188) Link to NodeManager Logs of Home and DAG details doesn't consider yarnProtocol
[ https://issues.apache.org/jira/browse/TEZ-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116491#comment-17116491 ] Eugene Chung edited comment on TEZ-4188 at 5/26/20, 7:02 AM: - [^TEZ-4188.01.patch] There is only one omitted case which 'env.app.yarnProtocol' must be used. was (Author: euigeun_chung): [^TEZ-4188.01.patch] There is one omitted case to check 'env.app.yarnProtocol'. > Link to NodeManager Logs of Home and DAG details doesn't consider yarnProtocol > -- > > Key: TEZ-4188 > URL: https://issues.apache.org/jira/browse/TEZ-4188 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Eugene Chung >Priority: Major > Attachments: DagDetails.png, TEZ-4188.01.patch, TablesOfHome.png > > > Even if yarnProtocol of configs.env is set to 'https', the link to > NodeManager 'Logs' always use 'http'. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4188) Link to NodeManager Logs of Home and DAG details doesn't consider yarnProtocol
[ https://issues.apache.org/jira/browse/TEZ-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116491#comment-17116491 ] Eugene Chung edited comment on TEZ-4188 at 5/26/20, 7:02 AM: - [^TEZ-4188.01.patch] There is an omitted case which 'env.app.yarnProtocol' must be used. was (Author: euigeun_chung): [^TEZ-4188.01.patch] There is only one omitted case which 'env.app.yarnProtocol' must be used. > Link to NodeManager Logs of Home and DAG details doesn't consider yarnProtocol > -- > > Key: TEZ-4188 > URL: https://issues.apache.org/jira/browse/TEZ-4188 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Eugene Chung >Priority: Major > Attachments: DagDetails.png, TEZ-4188.01.patch, TablesOfHome.png > > > Even if yarnProtocol of configs.env is set to 'https', the link to > NodeManager 'Logs' always use 'http'. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4188) Link to NodeManager Logs of Home and DAG details doesn't consider yarnProtocol
Eugene Chung created TEZ-4188: - Summary: Link to NodeManager Logs of Home and DAG details doesn't consider yarnProtocol Key: TEZ-4188 URL: https://issues.apache.org/jira/browse/TEZ-4188 Project: Apache Tez Issue Type: Bug Components: UI Reporter: Eugene Chung Attachments: DagDetails.png, TablesOfHome.png Even if yarnProtocol of configs.env is set to 'https', the link to NodeManager 'Logs' always use 'http'. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4135) Improve memory allocation when executing in-memory reads
[
https://issues.apache.org/jira/browse/TEZ-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116467#comment-17116467
]
TezQA commented on TEZ-4135:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
55s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} tez-runtime-library: The patch generated 0 new + 120
unchanged - 1 fixed = 120 total (was 121) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
50s{color} | {color:green} tez-runtime-library in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.9 Server=19.03.9 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/448/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4135 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13004003/TEZ-4135.04.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile |
| uname | Linux 41eb06c64085 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / f047d4a |
| Default Java | 1.8.0_252 |
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/448/testReport/ |
| Max. process+thread count | 152 (vs. ulimit of 5500) |
| modules | C: tez-runtime-library U: tez-runtime-library |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/448/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Improve memory allocation when executing in-memory reads
>
>
> Key: TEZ-4135
> URL: https://issues.apache.org/jira/browse/TEZ-4135
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: László Bodor
>Priority: Major
> Attachments: Screenshot 2020-03-16 at 6.34.42 PM.jpg,
> TEZ-4135.01.patch,
