[jira] [Commented] (TEZ-4173) TestRecovery flaky timeout on master
[
https://issues.apache.org/jira/browse/TEZ-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103453#comment-17103453
]
TezQA commented on TEZ-4173:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
9s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 6s{color}
| {color:red} tez-dag in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | tez.dag.app.dag.impl.TestDAGRecovery |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/414/artifact/out/Dockerfile |
| JIRA Issue | TEZ-4173 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13002499/TEZ-4173.01.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile |
| uname | Linux 6b8187d43c07 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 4e4b8e5 |
| Default Java | 1.8.0_252 |
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/414/artifact/out/patch-unit-tez-dag.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/414/testReport/ |
| Max. process+thread count | 228 (vs. ulimit of 5500) |
| modules | C: tez-dag U: tez-dag |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/414/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> TestRecovery flaky timeout on master
>
>
> Key: TEZ-4173
> URL: https://issues.apache.org/jira/browse/TEZ-4173
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>
[jira] [Updated] (TEZ-4173) TestRecovery flaky timeout on master
[ https://issues.apache.org/jira/browse/TEZ-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4173: -- Attachment: TEZ-4173.01.patch > TestRecovery flaky timeout on master > > > Key: TEZ-4173 > URL: https://issues.apache.org/jira/browse/TEZ-4173 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4173.01.patch, am.jstack.log, surefire.jstack.log, > tez4173.tar.gz > > > application logs and junit output in [^tez4173.tar.gz] > one of the running AM's jstack is [^am.jstack.log] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4173) TestRecovery flaky timeout on master
[
https://issues.apache.org/jira/browse/TEZ-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103290#comment-17103290
]
László Bodor edited comment on TEZ-4173 at 5/9/20, 4:59 PM:
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code:java}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code:java}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
if I remove [both of the
conditions|https://github.com/apache/tez/commit/e7c24f06e220cb707f114b4f5cc7210d27cce72d#diff-92831ef1b6960a063fa41c2d293823ffR2832]
introduced by the patch, it succeeds:
{code:java}
recoveryData.isVertexTasksStarted() && isVertexInitSkippedInParentVertices()
{code}
the test picks randomly from events and shuts the AM down after the picked
events, and I found that the issue only comes in case of the first vertex's
VertexConfigurationDoneEvent + enableAutoParallelism
when it tries to recover the first vertex, recoveryData.isVertexTasksStarted()
returns false, because recoveryData.taskRecoveryDataMap is empty, so it
doesn't hit the skipping codepath...so this is an edge case which should be
handled if i understand correctly, where vertex configure event is already seen
but tasks are not stored into the recovery data
was (Author: abstractdog):
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
if I remove [any of the
conditions|https://github.com/apache/tez/commit/e7c24f06e220cb707f114b4f5cc7210d27cce72d#diff-92831ef1b6960a063fa41c2d293823ffR2832]
introduced by the patch, it succeeds:
{code}
recoveryData.isVertexTasksStarted() && isVertexInitSkippedInParentVertices()
{code}
the test picks randomly from events and shuts the AM down after the picked
events, and I found that the issue only comes in case of the first vertex's
VertexConfigurationDoneEvent + enableAutoParallelism
> TestRecovery flaky timeout on master
>
>
> Key: TEZ-4173
> URL: https://issues.apache.org/jira/browse/TEZ-4173
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: am.jstack.log, surefire.jstack.log, tez4173.tar.gz
>
>
> application logs and junit output in [^tez4173.tar.gz]
> one of the running AM's jstack is [^am.jstack.log]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4173) TestRecovery flaky timeout on master
[
https://issues.apache.org/jira/browse/TEZ-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103290#comment-17103290
]
László Bodor edited comment on TEZ-4173 at 5/9/20, 4:59 PM:
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code:java}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code:java}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
if I remove [both of the
conditions|https://github.com/apache/tez/commit/e7c24f06e220cb707f114b4f5cc7210d27cce72d#diff-92831ef1b6960a063fa41c2d293823ffR2832]
introduced by the patch, it succeeds:
{code:java}
recoveryData.isVertexTasksStarted() && isVertexInitSkippedInParentVertices()
{code}
the test picks randomly from events and shuts the AM down after the picked
events, and I found that the issue only comes in case of the first vertex's
VertexConfigurationDoneEvent + enableAutoParallelism
when it tries to recover the first vertex, recoveryData.isVertexTasksStarted()
returns false, because recoveryData.taskRecoveryDataMap is empty, so it
doesn't hit the skipping codepath...so this is an edge case which should be
handled if i understand correctly, where "vertex configure done" event is
already seen but tasks are not stored into the recovery data
was (Author: abstractdog):
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code:java}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code:java}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
if I remove [both of the
conditions|https://github.com/apache/tez/commit/e7c24f06e220cb707f114b4f5cc7210d27cce72d#diff-92831ef1b6960a063fa41c2d293823ffR2832]
introduced by the patch, it succeeds:
{code:java}
recoveryData.isVertexTasksStarted() && isVertexInitSkippedInParentVertices()
{code}
the test picks randomly from events and shuts the AM down after the picked
events, and I found that the issue only comes in case of the first vertex's
VertexConfigurationDoneEvent + enableAutoParallelism
when it tries to recover the first vertex, recoveryData.isVertexTasksStarted()
returns false, because recoveryData.taskRecoveryDataMap is empty, so it
doesn't hit the skipping codepath...so this is an edge case which should be
handled if i understand correctly, where vertex configure event is already seen
but tasks are not stored into the recovery data
> TestRecovery flaky timeout on master
>
>
> Key: TEZ-4173
> URL: https://issues.apache.org/jira/browse/TEZ-4173
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: am.jstack.log, surefire.jstack.log, tez4173.tar.gz
>
>
> application logs and junit output in [^tez4173.tar.gz]
> one of the running AM's jstack is [^am.jstack.log]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4173) TestRecovery flaky timeout on master
[
https://issues.apache.org/jira/browse/TEZ-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103290#comment-17103290
]
László Bodor edited comment on TEZ-4173 at 5/9/20, 3:33 PM:
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
if I remove [any of the
conditions|https://github.com/apache/tez/commit/e7c24f06e220cb707f114b4f5cc7210d27cce72d#diff-92831ef1b6960a063fa41c2d293823ffR2832]
introduced by the patch, it succeeds:
{code}
recoveryData.isVertexTasksStarted() && isVertexInitSkippedInParentVertices()
{code}
the test picks randomly from events and shuts the AM down after the picked
events, and I found that the issue only comes in case of the first vertex's
VertexConfigurationDoneEvent + enableAutoParallelism
was (Author: abstractdog):
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
if I remove [any of the
conditions|https://github.com/apache/tez/commit/e7c24f06e220cb707f114b4f5cc7210d27cce72d#diff-92831ef1b6960a063fa41c2d293823ffR2832]
introduced by the patch, it succeeds:
{code}
recoveryData.isVertexTasksStarted() && isVertexInitSkippedInParentVertices()
{code}
the test picks randomly from events and shuts the AM down after the picked
events, and I found that the issue only comes in case of
VertexConfigurationDoneEvent + enableAutoParallelism
> TestRecovery flaky timeout on master
>
>
> Key: TEZ-4173
> URL: https://issues.apache.org/jira/browse/TEZ-4173
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: am.jstack.log, surefire.jstack.log, tez4173.tar.gz
>
>
> application logs and junit output in [^tez4173.tar.gz]
> one of the running AM's jstack is [^am.jstack.log]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4173) TestRecovery flaky timeout on master
[
https://issues.apache.org/jira/browse/TEZ-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103290#comment-17103290
]
László Bodor edited comment on TEZ-4173 at 5/9/20, 3:13 PM:
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
if I remove [any of the
conditions|https://github.com/apache/tez/commit/e7c24f06e220cb707f114b4f5cc7210d27cce72d#diff-92831ef1b6960a063fa41c2d293823ffR2832]
introduced by the patch, it succeeds:
{code}
recoveryData.isVertexTasksStarted() && isVertexInitSkippedInParentVertices()
{code}
the test picks randomly from events and shuts the AM down after the picked
events, and I found that the issue only comes in case of
VertexConfigurationDoneEvent + enableAutoParallelism
was (Author: abstractdog):
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
if I remove [any of the
conditions|https://github.com/apache/tez/commit/e7c24f06e220cb707f114b4f5cc7210d27cce72d#diff-92831ef1b6960a063fa41c2d293823ffR2832]
introduced by the patch, it succeeds:
{code}
recoveryData.isVertexTasksStarted() && isVertexInitSkippedInParentVertices()
{code}
> TestRecovery flaky timeout on master
>
>
> Key: TEZ-4173
> URL: https://issues.apache.org/jira/browse/TEZ-4173
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: am.jstack.log, surefire.jstack.log, tez4173.tar.gz
>
>
> application logs and junit output in [^tez4173.tar.gz]
> one of the running AM's jstack is [^am.jstack.log]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4173) TestRecovery flaky timeout on master
[
https://issues.apache.org/jira/browse/TEZ-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103290#comment-17103290
]
László Bodor edited comment on TEZ-4173 at 5/9/20, 2:29 PM:
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
if I remove [any of the
conditions|https://github.com/apache/tez/commit/e7c24f06e220cb707f114b4f5cc7210d27cce72d#diff-92831ef1b6960a063fa41c2d293823ffR2832]
introduced by the patch, it succeeds:
{code}
recoveryData.isVertexTasksStarted() && isVertexInitSkippedInParentVertices()
{code}
was (Author: abstractdog):
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
> TestRecovery flaky timeout on master
>
>
> Key: TEZ-4173
> URL: https://issues.apache.org/jira/browse/TEZ-4173
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: am.jstack.log, surefire.jstack.log, tez4173.tar.gz
>
>
> application logs and junit output in [^tez4173.tar.gz]
> one of the running AM's jstack is [^am.jstack.log]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4173) TestRecovery flaky timeout on master
[
https://issues.apache.org/jira/browse/TEZ-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103290#comment-17103290
]
László Bodor edited comment on TEZ-4173 at 5/9/20, 2:28 PM:
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
{code}
mvn test -Dtest=TestRecovery#testRecovery_OrderedWordCount -pl ./tez-tests -pl
tez-dag
{code}
it stucks here according to the logs:
{code}
DAG: State: RUNNING Progress: 0% TotalTasks: 5 Succeeded: 0 Running: 0 Failed:
0 Killed: 0
VertexStatus: VertexName: Tokenizer Progress: 0% TotalTasks: -1
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Summation Progress: 0% TotalTasks: 5
Succeeded: 0 Running: 0 Failed: 0 Killed: 0
VertexStatus: VertexName: Sorter Progress: 0% TotalTasks: 1 Succeeded:
0 Running: 0 Failed: 0 Killed: 0
{code}
was (Author: abstractdog):
seems like it's broken by TEZ-4140, or at least I've run the test successfully
twice in a row with reverted TEZ-4140, and then it failed with the patch for
the first time
cc: [~srahman], if you have any pointers about this
> TestRecovery flaky timeout on master
>
>
> Key: TEZ-4173
> URL: https://issues.apache.org/jira/browse/TEZ-4173
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: am.jstack.log, surefire.jstack.log, tez4173.tar.gz
>
>
> application logs and junit output in [^tez4173.tar.gz]
> one of the running AM's jstack is [^am.jstack.log]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103297#comment-17103297
]
TezQA commented on TEZ-4175:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
32s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange} tez-api: The patch generated 1 new + 126
unchanged - 3 fixed = 127 total (was 129) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} tez-mapreduce: The patch generated 0 new + 18
unchanged - 1 fixed = 18 total (was 19) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} tez-dag: The patch generated 0 new + 169 unchanged -
2 fixed = 169 total (was 171) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} tez-tests: The patch generated 0 new + 8 unchanged -
2 fixed = 8 total (was 10) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch passed checkstyle in tez-history-parser
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch passed checkstyle in tez-aux-services
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
21s{color} | {color:red} tez-api generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 46s{color}
| {color:red} tez-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
7s{color} | {color:green} tez-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
28s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 43s{color}
| {color:red} tez-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
11s{color} | {color:green} tez-history-parser in the patch passed. {color} |
| {color:green}+1{color}
[jira] [Commented] (TEZ-4173) TestRecovery flaky timeout on master
[ https://issues.apache.org/jira/browse/TEZ-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103290#comment-17103290 ] László Bodor commented on TEZ-4173: --- seems like it's broken by TEZ-4140, or at least I've run the test successfully twice in a row with reverted TEZ-4140, and then it failed with the patch for the first time cc: [~srahman], if you have any pointers about this > TestRecovery flaky timeout on master > > > Key: TEZ-4173 > URL: https://issues.apache.org/jira/browse/TEZ-4173 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: am.jstack.log, surefire.jstack.log, tez4173.tar.gz > > > application logs and junit output in [^tez4173.tar.gz] > one of the running AM's jstack is [^am.jstack.log] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4175:
--
Attachment: TEZ-4175.02.patch
> Consider removing YarnConfiguration where it's possible
> ---
>
> Key: TEZ-4175
> URL: https://issues.apache.org/jira/browse/TEZ-4175
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4175.01.patch, TEZ-4175.02.patch
>
>
> A comment in DAGAppmaster made me think that we don't need to rely on
> [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java]
> in all cases, what if it can be replace with base Configuration...
> {code}
> // TODO Does this really need to be a YarnConfiguration ?
> Configuration conf = new Configuration(new YarnConfiguration());
> {code}
> From hadoop 3.1.3, I cannot see that it adds e.g. yarn-site as a resource:
> {code}
> public YarnConfiguration() {
> super();
> }
>
> public YarnConfiguration(Configuration conf) {
> super(conf);
> if (! (conf instanceof YarnConfiguration)) {
> this.reloadConfiguration();
> }
> }
> {code}
> in current codebase:
> {code}
> grep -iRH "new YarnConfiguration" --include="*.java"
> tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
> YarnConfiguration yarnConf = new YarnConfiguration(conf);
> tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
> super.serviceInit(new YarnConfiguration(conf));
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
> tezClient.init(new TezConfiguration(false), new YarnConfiguration());
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:return
> getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
> ugi);
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
> tezConf = new TezConfiguration(new YarnConfiguration());
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
>tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
> tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
> Configuration testConf = new YarnConfiguration(
> tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
> this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103259#comment-17103259
]
TezQA commented on TEZ-4175:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
34s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} tez-api: The patch generated 5 new + 125
unchanged - 3 fixed = 130 total (was 128) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 10s{color} | {color:orange} tez-mapreduce: The patch generated 1 new + 18
unchanged - 1 fixed = 19 total (was 19) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} tez-dag: The patch generated 0 new + 169 unchanged -
2 fixed = 169 total (was 171) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} tez-tests: The patch generated 0 new + 8 unchanged -
2 fixed = 8 total (was 10) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch passed checkstyle in tez-history-parser
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch passed checkstyle in tez-aux-services
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} tez-api generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
40s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
3s{color} | {color:green} tez-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
58s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 48s{color}
| {color:red} tez-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
7s{color} | {color:green} tez-history-parser in the patch passed. {color} |
|
[jira] [Commented] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103232#comment-17103232
]
László Bodor commented on TEZ-4175:
---
uploaded experimental patch [^TEZ-4175.01.patch] for kicking off tests
> Consider removing YarnConfiguration where it's possible
> ---
>
> Key: TEZ-4175
> URL: https://issues.apache.org/jira/browse/TEZ-4175
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4175.01.patch
>
>
> A comment in DAGAppmaster made me think that we don't need to rely on
> [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java]
> in all cases, what if it can be replace with base Configuration...
> {code}
> // TODO Does this really need to be a YarnConfiguration ?
> Configuration conf = new Configuration(new YarnConfiguration());
> {code}
> From hadoop 3.1.3, I cannot see that it adds e.g. yarn-site as a resource:
> {code}
> public YarnConfiguration() {
> super();
> }
>
> public YarnConfiguration(Configuration conf) {
> super(conf);
> if (! (conf instanceof YarnConfiguration)) {
> this.reloadConfiguration();
> }
> }
> {code}
> in current codebase:
> {code}
> grep -iRH "new YarnConfiguration" --include="*.java"
> tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
> YarnConfiguration yarnConf = new YarnConfiguration(conf);
> tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
> super.serviceInit(new YarnConfiguration(conf));
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
> tezClient.init(new TezConfiguration(false), new YarnConfiguration());
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:return
> getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
> ugi);
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
> tezConf = new TezConfiguration(new YarnConfiguration());
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
>tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
> tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
> Configuration testConf = new YarnConfiguration(
> tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
> this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
>
[jira] [Updated] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4175:
--
Attachment: TEZ-4175.01.patch
> Consider removing YarnConfiguration where it's possible
> ---
>
> Key: TEZ-4175
> URL: https://issues.apache.org/jira/browse/TEZ-4175
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4175.01.patch
>
>
> A comment in DAGAppmaster made me think that we don't need to rely on
> [YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java]
> in all cases, what if it can be replace with base Configuration...
> {code}
> // TODO Does this really need to be a YarnConfiguration ?
> Configuration conf = new Configuration(new YarnConfiguration());
> {code}
> From hadoop 3.1.3, I cannot see that it adds e.g. yarn-site as a resource:
> {code}
> public YarnConfiguration() {
> super();
> }
>
> public YarnConfiguration(Configuration conf) {
> super(conf);
> if (! (conf instanceof YarnConfiguration)) {
> this.reloadConfiguration();
> }
> }
> {code}
> in current codebase:
> {code}
> grep -iRH "new YarnConfiguration" --include="*.java"
> tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
> YarnConfiguration yarnConf = new YarnConfiguration(conf);
> tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
> super.serviceInit(new YarnConfiguration(conf));
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
> YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
> tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
> tezClient.init(new TezConfiguration(false), new YarnConfiguration());
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:
> amConfig.setYarnConfiguration(new
> YarnConfiguration(amConfig.getTezConfiguration()));
> tez-api/src/main/java/org/apache/tez/client/TezClient.java:return
> getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
> ugi);
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
> tezConf = new TezConfiguration(new YarnConfiguration());
> tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
>tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
> tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
> Configuration testConf = new YarnConfiguration(
> tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
> this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration conf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
> tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
> Configuration tezConf = new Configuration(new YarnConfiguration());
>
[jira] [Updated] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4175:
--
Description:
A comment in DAGAppmaster made me think that we don't need to rely on
[YarnConfiguration|https://github.com/apache/hadoop/blob/branch-3.1.3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java]
in all cases, what if it can be replace with base Configuration...
{code}
// TODO Does this really need to be a YarnConfiguration ?
Configuration conf = new Configuration(new YarnConfiguration());
{code}
>From hadoop 3.1.3, I cannot see that it adds e.g. yarn-site as a resource:
{code}
public YarnConfiguration() {
super();
}
public YarnConfiguration(Configuration conf) {
super(conf);
if (! (conf instanceof YarnConfiguration)) {
this.reloadConfiguration();
}
}
{code}
in current codebase:
{code}
grep -iRH "new YarnConfiguration" --include="*.java"
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
YarnConfiguration yarnConf = new YarnConfiguration(conf);
tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
super.serviceInit(new YarnConfiguration(conf));
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
tezClient.init(new TezConfiguration(false), new YarnConfiguration());
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:return
getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
ugi);
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration());
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
Configuration testConf = new YarnConfiguration(
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
Configuration conf = new Configuration(new YarnConfiguration());
{code}
was:
A comment in DAGAppmaster made me think that we don't need to rely on
YarnConfiguration in all cases, what if it can be replace with base
Configuration...
{code}
// TODO Does this really need to be a YarnConfiguration ?
Configuration conf = new Configuration(new YarnConfiguration());
{code}
>From
[jira] [Updated] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4175:
--
Description:
A comment in DAGAppmaster made me think that we don't need to rely on
YarnConfiguration in all cases, what if it can be replace with base
Configuration...
{code}
// TODO Does this really need to be a YarnConfiguration ?
Configuration conf = new Configuration(new YarnConfiguration());
{code}
>From hadoop 3.1.3, I cannot see that it adds e.g. yarn-site as a resource:
{code}
public YarnConfiguration() {
super();
}
public YarnConfiguration(Configuration conf) {
super(conf);
if (! (conf instanceof YarnConfiguration)) {
this.reloadConfiguration();
}
}
{code}
in current codebase:
{code}
grep -iRH "new YarnConfiguration" --include="*.java"
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
YarnConfiguration yarnConf = new YarnConfiguration(conf);
tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
super.serviceInit(new YarnConfiguration(conf));
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
tezClient.init(new TezConfiguration(false), new YarnConfiguration());
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:return
getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
ugi);
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration());
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
Configuration testConf = new YarnConfiguration(
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
Configuration conf = new Configuration(new YarnConfiguration());
{code}
was:
A comment in DAGAppmaster made me think that we don't need to rely on
YarnConfiguration in all cases, what if it can be replace with base
Configuration...
{code}
// TODO Does this really need to be a YarnConfiguration ?
Configuration conf = new Configuration(new YarnConfiguration());
{code}
>From hadoop 3.1.3:
{code}
public YarnConfiguration() {
super();
}
public YarnConfiguration(Configuration conf) {
super(conf);
if (! (conf instanceof
[jira] [Updated] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[
https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4175:
--
Description:
A comment in DAGAppmaster made me think that we don't need to rely on
YarnConfiguration in all cases, what if it can be replace with base
Configuration...
{code}
// TODO Does this really need to be a YarnConfiguration ?
Configuration conf = new Configuration(new YarnConfiguration());
{code}
>From hadoop 3.1.3:
{code}
public YarnConfiguration() {
super();
}
public YarnConfiguration(Configuration conf) {
super(conf);
if (! (conf instanceof YarnConfiguration)) {
this.reloadConfiguration();
}
}
{code}
in current codebase:
{code}
grep -iRH "new YarnConfiguration" --include="*.java"
tez-plugins/tez-history-parser/src/main/java/org/apache/tez/history/ATSImportTool.java:
YarnConfiguration yarnConf = new YarnConfiguration(conf);
tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:
super.serviceInit(new YarnConfiguration(conf));
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/dag/api/client/rpc/TestDAGClient.java:
YarnConfiguration yarnConf = new YarnConfiguration(tezConf);
tez-api/src/test/java/org/apache/tez/client/TestTezClient.java:
tezClient.init(new TezConfiguration(false), new YarnConfiguration());
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:
amConfig.setYarnConfiguration(new
YarnConfiguration(amConfig.getTezConfiguration()));
tez-api/src/main/java/org/apache/tez/client/TezClient.java:return
getDAGClient(appId, tezConf, new YarnConfiguration(tezConf), frameworkClient,
ugi);
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration());
tez-tests/src/test/java/org/apache/tez/test/FaultToleranceTestRunner.java:
tezConf = new TezConfiguration(new YarnConfiguration(this.conf));
tez-mapreduce/src/test/java/org/apache/tez/mapreduce/hadoop/TestMRInputHelpers.java:
Configuration testConf = new YarnConfiguration(
tez-mapreduce/src/main/java/org/apache/tez/mapreduce/client/YARNRunner.java:
this(conf, new ResourceMgrDelegate(new YarnConfiguration(conf)));
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration conf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/test/java/org/apache/tez/dag/app/rm/TestContainerReuse.java:
Configuration tezConf = new Configuration(new YarnConfiguration());
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
Configuration conf = new Configuration(new YarnConfiguration());
{code}
> Consider removing YarnConfiguration where it's possible
> ---
>
> Key: TEZ-4175
> URL: https://issues.apache.org/jira/browse/TEZ-4175
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
>
> A comment in DAGAppmaster made me think that we don't need to rely on
> YarnConfiguration in all cases, what if it can be replace with base
>
[jira] [Assigned] (TEZ-4175) Consider removing YarnConfiguration where it's possible
[ https://issues.apache.org/jira/browse/TEZ-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned TEZ-4175: - Assignee: László Bodor > Consider removing YarnConfiguration where it's possible > --- > > Key: TEZ-4175 > URL: https://issues.apache.org/jira/browse/TEZ-4175 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4175) Consider removing YarnConfiguration where it's possible
László Bodor created TEZ-4175: - Summary: Consider removing YarnConfiguration where it's possible Key: TEZ-4175 URL: https://issues.apache.org/jira/browse/TEZ-4175 Project: Apache Tez Issue Type: Improvement Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-2672) Allow specifying a new payload for plugins when a new DAG starts
[
https://issues.apache.org/jira/browse/TEZ-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103141#comment-17103141
]
TezQA commented on TEZ-2672:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m
0s{color} | {color:blue} prototool was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
8s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} tez-api: The patch generated 1 new + 69
unchanged - 1 fixed = 70 total (was 70) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} tez-dag: The patch generated 0 new + 126 unchanged -
1 fixed = 126 total (was 127) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
50s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
8s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 base:
https://builds.apache.org/job/PreCommit-TEZ-Build/411/artifact/out/Dockerfile |
| JIRA Issue | TEZ-2672 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13002466/TEZ-2672.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
checkstyle compile cc prototool |
| uname | Linux 8961275bfcda 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 4e4b8e5 |
| Default Java | 1.8.0_252 |
| checkstyle |
[jira] [Commented] (TEZ-4160) [Umbrella] Speed up unit tests
[
https://issues.apache.org/jira/browse/TEZ-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103136#comment-17103136
]
László Bodor commented on TEZ-4160:
---
run unit tests locally, results in [^mvntest.log]
{code}
grep "Tests run" mvntest.log | grep "org.apache" | awk '{split($0,a,":"); print
a[6]}' | sort -n -r
{code}
so here is what should be in focus:
{code}
237.902 s - in org.apache.tez.test.TestSecureShuffle
211.856 s - in org.apache.tez.test.TestTezJobs
205.894 s - in org.apache.tez.test.TestAMRecovery
201.936 s - in org.apache.tez.test.TestFaultTolerance
143.215 s - in org.apache.tez.mapreduce.TestMRRJobsDAGApi
126.698 s - in org.apache.tez.auxservices.TestShuffleHandlerJobs
126.051 s - in
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter
112.679 s - in org.apache.tez.tests.TestExternalTezServicesErrors
111.228 s - in org.apache.tez.analyzer.TestAnalyzer
99.141 s - in org.apache.tez.history.TestHistoryParser
85.936 s - in org.apache.tez.mapreduce.TestMRRJobs
72.701 s - in org.apache.tez.test.TestDAGRecovery
72.549 s - in org.apache.tez.dag.history.ats.acls.TestATSHistoryWithACLs
63.041 s - in org.apache.tez.tests.TestExternalTezServices
59.068 s - in
org.apache.tez.runtime.library.common.sort.impl.TestPipelinedSorter
58.179 s - in org.apache.tez.dag.history.ats.acls.TestATSHistoryV15
54.081 s - in org.apache.tez.test.TestDAGRecovery2
46.971 s - in org.apache.tez.test.TestPipelinedShuffle
45.966 s - in org.apache.tez.test.TestLocalMode
44.721 s - in
org.apache.tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster
44.116 s - in org.apache.tez.test.TestExceptionPropagation
42.946 s - in org.apache.tez.client.TestTezClient
41.207 s - in org.apache.tez.dag.history.recovery.TestRecoveryService
18.657 s - in
org.apache.tez.dag.history.logging.ats.TestATSHistoryLoggingService
14.601 s - in
org.apache.tez.runtime.library.common.shuffle.impl.TestShuffleManager
12.898 s - in
org.apache.tez.dag.api.client.rpc.TestDAGClientAMProtocolBlockingPBServerImpl
12.088 s - in org.apache.tez.dag.app.dag.impl.TestDAGImpl
11.628 s - in org.apache.tez.http.TestHttpConnection
11.202 s - in org.apache.tez.tests.TestExtServicesWithLocalMode
11.132 s - in org.apache.tez.runtime.library.common.sort.impl.TestTezMerger
8.518 s - in
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestShuffleScheduler
8.335 s - in org.apache.tez.dag.app.TestSpeculation
7.967 s - in org.apache.tez.dag.app.TestRecoveryParser
6.987 s - in org.apache.tez.dag.app.TestMockDAGAppMaster
6.56 s - in
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.TestMergeManager
6.273 s - in org.apache.tez.dag.app.rm.TestContainerReuse
6.218 s - in org.apache.tez.dag.app.rm.TestTaskScheduler
6.182 s - in org.apache.tez.common.TestTezCommonUtils
6.16 s - in org.apache.tez.dag.api.client.rpc.TestDAGClient
6.006 s - in org.apache.tez.mapreduce.processor.map.TestMapProcessor
5.826 s - in org.apache.tez.dag.app.dag.impl.TestCommit
5.149 s - in org.apache.tez.mapreduce.hadoop.TestMRInputHelpers
4.986 s - in org.apache.tez.test.TestMiniTezCluster
4.638 s - in org.apache.tez.runtime.library.common.TestValuesIterator
4.206 s - in org.apache.tez.test.TestTaskErrorsUsingLocalMode
3.945 s - in org.apache.tez.dag.app.dag.impl.TestVertexImpl
3.659 s - in org.apache.tez.dag.app.TestPreemption
3.387 s - in
org.apache.tez.runtime.library.common.sort.impl.dflt.TestDefaultSorter
3.238 s - in
org.apache.tez.dag.library.vertexmanager.TestFairShuffleVertexManager
3.164 s - in org.apache.tez.client.TestTezClientUtils
3.061 s - in org.apache.tez.runtime.library.output.TestOnFileSortedOutput
2.77 s - in org.apache.tez.dag.app.dag.impl.TestDAGRecovery
2.365 s - in org.apache.tez.runtime.library.common.sort.impl.TestIFile
2.362 s - in org.apache.tez.dag.app.TestDAGAppMaster
1.934 s - in org.apache.hadoop.mapred.split.TestGroupedSplits
1.817 s - in org.apache.tez.auxservices.TestShuffleHandler
1.625 s - in org.apache.tez.mapreduce.output.TestMultiMROutput
1.487 s - in org.apache.tez.mapreduce.processor.reduce.TestReduceProcessor
1.415 s - in
org.apache.tez.dag.history.logging.ats.TestATSV15HistoryLoggingService
1.39 s - in org.apache.tez.mapreduce.common.TestMRInputAMSplitGenerator
1.349 s - in
org.apache.tez.dag.history.logging.proto.TestProtoHistoryLoggingService
1.347 s - in org.apache.tez.dag.app.rm.TestTaskSchedulerManager
1.277 s - in org.apache.tez.mapreduce.input.TestMultiMRInput
1.269 s - in org.apache.tez.dag.app.dag.impl.TestTaskAttempt
1.243 s - in org.apache.tez.auxservices.TestIndexCache
1.242 s - in
org.apache.tez.dag.history.logging.proto.TestDagManifestFileScanner
1.226 s - in org.apache.tez.dag.app.launcher.TestTezLocalCacheManager
1.223 s - in org.apache.tez.runtime.library.output.TestOnFileUnorderedKVOutput
1.191 s - in
[jira] [Updated] (TEZ-4160) [Umbrella] Speed up unit tests
[ https://issues.apache.org/jira/browse/TEZ-4160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4160: -- Attachment: mvntest.log > [Umbrella] Speed up unit tests > -- > > Key: TEZ-4160 > URL: https://issues.apache.org/jira/browse/TEZ-4160 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Turner Eagles >Priority: Major > Attachments: mvntest.log > > > Umbrella ticket to document and speed up unit tests in tez. This is an > umbrella ticket and shouldn't be assigned to anyone. Add subtasks to track > individual test classes, packages, or individual tests so they can be worked > on in small pieces. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (TEZ-4157) ShuffleHandler: upgrade to netty4
[ https://issues.apache.org/jira/browse/TEZ-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103133#comment-17103133 ] László Bodor edited comment on TEZ-4157 at 5/9/20, 6:41 AM: thanks [~jeagles] for taking a look writeAndFlush is a major api change, according to [guideline|https://netty.io/wiki/new-and-noteworthy-in-4.0.html], write() does not flush automatically, so this convenience method can be used while I was upgrading, and compiled successfully for the first time with using write() everywhere, 80% of unit tests were hanging with write(), so I can confirm that write() is not enough anymore was (Author: abstractdog): thanks [~jeagles] for taking a look writeAndFlush is a major change regarding netty changes, according to [guideline|https://netty.io/wiki/new-and-noteworthy-in-4.0.html], write() does not flush automatically, so this convenience method can be used while I was upgrading, and compiled successfully for the first time with using write() everywhere, 80% of unit tests were hanging with write(), so I can confirm that write() is not enough anymore > ShuffleHandler: upgrade to netty4 > - > > Key: TEZ-4157 > URL: https://issues.apache.org/jira/browse/TEZ-4157 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4157.01.patch, TEZ-4157.02.patch, TEZ-4157.03.patch > > > -In the dependency tree, there are 2 occurrences of compile scope direct > netty dependencies, however, they're not used at all. I compiled locally > successfully without them. E.g. when investigating blackduck alerts > (complaining about netty deps for current 3.10.5.Final), it would be cleaner > to start from a dependency tree where Tez doesn't depend on netty directly in > order to eliminate its responsibility (and move the focus to underlying > hadoop for instance).- > Tez depends on netty3 almost only in ShuffleHandler and some related classes. > We can eliminate netty3 by upgrading it, but this effort might involve some > testing due to fundamental [changes from > netty3->netty4|https://netty.io/wiki/new-and-noteworthy-in-4.0.html] + we > don't have a reference yet, as [hadoop's > ShuffleHandler|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java] > is still on netty3. > As per the netty documentation, we can also expect some performance > improvement (e.g. Pooled buffers). > Background: > netty4 migration guideline: > https://netty.io/wiki/new-and-noteworthy-in-4.0.html > articles of possible performance improvement: > https://blog.twitter.com/engineering/en_us/a/2013/netty-4-at-twitter-reduced-gc-overhead.html > https://developer.squareup.com/blog/upgrading-a-reverse-proxy-from-netty-3-to-4/ > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4157) ShuffleHandler: upgrade to netty4
[ https://issues.apache.org/jira/browse/TEZ-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103133#comment-17103133 ] László Bodor commented on TEZ-4157: --- thanks [~jeagles] for taking a look writeAndFlush is a major change regarding netty changes, according to [guideline|https://netty.io/wiki/new-and-noteworthy-in-4.0.html], write() does not flush automatically, so this convenience method can be used while I was upgrading, and compiled successfully for the first time with using write() everywhere, 80% of unit tests were hanging with write(), so I can confirm that write() is not enough anymore > ShuffleHandler: upgrade to netty4 > - > > Key: TEZ-4157 > URL: https://issues.apache.org/jira/browse/TEZ-4157 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4157.01.patch, TEZ-4157.02.patch, TEZ-4157.03.patch > > > -In the dependency tree, there are 2 occurrences of compile scope direct > netty dependencies, however, they're not used at all. I compiled locally > successfully without them. E.g. when investigating blackduck alerts > (complaining about netty deps for current 3.10.5.Final), it would be cleaner > to start from a dependency tree where Tez doesn't depend on netty directly in > order to eliminate its responsibility (and move the focus to underlying > hadoop for instance).- > Tez depends on netty3 almost only in ShuffleHandler and some related classes. > We can eliminate netty3 by upgrading it, but this effort might involve some > testing due to fundamental [changes from > netty3->netty4|https://netty.io/wiki/new-and-noteworthy-in-4.0.html] + we > don't have a reference yet, as [hadoop's > ShuffleHandler|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java] > is still on netty3. > As per the netty documentation, we can also expect some performance > improvement (e.g. Pooled buffers). > Background: > netty4 migration guideline: > https://netty.io/wiki/new-and-noteworthy-in-4.0.html > articles of possible performance improvement: > https://blog.twitter.com/engineering/en_us/a/2013/netty-4-at-twitter-reduced-gc-overhead.html > https://developer.squareup.com/blog/upgrading-a-reverse-proxy-from-netty-3-to-4/ > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-2672) Allow specifying a new payload for plugins when a new DAG starts
[ https://issues.apache.org/jira/browse/TEZ-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103130#comment-17103130 ] László Bodor commented on TEZ-2672: --- could you please take a look at the patch [~rbalamohan], [~sseth], [~ashutoshc]? in the patch 1. there is an introduced DAG.setDAGPayload in tez-api 2. the payload is wrapped in the dag plan and used by dag app master 3. the payload is exposed to task communicator through DAG/DAGImpl (4. unrelated: removed a misleading, outdated comment in TezContainerLauncherImpl) > Allow specifying a new payload for plugins when a new DAG starts > > > Key: TEZ-2672 > URL: https://issues.apache.org/jira/browse/TEZ-2672 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: TEZ-2003 >Reporter: Siddharth Seth >Assignee: László Bodor >Priority: Major > Attachments: TEZ-2672.01.patch, TEZ-2672.02.patch > > > This can include information specific to the DAG - constant query > identifiers, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-2672) Allow specifying a new payload for plugins when a new DAG starts
[ https://issues.apache.org/jira/browse/TEZ-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-2672: -- Attachment: TEZ-2672.02.patch > Allow specifying a new payload for plugins when a new DAG starts > > > Key: TEZ-2672 > URL: https://issues.apache.org/jira/browse/TEZ-2672 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: TEZ-2003 >Reporter: Siddharth Seth >Assignee: László Bodor >Priority: Major > Attachments: TEZ-2672.01.patch, TEZ-2672.02.patch > > > This can include information specific to the DAG - constant query > identifiers, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-2672) Allow specifying a new payload for plugins when a new DAG starts
[ https://issues.apache.org/jira/browse/TEZ-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-2672: -- Attachment: (was: TEZ-2672.02.patch) > Allow specifying a new payload for plugins when a new DAG starts > > > Key: TEZ-2672 > URL: https://issues.apache.org/jira/browse/TEZ-2672 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: TEZ-2003 >Reporter: Siddharth Seth >Assignee: László Bodor >Priority: Major > Attachments: TEZ-2672.01.patch > > > This can include information specific to the DAG - constant query > identifiers, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-2672) Allow specifying a new payload for plugins when a new DAG starts
[ https://issues.apache.org/jira/browse/TEZ-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-2672: -- Attachment: TEZ-2672.02.patch > Allow specifying a new payload for plugins when a new DAG starts > > > Key: TEZ-2672 > URL: https://issues.apache.org/jira/browse/TEZ-2672 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: TEZ-2003 >Reporter: Siddharth Seth >Assignee: László Bodor >Priority: Major > Attachments: TEZ-2672.01.patch, TEZ-2672.02.patch > > > This can include information specific to the DAG - constant query > identifiers, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
