[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711829#comment-16711829
]
Gopal V commented on TEZ-3957:
--
LGTM - +1
This counter is aggregated for capacity planning models with container reuse.
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.02.patch, TEZ-3957.02.patch,
> TEZ-3957.03.patch, TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709350#comment-16709350
]
TezQA commented on TEZ-3957:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} | {color:orange} tez-runtime-internals: The patch generated 3 new
+ 397 unchanged - 0 fixed = 400 total (was 397) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} tez-dag: The patch generated 6 new + 507
unchanged - 0 fixed = 513 total (was 507) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
47s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
44s{color} | {color:green} tez-runtime-internals in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
40s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/tez:d4a62de |
| JIRA Issue | TEZ-3957 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12950600/TEZ-3957.03.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux 1c02b687fd16 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | master / 282bb0a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.0.1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/67/artifact/out/diff-checkstyle-tez-runtime-internals.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/67/artifact/out/diff-checkstyle-tez-dag.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/67/testReport/ |
| Max. process+thread count | 223 (vs. ulimit of 1) |
| modules | C: tez-api
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677669#comment-16677669
]
Gopal V commented on TEZ-3957:
--
{code}
[ERROR] Process Exit Code: 1
[ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /testptch/tez/tez-runtime-internals &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx1024m
-XX:+HeapDumpOnOutOfMemoryError -jar
/testptch/tez/tez-runtime-internals/target/surefire/surefirebooter2594127067763002469.jar
/testptch/tez/tez-runtime-internals/target/surefire
2018-11-07T04-40-29_190-jvmRun1 surefire5873371651625085212tmp
surefire_91017432904624962456tmp
[ERROR] Error occurred in starting fork, check output in log
{code}
Looks like openjdk8 isn't working on these machines?
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.02.patch, TEZ-3957.02.patch,
> TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677665#comment-16677665
]
TezQA commented on TEZ-3957:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} tez-runtime-internals: The patch generated 3 new
+ 397 unchanged - 0 fixed = 400 total (was 397) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} tez-dag: The patch generated 6 new + 507
unchanged - 0 fixed = 513 total (was 507) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 28s{color}
| {color:red} tez-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s{color}
| {color:red} tez-runtime-internals in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 40s{color}
| {color:red} tez-dag in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/tez:d4a62de |
| JIRA Issue | TEZ-3957 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947178/TEZ-3957.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux a6ebc9251000 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | master / efc7331 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.0.1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/56/artifact/out/diff-checkstyle-tez-runtime-internals.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/56/artifact/out/diff-checkstyle-tez-dag.txt
|
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/56/artifact/out/patch-unit-tez-api.txt
|
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/56/artifact/out/patch-unit-tez-runtime-internals.txt
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677629#comment-16677629
]
Sergey Shelukhin commented on TEZ-3957:
---
Hmm.. all the tests failed to fork wm and don't appear to repro locally. I'm
running all the tests now.
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.02.patch, TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677600#comment-16677600
]
TezQA commented on TEZ-3957:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} tez-runtime-internals: The patch generated 3 new
+ 397 unchanged - 0 fixed = 400 total (was 397) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} tez-dag: The patch generated 6 new + 507
unchanged - 0 fixed = 513 total (was 507) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 30s{color}
| {color:red} tez-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 23s{color}
| {color:red} tez-runtime-internals in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 39s{color}
| {color:red} tez-dag in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/tez:d4a62de |
| JIRA Issue | TEZ-3957 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947164/TEZ-3957.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux 53d949612887 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | master / efc7331 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.0.1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/55/artifact/out/diff-checkstyle-tez-runtime-internals.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/55/artifact/out/diff-checkstyle-tez-dag.txt
|
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/55/artifact/out/patch-unit-tez-api.txt
|
| unit |
https://builds.apache.org/job/PreCommit-TEZ-Build/55/artifact/out/patch-unit-tez-runtime-internals.txt
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677567#comment-16677567
]
Sergey Shelukhin commented on TEZ-3957:
---
Updated
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.02.patch, TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677425#comment-16677425
]
Gopal V commented on TEZ-3957:
--
[~sershe]: can you fix the NS_TO_MS with the java time conversions?
MapReduce would not need a similar counter, the container lifetimes can be used
as a proxy for this in wall-clock seconds, but with Tez we're moved down from
the second to the millisecond time-frame where the ntpd slew would be visible.
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669397#comment-16669397
]
Eric Wohlstadter commented on TEZ-3957:
---
lgtm (unbinding)
[~jeagles]
I'm pretty sure MR doesn't have this counter (at least as of 4 years ago).
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667578#comment-16667578
]
Sergey Shelukhin commented on TEZ-3957:
---
CPU milliseconds is CPU time, this reports wall clock time.
No idea about MR.
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667570#comment-16667570
]
Jonathan Eagles commented on TEZ-3957:
--
What does this new counter offer above TaskCounter.CPU_MILLISECONDS? Also, does
MapReduce provide this counter?
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665845#comment-16665845
]
TezQA commented on TEZ-3957:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange} tez-runtime-internals: The patch generated 4 new
+ 397 unchanged - 0 fixed = 401 total (was 397) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} tez-dag: The patch generated 6 new + 507
unchanged - 0 fixed = 513 total (was 507) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
41s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} tez-runtime-internals in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
7s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | TEZ-3957 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945829/TEZ-3957.01.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-143-generic #192-Ubuntu SMP Tue
Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus/precommit/personality/tez.sh
|
| git revision | master / d4a62de |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_172 |
| findbugs | v3.0.1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/48/artifact/out/diff-checkstyle-tez-runtime-internals.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/48/artifact/out/diff-checkstyle-tez-dag.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/48/testReport/ |
| modules | C: tez-api tez-runtime-internals tez-dag U: . |
| Console output | https://builds.apache.org/job/PreCommit-TEZ-Build/48/console
|
| Powered by | Apache Yetus 0.8.0
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665699#comment-16665699
]
Sergey Shelukhin commented on TEZ-3957:
---
I'm going to fix some but not all checkstyle warnings... some of them don't
make any sense (e.g. DesignForExtension -
http://checkstyle.sourceforge.net/config_design.html#DesignForExtension
mentions that it only makes sense for library projects, as is it prevents
normal method overrides - perhaps it should be disabled in a separate patch.),
some diffing seems to be buggy (complaining about lack of javadoc for a field
that was already there and without javadoc).
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664725#comment-16664725
]
TezQA commented on TEZ-3957:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} tez-api: The patch generated 3 new + 65
unchanged - 1 fixed = 68 total (was 66) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} tez-runtime-internals: The patch generated 8 new
+ 295 unchanged - 0 fixed = 303 total (was 295) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} tez-dag: The patch generated 14 new + 507
unchanged - 0 fixed = 521 total (was 507) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
45s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 42s{color}
| {color:red} tez-runtime-internals in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
44s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | tez.runtime.task.TestTaskExecution2 |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | TEZ-3957 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945696/TEZ-3957.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-144-generic #193-Ubuntu SMP Thu
Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus/precommit/personality/tez.sh
|
| git revision | master / 79af4e8 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_172 |
| findbugs | v3.0.1 |
| checkstyle |
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664655#comment-16664655
]
Eric Wohlstadter commented on TEZ-3957:
---
[~sershe]
Do you want the counter to be continuously updated and reported back to the AM
from running task containers?
I was planning to just add the counter at DAG completion time, from information
in the AM about completed tasks. That's what I meant by "Report
TASK_DURATION_MILLIS as a Counter for completed tasks".
What I had in mind is less useful in general than what I understand this patch
is doing, but it doesn't add any overhead to AM <-> Container messaging.
Wanted to check that there is a use-case where having this as a running counter
is useful (it wasn't necessary for my use-case).
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517604#comment-16517604
]
Eric Wohlstadter commented on TEZ-3957:
---
I realize we can't just keep adding Counters for everything, but this one seems
like it would be a very common use-case.
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
