[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[
https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200713#comment-15200713
]
TezQA commented on TEZ-3166:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12794091/TEZ-3166.branch-0.7.second.patch
against master revision 44c660a.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1575//console
This message is automatically generated.
> Counters aren't fully updated and sent for failed tasks
> ---
>
> Key: TEZ-3166
> URL: https://issues.apache.org/jira/browse/TEZ-3166
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch, TEZ-3166.3.patch,
> TEZ-3166.branch-0.7.patch, TEZ-3166.branch-0.7.second.patch
>
>
> On master branch, if an application fails very soon, counters aren't fully
> updated so on tez-ui we can only see part of counters. We should publish all
> counters even task attempt fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[
https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198721#comment-15198721
]
TezQA commented on TEZ-3166:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12793871/TEZ-3166.3.patch
against master revision 42b61f4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestRecovery
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1569//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1569//console
This message is automatically generated.
> Counters aren't fully updated and sent for failed tasks
> ---
>
> Key: TEZ-3166
> URL: https://issues.apache.org/jira/browse/TEZ-3166
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch, TEZ-3166.3.patch
>
>
> On master branch, if an application fails very soon, counters aren't fully
> updated so on tez-ui we can only see part of counters. We should publish all
> counters even task attempt fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[
https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200325#comment-15200325
]
TezQA commented on TEZ-3166:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12793871/TEZ-3166.3.patch
against master revision 191447e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1572//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1572//console
This message is automatically generated.
> Counters aren't fully updated and sent for failed tasks
> ---
>
> Key: TEZ-3166
> URL: https://issues.apache.org/jira/browse/TEZ-3166
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch, TEZ-3166.3.patch
>
>
> On master branch, if an application fails very soon, counters aren't fully
> updated so on tez-ui we can only see part of counters. We should publish all
> counters even task attempt fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[ https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1521#comment-1521 ] Siddharth Seth commented on TEZ-3166: - +1 for the latest patch. [~aplusplus] - could you please post a patch for branch-0.7 as well. It's not direct backport since TezTaskRunner2 does not exist there. > Counters aren't fully updated and sent for failed tasks > --- > > Key: TEZ-3166 > URL: https://issues.apache.org/jira/browse/TEZ-3166 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch, TEZ-3166.3.patch > > > On master branch, if an application fails very soon, counters aren't fully > updated so on tez-ui we can only see part of counters. We should publish all > counters even task attempt fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[
https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198592#comment-15198592
]
TezQA commented on TEZ-3166:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12793871/TEZ-3166.3.patch
against master revision 42b61f4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1567//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1567//console
This message is automatically generated.
> Counters aren't fully updated and sent for failed tasks
> ---
>
> Key: TEZ-3166
> URL: https://issues.apache.org/jira/browse/TEZ-3166
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Zhiyuan Yang
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch, TEZ-3166.3.patch
>
>
> On master branch, if an application fails very soon, counters aren't fully
> updated so on tez-ui we can only see part of counters. We should publish all
> counters even task attempt fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[ https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198361#comment-15198361 ] Zhiyuan Yang commented on TEZ-3166: --- Thanks for taking time review my patch! Already uploaded a new one and created the JIRA regarding to CONTAINER_STOP_REQUESTED case. > Counters aren't fully updated and sent for failed tasks > --- > > Key: TEZ-3166 > URL: https://issues.apache.org/jira/browse/TEZ-3166 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch, TEZ-3166.3.patch > > > On master branch, if an application fails very soon, counters aren't fully > updated so on tez-ui we can only see part of counters. We should publish all > counters even task attempt fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[ https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200623#comment-15200623 ] Zhiyuan Yang commented on TEZ-3166: --- Last patch has problem with tasks that fail due to processor class not found error. In this case, we are not able to get their counters because tasks are not fully initialized, and this behavior is specified by TaskReported test. So I upload another patch which only verify counters when task fails during execution instead of initialization. > Counters aren't fully updated and sent for failed tasks > --- > > Key: TEZ-3166 > URL: https://issues.apache.org/jira/browse/TEZ-3166 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch, TEZ-3166.3.patch, > TEZ-3166.branch-0.7.patch, TEZ-3166.branch-0.7.second.patch > > > On master branch, if an application fails very soon, counters aren't fully > updated so on tez-ui we can only see part of counters. We should publish all > counters even task attempt fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[ https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200497#comment-15200497 ] Zhiyuan Yang commented on TEZ-3166: --- For branch-0.7, counters are always fully updated and sent. I've uploaded TEZ-3166.branch-0.7.patch which doesn't modify code but add test to make sure counter issue won't exists. > Counters aren't fully updated and sent for failed tasks > --- > > Key: TEZ-3166 > URL: https://issues.apache.org/jira/browse/TEZ-3166 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch, TEZ-3166.3.patch, > TEZ-3166.branch-0.7.patch > > > On master branch, if an application fails very soon, counters aren't fully > updated so on tez-ui we can only see part of counters. We should publish all > counters even task attempt fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[ https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197741#comment-15197741 ] Siddharth Seth commented on TEZ-3166: - [~aplusplus] - there's a couple more cases which need to be handled. In TaskRunner2Callable - it's possible that the task ends before the close. Look for cases where TaskRunner2Result is being returned. The counters should be updated in these cases as well. This will mostly occur when the AM asks the task to die - preempted for instance. That needs to be handled as well. TezTaskRunner2: case CONTAINER_STOP_REQUESTED:. I don't think we should try fixing this in this jira, since there's a bunch of AM side changes involved (The AM has essentially stopped tracking this task at this point, and will not accept another heartbeat request). The cases where the task fails itself will be covered by this jira. For this one, could you please update TaskRunner2Result with the counters being updated for the additional cases, and file a new jira to handle external events to the task. > Counters aren't fully updated and sent for failed tasks > --- > > Key: TEZ-3166 > URL: https://issues.apache.org/jira/browse/TEZ-3166 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch > > > On master branch, if an application fails very soon, counters aren't fully > updated so on tez-ui we can only see part of counters. We should publish all > counters even task attempt fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3166) Counters aren't fully updated and sent for failed tasks
[ https://issues.apache.org/jira/browse/TEZ-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200700#comment-15200700 ] Siddharth Seth commented on TEZ-3166: - +1. Committing. Thanks [~aplusplus]. I'm going to create a follow up to potentially handle updates for a partly initialized task. > Counters aren't fully updated and sent for failed tasks > --- > > Key: TEZ-3166 > URL: https://issues.apache.org/jira/browse/TEZ-3166 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3166.1.patch, TEZ-3166.2.patch, TEZ-3166.3.patch, > TEZ-3166.branch-0.7.patch, TEZ-3166.branch-0.7.second.patch > > > On master branch, if an application fails very soon, counters aren't fully > updated so on tez-ui we can only see part of counters. We should publish all > counters even task attempt fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
