[
https://issues.apache.org/jira/browse/TEZ-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383472#comment-15383472
]
TezQA commented on TEZ-3356:
----------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12818707/TEZ-3356.1.patch
against master revision a4247a7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1860//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1860//console
This message is automatically generated.
> Fix initializing of stats when custom ShuffleVertexManager is used
> ------------------------------------------------------------------
>
> Key: TEZ-3356
> URL: https://issues.apache.org/jira/browse/TEZ-3356
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Peter Slawski
> Attachments: TEZ-3356.1.patch
>
>
> When using a custom ShuffleVertexManager to set a vertex’s parallelism, the
> partition stats field will be left uninitialized even after the manager
> itself gets initialized. This results in a IllegalStateException to be thrown
> as the stats field will not yet be initialized when VertexManagerEvents are
> processed upon the start of the vertex. Note that these events contain
> partition sizes which are aggregated and stored in this stats field.
>
> Apache Pig’s grace auto-parallelism feature uses a custom
> ShuffleVertexManager which sets a vertex’s parallelism upon the completion of
> one of its parent’s parents. Thus, this corner case is hit and pig scripts
> with grace parallelism enabled would fail if the DAG consists of at least one
> vertex having grandparents.
>
> The fix should be straight forward. Before rather than after
> VertexManagerEvents are processed, simply update pending tasks to ensure the
> partition stats field will be initialized.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
