[
https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15363072#comment-15363072
]
Ming Ma commented on TEZ-3303:
------------------------------
Thanks [~ozawa]. When detailed partition stat is set, the memory_optimized stat
won't be set. Thus the following function should be moved out of the
proto.hasPartitionStats() check.
{noformat}
if (proto.hasDetailedPartitionStats()) {
List<Integer> detailedPartitionStats =
proto.getDetailedPartitionStats().getSizeInMbList();
parseDetailedPartitionStats(detailedPartitionStats);
}
{noformat}
This also means TestShuffleVertexManager#getVertexManagerEvent doesn't need to
set memory_optimized partition stat when reportDetailedStats is true.
Others are just nits:
* The following isn't referred in ShuffleVertexManager.
{noformat}
import org.apache.tez.runtime.library.shuffle.impl.ShuffleUserPayloads
{noformat}
* Typo "deatiled" in TestShuffleVertexManager
* Might be easier if the test code uses 963MB instead of "1010 * 1000l * 1000l".
* The following comment is no longer valid. Maybe we can just remove it.
{noformat}
Assert.assertEquals(964, manager.stats[2]); //100 MB bucket
{noformat}
> Have ShuffleVertexManager consume more precise partition stats
> --------------------------------------------------------------
>
> Key: TEZ-3303
> URL: https://issues.apache.org/jira/browse/TEZ-3303
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Ming Ma
> Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-3303.001.patch, TEZ-3303.002.patch,
> TEZ-3303.002.patch
>
>
> TEZ-3216 adds the support for more precise partition stats.
> ShuffleVertexManager should be updated to consume the more precise partition
> stats.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
