[
https://issues.apache.org/jira/browse/TEZ-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15331233#comment-15331233
]
Siddharth Seth commented on TEZ-3216:
-------------------------------------
bq. That will give you the floor value. The code is from DATA_RANGE_IN_MB. If
we were to use the java ceil function, then we have to cast it to double type
and cast the result back to int.
Makes sense.
bq. 16 bits equals to 65GB. This is just in case some apps have really skew
data. As you said, the extra buffer won't stay in AM for too long. Also
protobuf integer encoding might help with RPC size.
If we use all 32bits, would it make sense for this value to be in KB ?
[~rajesh.balamohan] - your thoughts on this from having run lots of jobs ? The
original partition stats range is 0, 1, 10, 100, 1000 (assuming MB level is
adequate ?)
Mostly minor comments.
- The comment about the ShuffleVertexManager. Can you please include a TODO
TEZ-3303 along with the comment.
- The ReportPartitionStats enum. I'd prefer not introducing a separate class
specifically for a single config parameter. This still creates a new class, but
can a simple enum be defined inside TezRuntimeConfiguration, and move the
Helper methods into a separate Utility class which is for private consumption
(and other config parameters and validation) ?
- ReportPartitionStats - Mark "false", "true" as deprecated in favor of the new
values.
- ReportPartitionStats - Documentation for the various values. Some information
on the impact of lossless on RPC / AM memory would be useful. (AM memory would
depend on how a user plugin handles this property. For the ShuffleVertexManager
- this is aggregation)
- Naming of the configs: We're dropping byte level information - so lossless
isn't exactly correct. Maybe "precise" or something else ? Any thoughts on a
name other than 'lossy'. Lossy would typically be seen as a negative - but is
sufficient in a lot of cases, and has the advantage of reduced memory pressure
/ network traffic.
- ReportPartitionStats fromString - Instead of returning null, throw an
IllegalArgumentException if the value specified is none of the expected values
or null.
> Support for more precise partition stats in VertexManagerEvent
> --------------------------------------------------------------
>
> Key: TEZ-3216
> URL: https://issues.apache.org/jira/browse/TEZ-3216
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Ming Ma
> Assignee: Ming Ma
> Attachments: TEZ-3216-2.patch, TEZ-3216.patch
>
>
> Follow up on TEZ-3206 discussion, at least for some use cases, more accurate
> partition stats will be useful for DataMovementEvent routing. Maybe we can
> provide a config option to allow apps to choose the more accurate partition
> stats over RoaringBitmap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
