[
https://issues.apache.org/jira/browse/TEZ-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated TEZ-3216:
-------------------------
Attachment: TEZ-3216-2.patch
Thanks [~sseth]! Reusing the existing config property while maintaining
backward compatibility is a great idea. Here is the new patch that addresses
some of your comments. For other issues you raised,
bq. If both patches end up changing a lot - lets just commit it here
Given we are going to use the existing property, we end up fixing it together.
bq. Is an integer required to report stats in MB ?
16 bits equals to 65GB. This is just in case some apps have really skew data.
As you said, the extra buffer won't stay in AM for too long. Also protobuf
integer encoding might help with RPC size.
bq. Can we use a "long << 20" to get the size in MB ?
That will give you the floor value. The code is from {{DATA_RANGE_IN_MB}}. If
we were to use the java ceil function, then we have to cast it to double type
and cast the result back to int.
> Support for more precise partition stats in VertexManagerEvent
> --------------------------------------------------------------
>
> Key: TEZ-3216
> URL: https://issues.apache.org/jira/browse/TEZ-3216
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Ming Ma
> Assignee: Ming Ma
> Attachments: TEZ-3216-2.patch, TEZ-3216.patch
>
>
> Follow up on TEZ-3206 discussion, at least for some use cases, more accurate
> partition stats will be useful for DataMovementEvent routing. Maybe we can
> provide a config option to allow apps to choose the more accurate partition
> stats over RoaringBitmap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
