[
https://issues.apache.org/jira/browse/TEZ-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561348#comment-14561348
]
Siddharth Seth commented on TEZ-2491:
-------------------------------------
The way counters are used right now is fairly inefficient. We haven't really
spent much time optimizing them.
For framework counters, it'd be good to completely avoid sending the name /
display name over RPC and instead rely on a numeric field id. This cuts
serialization/deserialization cost as well. This isn't trivial though - since
counters are part of tez-runtime-library - and can't really be considered to be
framework counters.
Agree with not sending counters which are 0. Publishing Name / Display name
only once.
> Optimize storage and exchange of Counters for better scaling
> ------------------------------------------------------------
>
> Key: TEZ-2491
> URL: https://issues.apache.org/jira/browse/TEZ-2491
> Project: Apache Tez
> Issue Type: Task
> Reporter: Rohini Palaniswamy
>
> Counters take up a lot of space in the task events generated and is a
> major bottleneck for scaling ATS. [~jlowe] found a lot of potential
> optimizations. Using this as an umbrella jira for documentation. Can create
> sub-tasks later after discussions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
