[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453470#comment-16453470 ]
Vineet Garg commented on TEZ-3911: ---------------------------------- [~ewohlstadter] [~t3rmin4t0r] Can you take a look at my first attempt for this at https://github.com/apache/tez/compare/master...vineetgarg02:TEZ-3911 and provide feedback? I would like to know if I am going at right track. This implementation has yet to implement the following: * Config flag to control extra aggregation (min/max) * Test coverage for new aggregation * methods to retrieve min/max from counters. Looking forward to your feedback. > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > ------------------------------------------------------------------------------------------------- > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature > Reporter: Eric Wohlstadter > Assignee: Eric Wohlstadter > Priority: Critical > Fix For: 0.9.next > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)