Gopal V commented on TEZ-2161:

bq. DAG aggregates Vertices which aggregates Tasks which chooses "bestAttempt". 
And the whole thing runs in various locks. This getAllCounters() flow executes 
locally on the AM.

CRDT was more for the P-N counter implementation for aggregates which store 
both the -ve and +ve movements of the counter.

bq. My plan is to add "aggregateAllCounters" to the CounterGroup classes, which 
will be used similarly to "incrAllCounters", except instead of only doing SUM, 
it also does MIN, AVG, MAX.

The Counter needs sub-classes which declare what it needs to aggregate on - 
adding fields to every counter will break everything downstream that exists 

Adding a MAX_GC_MILLIS counter with new semantics explicitly is better than 
messing with the existing GC_MILLIS counter.

> Support CRDT aggregation models for counters 
> ---------------------------------------------
>                 Key: TEZ-2161
>                 URL: https://issues.apache.org/jira/browse/TEZ-2161
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Hitesh Shah
>            Assignee: Eric Wohlstadter
>            Priority: Major
> Some counters such as last event received time need to be handled different 
> to say bytes read counters. Bytes reads requires a summation across all tasks 
> within a vertex. The received time requires doing a max() across all the 
> tasks. First event received time would likely need a min().

This message was sent by Atlassian JIRA

Reply via email to