[jira] [Commented] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467994#comment-16467994 ] Vineet Garg commented on TEZ-3911: -- Latest patch (007) addresses review comment. > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch, > TEZ-3911.003.patch, TEZ-3911.004.patch, TEZ-3911.005.patch, > TEZ-3911.006.patch, TEZ-3911.007.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated TEZ-3911: - Attachment: TEZ-3911.007.patch > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch, > TEZ-3911.003.patch, TEZ-3911.004.patch, TEZ-3911.005.patch, > TEZ-3911.006.patch, TEZ-3911.007.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated TEZ-3911: - Attachment: TEZ-3911.006.patch > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch, > TEZ-3911.003.patch, TEZ-3911.004.patch, TEZ-3911.005.patch, TEZ-3911.006.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated TEZ-3911: - Attachment: TEZ-3911.005.patch > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch, > TEZ-3911.003.patch, TEZ-3911.004.patch, TEZ-3911.005.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated TEZ-3911: - Attachment: TEZ-3911.004.patch > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch, > TEZ-3911.003.patch, TEZ-3911.004.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464218#comment-16464218 ] Vineet Garg commented on TEZ-3911: -- [~gopalv] Did you get a chance to take a look at it? Can you see if my approach is correct or does it need change? > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch, > TEZ-3911.003.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461760#comment-16461760 ] Vineet Garg commented on TEZ-3911: -- [~ewohlstadter] That was my mistake initializing it to 0. I have fixed that in the latest patch. Thanks for pointing it out. Latest patch(3) adds config flag as well as apis to retrieve min/max. [~gopalv] suggested to not use config flag and enable new aggregations by default. [~gopalv] Can you expand on your second comment about adding extra abstract class to handle this. It sounds like my approach might not be correct since. Please take a look at the latest patch I uploaded. > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch, > TEZ-3911.003.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated TEZ-3911: - Attachment: TEZ-3911.003.patch > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch, > TEZ-3911.003.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[
https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457082#comment-16457082
]
Vineet Garg commented on TEZ-3911:
--
[~ashutoshc] I plan to add config in {{VertexImpl::constructStatistics}}. This
config will control {{aggregateAllCounters}} call. This patch doesn't yet
provide getMin/getMax apis to retrieve min/max on TezCounter.
bq. Also, there is no 'avg' aggregation. I think sum(counter)/(number of tasks)
as avg would also be useful.
Isn't this trivial to compute by whomever is using the APIs? The reason we are
baking in min/max is so that consumers like History Logging service wouldn't
have to loop over task's counters to do so. Let me know if you still think avg
would be useful. That API probably should be added separately on Dag level if
we decided to implement it cc [~ewohlstadter] [~gopalv]
> Optional min/max/avg aggr. task counters reported to HistoryLoggingService at
> final counter aggr.
> -
>
> Key: TEZ-3911
> URL: https://issues.apache.org/jira/browse/TEZ-3911
> Project: Apache Tez
> Issue Type: New Feature
>Reporter: Eric Wohlstadter
>Assignee: Vineet Garg
>Priority: Critical
> Fix For: 0.9.next
>
> Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch
>
>
> Consumers of HistoryLoggingService reported counters are currently required
> to compute any task-level aggregations other than "sum". This is inefficient
> as Tez is already "scanning" over this data. Computing incremental aggregates
> shouldn't require additional scans by ATS consumers.
> Provide an option for Task counter aggregations other than "sum". Computation
> of these extra counters can be turned on/off.
> The option will generate "synthetic" counters at final aggregation time for
> reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS.
> Only incremental aggregations will be supported (min/max/avg). Aggregation
> computation will be folded into the existing "aggregation loop" beginning at
> VertexImpl.incrTaskCounters.
> Extra aggregations will only be supported during final counter aggregation.
> Aggregations will only include the "bestAttempt" for each task.
> A design doc will be provided.
> Because final task aggregation holds a lock, a performance report will be
> provided.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated TEZ-3911: - Attachment: TEZ-3911.002.patch > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch, TEZ-3911.002.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455518#comment-16455518 ] Vineet Garg commented on TEZ-3911: -- Thanks [~jeagles] > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned TEZ-3911: Assignee: Vineet Garg (was: Eric Wohlstadter) > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Vineet Garg >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453475#comment-16453475 ] Vineet Garg commented on TEZ-3911: -- [~ewohlstadter] Can you assign this to me? I am unable to edit this jira. Looks like I don't have permissions. > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated TEZ-3911: - Attachment: TEZ-3911.001.patch > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453474#comment-16453474 ] Vineet Garg commented on TEZ-3911: -- Attaching initial patch to get test run. > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.9.next > > Attachments: TEZ-3911.001.patch > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3911) Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
[ https://issues.apache.org/jira/browse/TEZ-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453470#comment-16453470 ] Vineet Garg commented on TEZ-3911: -- [~ewohlstadter] [~t3rmin4t0r] Can you take a look at my first attempt for this at https://github.com/apache/tez/compare/master...vineetgarg02:TEZ-3911 and provide feedback? I would like to know if I am going at right track. This implementation has yet to implement the following: * Config flag to control extra aggregation (min/max) * Test coverage for new aggregation * methods to retrieve min/max from counters. Looking forward to your feedback. > Optional min/max/avg aggr. task counters reported to HistoryLoggingService at > final counter aggr. > - > > Key: TEZ-3911 > URL: https://issues.apache.org/jira/browse/TEZ-3911 > Project: Apache Tez > Issue Type: New Feature >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.9.next > > > Consumers of HistoryLoggingService reported counters are currently required > to compute any task-level aggregations other than "sum". This is inefficient > as Tez is already "scanning" over this data. Computing incremental aggregates > shouldn't require additional scans by ATS consumers. > Provide an option for Task counter aggregations other than "sum". Computation > of these extra counters can be turned on/off. > The option will generate "synthetic" counters at final aggregation time for > reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. > Only incremental aggregations will be supported (min/max/avg). Aggregation > computation will be folded into the existing "aggregation loop" beginning at > VertexImpl.incrTaskCounters. > Extra aggregations will only be supported during final counter aggregation. > Aggregations will only include the "bestAttempt" for each task. > A design doc will be provided. > Because final task aggregation holds a lock, a performance report will be > provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
