[ https://issues.apache.org/jira/browse/SPARK-30108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991381#comment-16991381 ]
Herman van Hövell commented on SPARK-30108: ------------------------------------------- [~Ankitraj] Great to hear! Here are some pointers, the goal of this ticket to make the current AggregatingAccumulator to only add the result of a partition once. For this we need to do a couple of things: - We need to add some tracking mechanism to the AggregatingAccumulator that keeps track of the partitions that have been processed. A (compressible) bit map should be enough. - We need a way to tell the accumulator's merge method which partition it is processing. One way of doing this would be making a small change in the way the DAGScheduler interacts with the accumulator, and also pass Task/Partition info on calling update. A less intrusive way of doing this would be to pass back partition info with the to-be-merged accumulator. > Add robust accumulator for observable metrics > --------------------------------------------- > > Key: SPARK-30108 > URL: https://issues.apache.org/jira/browse/SPARK-30108 > Project: Spark > Issue Type: New Feature > Components: SQL > Affects Versions: 3.0.0 > Reporter: Herman van Hövell > Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org