Song Jun created SPARK-28332: -------------------------------- Summary: SQLMetric wrong initValue Key: SPARK-28332 URL: https://issues.apache.org/jira/browse/SPARK-28332 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.0.0 Reporter: Song Jun
Currently SQLMetrics.createSizeMetric create a SQLMetric with initValue set to -1. If there is a ShuffleMapStage with lots of Tasks which read 0 bytes data, these tasks will send the metric(the metric value still be the initValue with -1) to Driver, then Driver do metric merge for this Stage in DAGScheduler.updateAccumulators, this will cause the merged metric value of this Stage set to be a negative value. This is incorrect, we should set the initValue to 0 . Another same case in SQLMetrics.createTimingMetric. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org