Song Jun created SPARK-28332:
--------------------------------
Summary: SQLMetric wrong initValue
Key: SPARK-28332
URL: https://issues.apache.org/jira/browse/SPARK-28332
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: Song Jun
Currently SQLMetrics.createSizeMetric create a SQLMetric with initValue set to
-1.
If there is a ShuffleMapStage with lots of Tasks which read 0 bytes data, these
tasks will send the metric(the metric value still be the initValue with -1) to
Driver, then Driver do metric merge for this Stage in
DAGScheduler.updateAccumulators, this will cause the merged metric value of
this Stage set to be a negative value.
This is incorrect， we should set the initValue to 0 .
Another same case in SQLMetrics.createTimingMetric.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
