Eric Yang created SPARK-51299:
---------------------------------
Summary: MetricUtils.stringValue should use initValue to filter
out invalid metric values
Key: SPARK-51299
URL: https://issues.apache.org/jira/browse/SPARK-51299
Project: Spark
Issue Type: Bug
Components: SQL, Web UI
Affects Versions: 4.0.0
Reporter: Eric Yang
In method `org.apache.spark.util.MetricUtils.stringValue`, it uses a hardcoded
value 0 to filter out invalid metric values for SIZE_METRIC, TIMING_METRIC and
NS_TIMING_METRIC:
{code:java}
val validValues = values.filter(_ >= 0) {code}
However, we offer methods to create these types of metrics with initValue other
than -1:
{code:java}
def createSizeMetric(sc: SparkContext, name: String, initValue: Long = -1):
SQLMetric = {
... {code}
which means there is a chance that the metrics are created with a initValue !=
-1 and in this case the filter above will generate incorrect results.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
