Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10815#discussion_r50055832
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
---
@@ -102,15 +102,38 @@ class TaskMetrics extends Serializable {
private[spark] def incDiskBytesSpilled(value: Long): Unit =
_diskBytesSpilled += value
private[spark] def decDiskBytesSpilled(value: Long): Unit =
_diskBytesSpilled -= value
- /**
- * If this task reads from a HadoopRDD or from persisted data, metrics
on how much data was read
- * are stored here.
- */
private var _inputMetrics: Option[InputMetrics] = None
+ /**
+ * Metrics related to reading data from a
[[org.apache.spark.rdd.HadoopRDD]] or from persisted
+ * data, defined only in tasks with input.
+ */
def inputMetrics: Option[InputMetrics] = _inputMetrics
/**
+ * Get or create a new [[InputMetrics]] associated with this task.
+ */
+ private[spark] def registerInputMetrics(readMethod:
DataReadMethod.Value): InputMetrics = {
+ synchronized {
+ val metrics = _inputMetrics.getOrElse {
+ val metrics = new InputMetrics(readMethod)
+ _inputMetrics = Some(metrics)
+ metrics
+ }
+ // If there already exists an InputMetric with the same read method,
we can just return
+ // that one. Otherwise, if the read method is different from the one
previously seen by
+ // this task, we return a new dummy one to avoid clobbering the
values of the old metrics.
+ // In the future we should try to store input metrics from all
different read methods at
+ // the same time (SPARK-5225).
--- End diff --
Before, I guess we'd unconditionally overwrite so we might handle it wrong
even for the same read method case? This looks good to me, just curious RE:
whether this fixed another bug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
