Kay Ousterhout created SPARK-3574:
-------------------------------------
Summary: Shuffle finish time always reported as -1
Key: SPARK-3574
URL: https://issues.apache.org/jira/browse/SPARK-3574
Project: Spark
Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Kay Ousterhout
shuffleFinishTime is always reported as -1. I think the way we fix this should
be to set the shuffleFinishTime in each ShuffleWriteMetrics as the shuffles
finish, but when aggregating the metrics, only report shuffleFinishTime as
something other than -1 when *all* of the shuffles have completed.
[~sandyr], it looks like this was introduced in your recent patch to
incrementally report metrics. Any chance you can fix this?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
