Merge pull request #205 from kayousterhout/logging Added logging of scheduler delays to UI
This commit adds two metrics to the UI: 1) The time to get task results, if they're fetched remotely 2) The scheduler delay. When the scheduler starts getting overwhelmed (because it can't keep up with the rate at which tasks are being submitted), the result is that tasks get delayed on the tail-end: the message from the worker saying that the task has completed ends up in a long queue and takes a while to be processed by the scheduler. This commit records that delay in the UI so that users can tell when the scheduler is becoming the bottleneck. Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/3fb302c0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/3fb302c0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/3fb302c0 Branch: refs/heads/scala-2.10 Commit: 3fb302c08d078decd1fa7dd0fc008faff132ab7f Parents: 87676a6 58b3aff Author: Matei Zaharia <[email protected]> Authored: Fri Dec 6 11:03:32 2013 -0800 Committer: Matei Zaharia <[email protected]> Committed: Fri Dec 6 11:03:32 2013 -0800 ---------------------------------------------------------------------- .../org/apache/spark/ui/jobs/StagePage.scala | 36 ++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/3fb302c0/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ----------------------------------------------------------------------
