I think it would be very useful to have this. We could put the ui display either behind a flag or a url parameter
Shivaram On Wed, Jul 9, 2014 at 12:25 PM, Reynold Xin <r...@databricks.com> wrote: > Maybe it's time to create an advanced mode in the ui. > > > On Wed, Jul 9, 2014 at 12:23 PM, Kay Ousterhout <k...@eecs.berkeley.edu> > wrote: > > > Hi all, > > > > I've been doing a bunch of performance measurement of Spark and, as part > of > > doing this, added metrics that record the average CPU utilization, disk > > throughput and utilization for each block device, and network throughput > > while each task is running. These metrics are collected by reading the > > /proc filesystem so work only on Linux. I'm happy to submit a pull > request > > with the appropriate changes but first wanted to see if sufficiently many > > people think this would be useful. I know the metrics reported by Spark > > (and in the UI) are already overwhelming to some folks so don't want to > add > > more instrumentation if it's not widely useful. > > > > These metrics are slightly more difficult to interpret for Spark than > > similar metrics reported by Hadoop because, with Spark, multiple tasks > run > > in the same JVM and therefore as part of the same process. This means > > that, for example, the CPU utilization metrics reflect the CPU use across > > all tasks in the JVM, rather than only the CPU time used by the > particular > > task. This is a pro and a con -- it makes it harder to determine why > > utilization is high (it may be from a different task) but it also makes > the > > metrics useful for diagnosing straggler problems. Just wanted to clarify > > this before asking folks to weigh in on whether the added metrics would > be > > useful. > > > > -Kay > > > > (if you're curious, the instrumentation code is on a very messy branch > > here: > > > > > https://github.com/kayousterhout/spark-1/tree/proc_logging_perf_minimal_temp/core/src/main/scala/org/apache/spark/performance_logging > > ) > > >
