[
https://issues.apache.org/jira/browse/SPARK-9739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-9739:
--------------------------------
Labels: bulk-closed (was: )
> Execution visualizer
> --------------------
>
> Key: SPARK-9739
> URL: https://issues.apache.org/jira/browse/SPARK-9739
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Reporter: Zoltán Zvara
> Priority: Major
> Labels: bulk-closed
>
> Apache Spark, especially its user interface provided by the Web UI component
> lacks a tool that helps to understand the physical plan of the task scheduler
> and the possibility to monitor execution at a very low level, along with the
> communication triggered by data-flow and remote block-requests. We propose a
> tool that would allow users real-time monitoring and later to replay, examine
> job executions on any cluster currently supported by Spark.
> The visualizer we implement would allow users to monitor Spark program’s
> data-flow at task level during execution in the current web user interface
> provided by the master. One would be able to see where executors, tasks get
> deployed on the cluster, along with communication triggered by tasks on a
> representative graph.
> For this, we minimally modify Spark’s core to be able to collect information
> related to block requests. Slight modification and evident refactoring
> impacts the tasks’ code to allow reporting of execution state to the driver’s
> monitoring object, which has been added to SparkContext. Most aspect of the
> proposed module are configurable.
> Our execution-visualizer would not raise any measurable performance impact on
> Spark programs, but would introduce the following benefits.
> *Benefits*
> We think the execution-visualizer would give the following benefits to
> end-users:
> - understand the execution mechanism of Spark and demonstrate how executors,
> tasks work internally, which would attract new users;
> - provided by the advanced visual monitoring of programs, the ability to
> discover issues of executors and tasks in a more detailed and convenient way;
> - the possibility to highlight inefficient communication patterns of certain
> workflows, that would add insight to advanced optimization strategies.
> *Implementation*
> We modified tasks to send more detailed information to the driver before and
> after their effective work, which we collect as JSON on the driver’s file
> system. The logs would be read on every interval by the visualizer written
> using the D3 JavaScript library. The visualizer would provide the following
> main features:
> - dynamically show hosts, executors, tasks currently running and finishing in
> a graph;
> - show critical and additional backend information related to hosts,
> executors (along with available resources);
> - show useful information about running tasks: RDD and split to compute,
> dependencies, stages and others;
> - show failed executors and tasks;
> - show task metrics and provide multiple ways to summarize;
> - show communication as directed edges between executors in form of block
> requests;
> - let the user to replay executions in a different speed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
