GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/290
[SPARK-1386] Web UI for Spark Streaming [WIP]
When debugging Spark Streaming applications it is necessary to monitor
certain metrics that are not shown in the Spark application UI. For example,
what is average processing time of batches? What is the scheduling delay? Is
the system able to process as fast as it is receiving data? How many records I
am receiving through my receivers?
While the StreamingListener interface introduced in the 0.9 provided some
of this information, it could only be accessed programmatically. A UI that
shows information specific to the streaming applications is necessary for
easier debugging. This PR introduces such a UI. It shows various statistics
related to the streaming application. Here is a screenshot of the UI running on
my local machine.
http://i.imgur.com/Sf9TnG5.png
This is still a WIP. The UI current runs on a different port (6060). We do
not want to make everyone open a new port in their firewalls, so we would like
to integrate this UI into the Spark UI running at 4040. This is still to be
done.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark streaming-web-ui
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/290.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #290
----
commit 56cc7fbcaf04a5aab88296d20da2cfc5b84a7651
Author: Tathagata Das <[email protected]>
Date: 2014-03-28T21:45:46Z
First cut implementation of Streaming UI.
commit 93f1c69e067fb02bcbb1dcab93d1dff4905c2e17
Author: Tathagata Das <[email protected]>
Date: 2014-03-31T23:31:48Z
Added network receiver information to the Streaming UI.
commit 4d86e985cb7bbc7f4f125e52d72f4e4bd560677e
Author: Tathagata Das <[email protected]>
Date: 2014-04-01T18:02:23Z
Added basic stats to the StreamingUI and refactored the UI to a Page to
make it easier to transition to using SparkUI later.
commit db27bad1c781345a4bd0de6003e1a8a10508e024
Author: Tathagata Das <[email protected]>
Date: 2014-04-01T20:23:29Z
Added last batch processing time to StreamingUI.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
