[
https://issues.apache.org/jira/browse/SPARK-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520432#comment-14520432
]
Tathagata Das edited comment on SPARK-7111 at 4/29/15 10:40 PM:
----------------------------------------------------------------
Here is the design that I propose. If this looks good, then mind implementing
this quickly? :) Spark 1.4 code freeze is really close.
https://docs.google.com/document/d/122QvcwPoLkI2OW4eM7nyBOAqffk2uxgsNT38WI-M5vQ/edit#heading=h.9eluy73ulzuz
Also looping in [~zsxwing] as he ultimately uses {{BatchInfo}} to generate the
updated streaming UI in https://github.com/apache/spark/pull/5533
was (Author: tdas):
Here is the design that I propose. If this looks good, then mind implementing
this quickly? :) Spark 1.4 code freeze is really close.
Also looping in [~zsxwing] as he ultimately uses {{BatchInfo}} to generate the
updated streaming UI in https://github.com/apache/spark/pull/5533
> Add a tracker to track the direct (receiver-less) streams
> ---------------------------------------------------------
>
> Key: SPARK-7111
> URL: https://issues.apache.org/jira/browse/SPARK-7111
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Saisai Shao
>
> Currently for receiver-based input streams, Spark Streaming offers
> ReceiverTracker and ReceivedBlockTracker to track the status of receivers as
> well as block information. Also this status and block information can be
> retrieved from StreamingListener to expose to the users.
> But for direct-based (receiver-less) input streams, Current Spark Streaming
> lacks such mechanism to track the registered direct streams, also lacks the
> way to track the processed number of data for direct-based input streams.
> Here propose a mechanism to track the register direct stream, also expose the
> processing statistics to the BatchInfo and StreamingListener.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
