[
https://issues.apache.org/jira/browse/SPARK-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tathagata Das resolved SPARK-7111.
----------------------------------
Resolution: Fixed
Fix Version/s: 1.4.0
> Exposing of input data rates of non-receiver streams like Kafka Direct stream
> -----------------------------------------------------------------------------
>
> Key: SPARK-7111
> URL: https://issues.apache.org/jira/browse/SPARK-7111
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Saisai Shao
> Assignee: Saisai Shao
> Priority: Blocker
> Fix For: 1.4.0
>
>
> Currently for receiver-based input streams, Spark Streaming offers
> ReceiverTracker and ReceivedBlockTracker to track the status of receivers as
> well as block information. Also this status and block information can be
> retrieved from StreamingListener to expose to the users.
> But for direct-based (receiver-less) input streams, Current Spark Streaming
> lacks such mechanism to track the registered direct streams, also lacks the
> way to track the processed number of data for direct-based input streams.
> Here propose a mechanism to track the register direct stream, also expose the
> processing statistics to the BatchInfo and StreamingListener.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
