[
https://issues.apache.org/jira/browse/SPARK-6051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048895#comment-15048895
]
Cody Koeninger commented on SPARK-6051:
---------------------------------------
Responded on the mailing list, but for posterity's sake:
Which version of spark are you on? I thought that was added to the spark UI in
recent versions.
DIrect api doesn't have any inherent interaction with zookeeper. If you need
number of messages per batch and aren't on a recent enough version of spark to
see them in the ui, you can get them programmatically from the offset ranges.
See the definition of count() in recent versions of KafkaRDD for an example.
> Add an option for DirectKafkaInputDStream to commit the offsets into ZK
> -----------------------------------------------------------------------
>
> Key: SPARK-6051
> URL: https://issues.apache.org/jira/browse/SPARK-6051
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.3.0
> Reporter: Saisai Shao
>
> Currently in DirectKafkaInputDStream, offset is managed by Spark Streaming
> itself without ZK or Kafka involved, which will make several third-party
> offset monitoring tools fail to monitor the status of Kafka consumer. So here
> as a option to commit the offset to ZK when each job is finished, the process
> is implemented as a asynchronized way, so the main processing flow will not
> be blocked, already tested with KafkaOffsetMonitor tools.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
