GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/4805
[SPARK-6051][Streaming] Add ZooKeeper offest posting for
DirectKafkaInputDStream
Currently in DirectKafkaInputDStream, offset is managed by Spark Streaming
itself without ZK or Kafka involved, which will make several third-party offset
monitoring tools fail to monitor the status of Kafka consumer. So here add an
option to post the offset to ZK when each job is finished
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark kafkaListener
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4805.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4805
----
commit fb19f20bc8148c178530af8c649f16505ce655e8
Author: Saisai Shao <[email protected]>
Date: 2015-02-26T22:45:26Z
Add ZooKeeper offest store for DirectKafkaInputDStream
commit 0ac450af825ca3ec71cacc786aa01a4005177e03
Author: Saisai Shao <[email protected]>
Date: 2015-02-27T05:31:36Z
Bug fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
