GitHub user koeninger opened a pull request:
https://github.com/apache/spark/pull/11863
[SPARK-12177][Streaming][Kafka] Update KafkaDStreams to new Kafka 0.9
Consumer API
for discussion, DO NOT MERGE, blocked by SPARK-13877
## What changes were proposed in this pull request?
New Kafka consumer api.
## How was this patch tested?
Unit tests, manual tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/koeninger/spark-1 kafka-0.9
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11863.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11863
----
commit aaef0fc6e7e3aae18e4e03271bc0707d09d243e4
Author: cody koeninger <[email protected]>
Date: 2016-03-03T22:55:16Z
[SPARK-12177][Streaming][Kafka] separate project for starting work on beta
consumer, modify getPreferredLocations to choose a consistent executor per
topicpartition
commit d4e504a35f39831cd5c6ea077ea491307a5c2bda
Author: cody koeninger <[email protected]>
Date: 2016-03-04T22:12:40Z
[SPARK-12177][Streaming][Kafka] WIP modify KafkaRDD to use cached new
consumers
commit 37cca1ea2acfbad81d0466eb6bd49352570194da
Author: cody koeninger <[email protected]>
Date: 2016-03-07T22:21:41Z
[SPARK-12177][Streaming][Kafka] WIP modify DirectKafkaInputDStream to use
new consumer
commit 1c0699dbdf16a533bfbd52dbd125c8331c3f3318
Author: cody koeninger <[email protected]>
Date: 2016-03-07T22:22:23Z
Merge branch 'master' into kafka-0.9
commit f37717cd210367818be4fc6bd16557d94360b11e
Author: cody koeninger <[email protected]>
Date: 2016-03-07T22:47:53Z
[SPARK-12177][Streaming][Kafka] WIP merge backpressure fixes from master
commit e01daea67beccd14c1c6f77a3efd57282b42f1dd
Author: cody koeninger <[email protected]>
Date: 2016-03-08T19:25:38Z
[SPARK-12177][Streaming][Kafka] WIP remove KafkaUtils until interface for
new consumers is finished
commit 20b59e1970753f242cdde01ac9a6f131833cd6d4
Author: cody koeninger <[email protected]>
Date: 2016-03-08T20:45:50Z
[SPARK-12177][Streaming][Kafka] fix handling of kafkaParams
commit b55a3d41973d6838b1cb45ec2aa45d8bc6585cd3
Author: cody koeninger <[email protected]>
Date: 2016-03-08T23:01:29Z
[SPARK-12177][Streaming][Kafka] better handling of dynamic topic
subscriptions
commit d2949f802489f29a2107c2ee441a89d394ed7769
Author: cody koeninger <[email protected]>
Date: 2016-03-09T17:48:42Z
[SPARK-12177][Streaming][Kafka] WIP cached consumers basically working, but
dont handle recalculating the same RDD efficiently
commit 47209c02c6c095e58debfd10b182ec74ef979af8
Author: cody koeninger <[email protected]>
Date: 2016-03-09T17:57:26Z
[SPARK-12177][Streaming][Kafka] WIP remove unused class
commit dea8ec08d54050c27891fb39bba0d183aede55ab
Author: cody koeninger <[email protected]>
Date: 2016-03-10T20:02:31Z
[SPARK-12177][Streaming][Kafka] WIP add java convenience constructors, fix
tests
commit 43c27e929a40a331d2efc12e9e5cb2e18b44bcf3
Author: cody koeninger <[email protected]>
Date: 2016-03-11T19:32:40Z
[SPARK-12177][Streaming][Kafka] WIP allow user configuration of consumer
for dynamic topics, listener, etc
commit c04bc68726a5a9fb033e8a9cf57debf306d87a62
Author: cody koeninger <[email protected]>
Date: 2016-03-15T16:03:02Z
[SPARK-12177][Streaming][Kafka] WIP allow manual commit of offsets to kafka
commit 546246e6c40e7b0960af0d9e7c02623421d2009f
Author: cody koeninger <[email protected]>
Date: 2016-03-15T16:36:36Z
[SPARK-12177][Streaming][Kafka] WIP pause topicpartitions to prevent
consuming messages on driver
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
