GitHub user koeninger opened a pull request:
https://github.com/apache/spark/pull/3798
[SPARK-4964] [Streaming] Exactly-once semantics for Kafka
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/koeninger/spark-1 kafkaRdd
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3798.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3798
----
commit 76913e23179228481c98fbba36a54ca32fe20aed
Author: cody koeninger <[email protected]>
Date: 2014-11-23T03:15:30Z
Batch oriented kafka rdd, WIP. todo: cluster metadata / finding leader
commit 1d706257ac848d37caeaff0409bf60b080d66e48
Author: cody koeninger <[email protected]>
Date: 2014-11-23T06:10:56Z
WIP on kafka cluster
commit 0b94b3363cbc97c5d99e78c42f9be1c08a974fb1
Author: cody koeninger <[email protected]>
Date: 2014-11-24T14:49:24Z
use dropWhile rather than filter to trim beginning of fetch response
commit 4dafd1b0d58215cb27218e569cb5bea9d5146815
Author: cody koeninger <[email protected]>
Date: 2014-11-24T16:45:40Z
method to get leader offsets, switch rdd bound to being exclusive start,
inclusive end to match offsets typically returned from cluster
commit ce91c591569b8ac4e91dd29d013961fe0ee5c316
Author: cody koeninger <[email protected]>
Date: 2014-11-24T18:07:24Z
method to get consumer offsets, explicit error handling
commit 7d050bcb0bcacfbd4a7b858cffae809fd2af8e9d
Author: cody koeninger <[email protected]>
Date: 2014-11-24T22:11:24Z
methods to set consumer offsets and get topic metadata, switch back to
inclusive start / exclusive end to match typical kafka consumer behavior
commit 783b4775c89dbcbde9172d34653eab2718eee494
Author: cody koeninger <[email protected]>
Date: 2014-11-25T14:29:20Z
update tests for kafka 8.1.1
commit 29c6b430cc6bf5e2354b397289c4445f4993fc5b
Author: cody koeninger <[email protected]>
Date: 2014-11-25T15:33:45Z
cleanup logging
commit 3c2a96af2322754e8c76000b083ec3630a03e8c8
Author: cody koeninger <[email protected]>
Date: 2014-11-25T20:02:37Z
fix scalastyle errors
commit 4b078bf1e71745a6bc160c0836b54cc7b0d4171d
Author: cody koeninger <[email protected]>
Date: 2014-11-25T20:48:32Z
differentiate between leader and consumer offsets in error message
commit 8d7de4ab5a447a53f65be852702ca90512b2a639
Author: cody koeninger <[email protected]>
Date: 2014-11-25T23:54:40Z
make sure leader offsets can be found even for leaders that arent in the
seed brokers
commit 979da25f4d48e5ffccf13ba1ff66eb2527ff01f9
Author: cody koeninger <[email protected]>
Date: 2014-11-26T15:31:38Z
dont allow empty leader offsets to be returned
commit 38bb727cf31744fa625248c86c2a666920e83c36
Author: cody koeninger <[email protected]>
Date: 2014-12-03T21:42:25Z
give easy access to the parameters of a KafkaRDD
commit 326ff3cbda37066ebef7492241276754164d2879
Author: cody koeninger <[email protected]>
Date: 2014-12-16T21:27:44Z
add some tests
commit 6bf14f2850f9f40f53b4c1eec373214e1b6d3465
Author: cody koeninger <[email protected]>
Date: 2014-12-24T17:38:52Z
first attempt at a Kafka dstream that allows for exactly-once semantics
commit bcca8a4b69f73b48f71b9558adf718b5324ed933
Author: cody koeninger <[email protected]>
Date: 2014-12-24T20:35:43Z
Merge branch 'master' of https://github.com/apache/spark into kafkaRdd
commit 37d305320e72de1ee6ffcd42f6a45d331a4d465d
Author: cody koeninger <[email protected]>
Date: 2014-12-25T04:41:40Z
make KafkaRDDPartition available to users so offsets can be committed per
partition
commit cac63eec4a0bee6b662c4577404622a08904f0cb
Author: cody koeninger <[email protected]>
Date: 2014-12-25T07:11:58Z
additional testing, fix fencepost error
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
