GitHub user rmetzger opened a pull request:
https://github.com/apache/flink/pull/500
Improvements to Kafka Source/Sink & some bugfixes in Flink Streaming
Hi,
I've spend the last two days on improving the Kafka connectors and the
streaming fault tolerance.
Since its the first time I'm touching this code, I would like to get some
feedback from the streaming guys (in particular on the changes with the state
handling).
Please note that this PR is based on
https://github.com/apache/flink/pull/474, so it also contains commits from
@gaborhermann.
I'll squash the commits after the first round of feedback.
If you want I can separate the the critical fixes from the Kafka connector
rewrite, but i would prefer merging everything at once.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/flink kafka-gabor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/500.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #500
----
commit b77196d73ef152330e172644aa83b54c44e05906
Author: Gábor Hermann <reckone...@gmail.com>
Date: 2015-03-09T18:03:11Z
[streaming] Added topic distribution to low level Kafka consumer
commit f70f31a5dc69f22c419911cf43062dd218308d4c
Author: Gábor Hermann <reckone...@gmail.com>
Date: 2015-03-10T14:12:29Z
[streaming] Added leader searching to Kafka connector at broker failure
commit 5de12983a7343a05f8850f9078a157b229a71136
Author: Gábor Hermann <reckone...@gmail.com>
Date: 2015-03-11T09:10:45Z
[streaming] Updated Kafka connector documentation
commit 9310711766bdfb41f19d89f776195cc02681f809
Author: Gábor Hermann <reckone...@gmail.com>
Date: 2015-03-11T15:50:45Z
[streaming] Kafka consumers connecting to Zookeeper instead of a broker
commit c421215ffdfc4bc8761d1403d3c37cd6ff875cd5
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-16T16:16:05Z
Let the KafkaConfigWrapper use the correct classloader
commit d3d08122c90c9299f78980fa75b5967f23c3f12d
Author: Gábor Hermann <reckone...@gmail.com>
Date: 2015-03-17T01:45:11Z
[streaming] [wip] Kafka integration test
commit fb165f6f8f8fbb4dfc6cea939ca8121ac4668275
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-17T09:14:34Z
fix
commit 4cdac3dc56276b3061ea49c8ccde50af4c235927
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-17T10:57:44Z
pom cleanup
commit f6d3d41402d434a364cf3700567fd1f521e46acd
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-17T13:45:37Z
wip
commit 2d5a8e86e90b42b8d86c7a2e870dc679f56e79db
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-17T14:03:31Z
local test is able to reproduce issue
commit 38bb7b44e862f5f61d72bc3db3ba35fc2ca63e3f
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-17T14:59:59Z
Fixed exception forwarding in streaming code
commit d2909a584d085131eff9cfb4c304c034f5049c12
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-17T18:40:20Z
wip
commit c74a558dcc3f179cfa358d703cf913503ac5ead4
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-17T21:06:19Z
very close
commit 3ad0fa830e86195a0fa6daaa1b212bd77825b1a2
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-17T21:49:56Z
That should work
commit f6c95d9a73472eccb68cf363ac3fe712d35c8a3c
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-17T22:00:21Z
current
commit f7603fad2f644bf40d2c6d7ae454838dcbca170e
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-18T10:24:35Z
checkstyle
commit d38c7befe239d5cbe5983ffa26d45e53edf0f8cc
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-18T13:31:46Z
make it ser
commit 005ce8c43333f660d8c66f5ac3eb8534d6d04b40
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-18T14:09:39Z
use usercode classloader to deserialize state
commit 57159e6e9aa7d8da9b9c88bec33bb2adc92879be
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-18T14:13:21Z
checkstyle
commit 29cd3d010ff058b52a253e405ae3a5d65d879301
Author: Robert Metzger <rmetz...@apache.org>
Date: 2015-03-18T17:13:04Z
wip
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
