GitHub user majorendre opened a pull request:
https://github.com/apache/flume/pull/235
Flume 3281 Update to Kafka 2.0
This has been tested with unit tests. The main difference that caused the
most problems is the consumer.poll(Duration) change. This does not block even
when it fetches meta data whereas the previous poll(long timeout) blocked
indefinitely for meta data fetching.
This has resulted in many test timing issues. I tried to do minimal changes
at the tests, just enough to make them pass.
Kafka 2.0 requires a higher version for slf4j, I had to update it to 1.7.25.
Option migrateZookeeperOffsets is deprecated in this PR. This will allow us
to get rid of Kafka server libraries in Flume.
Compatibility testing.
Modified the TestUtil to be able to use external servers. This I could test
against a variety of Kafka Server versions using the normal unit tests.
Channel tests using 2.0.1 client:
Kafka_2.11-0.11.0.3 - timeouts in TestPartitions when creating topics
Kafka_2.11-1.0.2 - passed
Kafka_2.11-1.1.1 - passed
Kafka_2.11-2.0.1 - passed
I will publish further results here, today or tomorrow.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/majorendre/flume FLUME-3281
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flume/pull/235.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #235
----
commit 2b6818ad2f8c9d8367ba5526f800583f29967464
Author: Endre Major <emajor@...>
Date: 2018-11-08T14:36:47Z
FLUME-3281 Update to Kafka 2.0
commit e1a98bf98bdfa2fc3f524e94ed5e5603477f7820
Author: Endre Major <emajor@...>
Date: 2018-11-10T16:14:06Z
FLUME-3281 TestUtil improvements external servers
commit 34ce07c9ba3cc388630a6fd2e5cb14f94b665ca5
Author: Endre Major <emajor@...>
Date: 2018-11-12T13:31:34Z
FLUME-3281 Deprecating migrateZookeeperOffsets
----
---
