With the new release of Kafka I wanted to start the discussion on how best
to handle updating Flume to be able to make use of some of the new features
available in 0.9.
First, it is important for Flume to adopt the 0.9 Kafka Clients as the new
Consumer / Producer API's are the only APIs that support new Security
features put into the latest Kafka release such as SSL.
If we agree that this is important, then we need to consider how best to
make this switch. With many projects, we could just update the jars/clients
and move along happily, however, the Kafka compatibility story complicates
this.
-
Kafka promises to be backward compatible with clients
-
i.e. A 0.8.x client can talk to a 0.9.x broker
-
Kafka does not promise to be forward compatible (at all) from client
perspective:
-
i.e. A 0.9.x client can not talk to a 0.8.x broker
-
If it works, its is by luck and not reliable, even for old
functionality
-
This is due to protocol changes and no way for the client to know the
version of Kafka it’s talking to. Hopefully KIP-35 (Retrieving protocol
version) will move this in the right direction.
-
Integrations that utilize Kafka 0.9.x clients will not be able to talk
to Kafka 0.8.x brokers at all and may get cryptic error messages when doing
so.
-
Integrations will only be able to support one major version of Kafka at
a time without more complex class-loading
-
Note: The kafka_2.10 artifact depends on the kafka-clients artifact
so you cannot have both kafka-clients & kafka_2.10 of different
versions at
the same time without collision
-
However older clients (0.8.x) will work when talking to 0.9.x server.
-
But that is pretty much useless as the benefits of 0.9.x (security
features) won’t be available.
Given these constraints, and after careful consideration, I propose that we
do the following.
1) Update the Kafka libraries to the latest 0.9/0.9+ release and update the
Source, Sink and Channel Implementations to make use of the new Kafka
Clients
2) Document that Flume no longer supports Kafka Brokers < 0.9
Given that both producer and clients will be updated, there will need to be
changes in agent configurations to support the new clients.
This means if upgrading Flume, existing agent configurations will break. I
don't see a clean way around this, unfortunately. This seems to be a
situation where we break things, and document this to be the case.
--
Jeff Holoman
Systems Engineer
