GitHub user rhauch opened a pull request:
https://github.com/apache/kafka/pull/2984
KAFKA-4667 Connect uses AdminClient to create internal topics when needed
The backing store for offsets, status, and configs now attempts to use the
new AdminClient to look up the internal topics and create them if they donât
yet exist. If the necessary APIs are not available in the connected broker, the
stores fall back to the old behavior of relying upon auto-created topics. Kafka
Connect requires a minimum of Apache Kafka 0.10.0.1-cp1, and the AdminClient
can work with all versions since 0.10.0.0.
All three of Connectâs internal topics are created as compacted topics,
and new distributed worker configuration properties control the replication
factor for all three topics and the number of partitions for the offsets and
status topics; the config topic requires a single partition and does not allow
it to be set via configuration. All of these new configuration properties have
sensible defaults, meaning users can upgrade without having to change any of
the existing configurations. In most situations, existing Connect deployments
will have already created the storage topics prior to upgrading.
The replication factor defaults to 3, so anyone running Kafka clusters with
fewer nodes than 3 will receive an error unless they explicitly set the
replication factor for the three internal topics. This is actually desired
behavior, since it signals the users that they should be aware they are not
using sufficient replication for production use.
The integration tests use a cluster with a single broker, so they were
changed to explicitly specify a replication factor of 1 and a single partition.
The `KafkaAdminClientTest` was refactored to extract a utility for setting
up a `KafkaAdminClient` with a `MockClient` for unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rhauch/kafka kafka-4667
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2984.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2984
----
commit ea181ba29a678e47a5ff88ca0ad4c995aa4e017b
Author: Randall Hauch <[email protected]>
Date: 2017-05-05T17:14:07Z
KAFKA-4667 Connect can create internal topics using AdminClient
The backing store for offsets, status, and configs now attempts to use the
new AdminClient to look up the
internal topics and create them if they donât yet exist. If the necessary
APIs are not available in the connected
broker, the stores fall back to the old behavior of relying upon
auto-created topics. Kafka Connect requires
a minimum of Apache Kafka 0.10.0.1-cp1, and the AdminClient can work with
all versions since 0.10.0.0.
All three of Connectâs internal topics are created as compacted topics,
and new distributed worker configuration
properties control the replication factor for all three topics and the
number of partitions for the offsets
and status topics; the config topic requires a single partition and does
not allow it to be set via configuration.
All of these new configuration properties have sensible defaults, meaning
users can upgrade without having to
change any of the existing configurations. In most situations, existing
Connect deployments will have already
created the storage topics prior to upgrading.
The replication factor defaults to 3, so anyone running Kafka clusters with
fewer nodes than 3 will receive an
error unless they explicitly set the replication factor for the three
internal topics. This is actually desired behavior,
since it signals the users that they should be aware they are not using
sufficient replication for production use.
The integration tests use a cluster with a single broker, so they were
changed to explictly specify a replication factor of 1
and a single partition.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
