Hi,
Kafka 2.8.x changed the behaviour of auto topic creation (KAFKA-9751) so that
it's now done asynchronously by a forwarded request to the controller rather
than being immediately started by the broker that received the MetadataRequest.
In 2.7.x and earlier the client used to get LEADER_NOT_AVAILABLE on the first
MetadataResponse as the topic creation was immediately in-progress and the
partitions and leaders were just being assigned. With 2.8.x onward, instead
they now get UNKNOWN_TOPIC_OR_PARTITION until the Controller has actioned the
create. That seems fine and is easy to handle.
However, historically the __consumer_offsets topic was special cased such that
it would get auto-created on a Metadata request even if normal topic
auto-creation (`auto.create.topics.enable`) was disabled in the broker. Does
anyone know if something has changed there?
If I startup a brand new empty 2.8.x cluster and just send MetadataRequests for
the __consumer_offsets topic I always get UNKNOWN_TOPIC_OR_PARTITION
responses and the auto-creation doesn't seem to ever get actioned for that
alone. As you'd expect, if I start up a consumer group the JoinGroup is
sufficient to cause the __consumer_offsets topic to be created. On 2.7.x and
earlier, just sending MetadataRequests is sufficient for the __consumer_offsets
topic to be created.
The behaviour can be observed using (e.g.,) kcat (neé kafkacat) after starting
a brand new cluster with auto.create.topics.enable=false:
> kcat -L -b localhost:29091,localhost:29092,localhost:29093 -t
> __consumer_offsets -d metadata,topic
And no auto-creation ever happens despite allowAutoCreate true in the Metadata
request body:
Kafka (Metadata v4 Request)
Length: 42
API Key: Metadata (3)
API Version: 4
Correlation ID: 3
Client ID: rdkafka
Topic
Topic Name: __consumer_offsets
Allow Auto Topic Creation: True
Is this a regression or an intentional change in behaviour? I couldn't find any
reference to this in the changelog / release notes.
Best regards,
Dom
--
Dominic Evans
IBM Software Group, Hursley Park, UK
