krishvishal opened a new issue, #4514: URL: https://github.com/apache/iggy/issues/4514
## Summary Metadata ignores `allow_auto_topic_creation`, and nothing creates a topic on first use. Who hits it: - Kafka's broker default is `auto.create.topics.enable=true`. The Java producer asks for creation on every Metadata call, and the Java consumer asks by default. librdkafka producers, Sarama, KafkaJS, kafka-python 3 and aiokafka ask by default too. - Connect source connectors without `topic.creation.*`, Debezium per-table topics, the OpenTelemetry Kafka exporter, Spring and SmallRye dead letter topics, and log shippers rely on it. - The Java producer waits `max.block.ms` (60 s) per send, then fails with "Topic X not present in metadata". librdkafka gives up after 30 s and aiokafka after 40 s. Connect source tasks and MirrorMaker 2 block forever, because they set `max.block.ms` to `Long.MAX_VALUE`. ## Open question Is auto-creation on by default? Kafka's default is on. ## Scope - [ ] A gateway setting for auto-creation, with a default partition count and replication factor - [ ] On a Metadata request that asks for creation, start the create, and answer `UNKNOWN_TOPIC_OR_PARTITION` until the topic exists, as Kafka does - [ ] With SASL on, require `manage_topics` from the principal - [ ] Document the setting ## Acceptance criteria - [ ] With the setting on, a Java producer sends to a new topic with no CreateTopics call and succeeds - [ ] With the setting off, nothing changes from today ## References - Broker default: [ServerLogConfigs.java:150-151](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/server-common/src/main/java/org/apache/kafka/server/config/ServerLogConfigs.java#L150-L151) - Producer Metadata request: [ProducerMetadata.java:62-69](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerMetadata.java#L62-L69) - Kafka's answer while it creates: [DefaultAutoTopicCreationManager.java:199-207](https://github.com/apache/kafka/blob/b15258c7398889248d7fbf2a2218cc01346ebf92/server/src/main/java/org/apache/kafka/server/DefaultAutoTopicCreationManager.java#L199-L207) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
