krishvishal opened a new pull request, #4747: URL: https://github.com/apache/iggy/pull/4747
## Which issue does this PR address? Relates to #4508 ## Rationale Kafka frameworks create topics with specific configurations and read those configurations back. The gateway currently rejects most of them. As a result, Kafka Connect, Kafka Streams, MirrorMaker 2, Debezium, kafka-ui, Terraform, and the Strimzi Topic Operator can stop at startup or when they create their first topic. This PR proposes the following answers to the three open questions in #4508. 1. Compaction. The gateway accepts `cleanup.policy=compact`, creates the topic without an expiry or size cap, and logs a warning. Until Iggy supports compaction (#4521), the gateway does not report `compact` in `DescribeConfigs`. Persisting the policy requires a new Iggy topic option with a typed field in all eight SDKs. 2. Replication factor. The gateway stores the requested replication factor in the Iggy `replication_factor` topic option introduced in #4692. 3. Settings Iggy cannot apply. If a setting has no client-visible effect, the gateway accepts any value Kafka accepts and reports the Kafka default in `DescribeConfigs`. It returns error `40` for `message.timestamp.type=LogAppendTime` and non-default values of `max.message.bytes`, because these settings change what clients receive. The following gaps keep #4508 open. - Schema Registry reads `cleanup.policy` on its second startup and stops unless `kafkastore.topic.skip.validation=true` is set. - Apicurio Registry 3 reads back `cleanup.policy=delete` on every startup and fails to start. - Retention changes apply to a topic created with `compact` if the alter request omits `cleanup.policy`. - Replication factor 2 requires both replicas to be available for writes because Iggy commits on a majority. Kafka can continue accepting writes with one replica down. - Metadata lists gateways as partition replicas instead of the Iggy nodes that host each partition. ## What changed? - `CreateTopics` previously returned error `40` for any configuration entry and error `38` for any replication factor other than `1` or `-1`. It now stores `retention.ms`, `retention.bytes`, `segment.bytes`, and the replication factor as Iggy topic options. It also validates every Kafka 4.4 topic key using Kafka's validation rules and error messages. - `DescribeConfigs` previously returned only `retention.ms` for topics and error `42` for a `BROKER` resource. It now returns the Kafka 4.4 topic keys for topics and the keys the Strimzi Topic Operator reads for brokers. - Three new variables control retention and durability for topics the gateway creates. Their defaults preserve the current behavior. Two changes affect existing clients. - `AlterConfigs` now replaces the full topic configuration, matching Kafka's behavior. An empty list resets the topic configuration to its defaults. - A `retention.bytes` value smaller than one segment now returns error `40` because Iggy always retains one sealed segment per partition. `gateways/kafka/docs/CONFIGS.md` documents each key, the gateway's response, and its limitations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
