Yash Mayya created KAFKA-14455: ---------------------------------- Summary: Kafka Connect create and update REST APIs should surface failures while writing to the config topic Key: KAFKA-14455 URL: https://issues.apache.org/jira/browse/KAFKA-14455 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Yash Mayya Assignee: Yash Mayya
Kafka Connect's `POST /connectors` and `PUT /connectors/\{connector}/config` REST APIs internally simply write a message to the Connect cluster's internal config topic (which is then processed asynchronously by the herder). However, no callback is passed to the producer's send method and there is no error handling in place for producer send failures (see [here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L716] / [here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L726]). Consider one such case where the Connect worker's principal doesn't have a WRITE ACL on the cluster's config topic. Now suppose the user submits a connector's configs via one of the above two APIs. The producer send [here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L716] / [here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L726] won't succeed (due to a TopicAuthorizationException) but the API responses will be `201 Created` success responses anyway. This is a very poor UX because the connector will actually never be created but the API response indicated success. Furthermore, this failure would only be detectable if TRACE logs are enabled (via [this log)|https://github.com/apache/kafka/blob/df29b17fc40f7c15460988d58bc652c3d66b60f8/clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java] making it near impossible for users to debug. Producer callbacks should be used to surface write failures back to the user via the API response. -- This message was sent by Atlassian Jira (v8.20.10#820010)