github-actions[bot] commented on code in PR #4400:
URL: https://github.com/apache/iggy/pull/4400#discussion_r4179121121
##########
gateways/kafka/tests/common/scope.rs:
##########
@@ -37,6 +37,7 @@ pub const SCOPED_API_KEYS: &[(i16, &str, i16, i16)] = &[
(14, "SyncGroup", 0, 5),
(18, "ApiVersions", 0, 3),
(19, "CreateTopics", 2, 5),
+ (20, "DeleteTopics", 1, 5),
Review Comment:
warning: The change puts key 20 in `SCOPED_API_KEYS`, but
`OUT_OF_SCOPE_API_KEYS` still lists it at `tests/common/wire.rs:39`, so the two
tables contradict each other. `out_of_scope_api_keys_close_without_panic` now
passes because the v0 response fails to encode, not because the key is unknown,
and `server_e2e_tests.rs:425` hides the contradiction with `[..4]`. Remove the
`(20, "DeleteTopics")` entry and that slice.
##########
gateways/kafka/tests/version_firewall_tests.rs:
##########
@@ -100,9 +101,10 @@ fn is_supported_version_matches_scope_table() {
/// `advertised_min_version` - the same reasoning applies to it as the
function under test.
///
/// Relies on `SUPPORTED_RANGES` (src) and `SCOPED_API_KEYS` (test) sharing
declaration order
-/// (Produce, Fetch, `ListOffsets`, Metadata, `ApiVersions`, `CreateTopics`,
`FindCoordinator`,
-/// `JoinGroup`, Heartbeat, `LeaveGroup`, `SyncGroup`) -
`supported_ranges_table_has_twelve_entries` plus
-/// `is_supported_version_matches_scope_table` already pin that both tables
cover the same keys.
+/// (Produce, Fetch, `ListOffsets`, Metadata, `ApiVersions`, `CreateTopics`,
`DeleteTopics`,
Review Comment:
nit: The comment says the test relies on the two tables sharing declaration
order, but `api_versions.rs:86` sorts the advertisement by key and the
declarations differ. Say that the advertisement is key-ascending, and add
`InitProducerId` to the listed names.
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the
backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why:
the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no
way to tell
+//! whether the stream it just emptied is actually abandoned or just
temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+
(`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any
text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded
at every version it
+//! supports.
+
+use std::time::Duration;
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse,
TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT,
GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, handle_versioned_request,
is_supported_version,
+ respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`,
topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name
`TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this
message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the
bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted
differently: every
+/// name here costs its own bridge round trip against the single lockstep
`IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this
charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+/// Same bounds as `create_topics::clamp_request_timeout` - this request
carries the identical
+/// KIP-4 `timeout_ms` field for the identical reason (a client-supplied
deadline that becomes
+/// the aggregate bridge-work budget, otherwise unchecked).
+const MIN_REQUEST_TIMEOUT: Duration = Duration::from_millis(1_000);
+const MAX_REQUEST_TIMEOUT: Duration = Duration::from_secs(30);
+
+fn clamp_request_timeout(timeout_ms: i32) -> Duration {
Review Comment:
simplification: `MIN_REQUEST_TIMEOUT`, `MAX_REQUEST_TIMEOUT`,
`clamp_request_timeout`, and the three tests at line 270 copy
`create_topics.rs` verbatim. Simpler: move one copy into `handlers/mod.rs` as
`pub(crate)` and call it from both handlers. Saves about 20 lines and keeps one
place for the bounds.
##########
gateways/kafka/src/protocol/handlers/delete_topics.rs:
##########
@@ -0,0 +1,286 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! `DeleteTopics` (API key 20).
+//!
+//! Deletes the Iggy topic a Kafka topic name resolves to. Never deletes the
backing Iggy
+//! stream - see [`IggyBridge::delete_kafka_topic`]'s own doc comment for why:
the shared
+//! `default_stream` can hold other Kafka topics' data, and this bridge has no
way to tell
+//! whether the stream it just emptied is actually abandoned or just
temporarily topic-less.
+//!
+//! `DeletableTopicResult.error_message` is only encoded on the wire at v5+
(`kafka_protocol`'s
+//! own gate) - a client negotiating v1-v4 gets `error_code` only, with any
text this module sets
+//! silently dropped. Unlike `CreateTopics`, whose equivalent field is encoded
at every version it
+//! supports.
+
+use std::time::Duration;
+
+use bytes::Bytes;
+use iggy::prelude::IggyError;
+use kafka_protocol::messages::delete_topics_response::DeletableTopicResult;
+use kafka_protocol::messages::{DeleteTopicsRequest, DeleteTopicsResponse,
TopicName};
+use kafka_protocol::protocol::StrBytes;
+
+use tokio::time::Instant;
+
+use crate::bridge::{BridgeError, IggyBridge};
+use crate::error::Result;
+use crate::protocol::api::{
+ API_KEY_DELETE_TOPICS, ApiVersionRange, ERROR_INVALID_REQUEST, ERROR_NONE,
+ ERROR_NOT_CONTROLLER, ERROR_POLICY_VIOLATION, ERROR_REQUEST_TIMED_OUT,
GatewayState,
+ HandleOutcome,
+};
+use crate::protocol::bounds_guard::validate_delete_topics_shape;
+use crate::protocol::handlers::{
+ decode_guarded, encode_message, handle_versioned_request,
is_supported_version,
+ respond_or_close, unsupported_version_response,
+};
+
+/// This bridge never advertises v6 (the `topics: Vec<DeleteTopicState>`,
topic-id-based shape).
+///
+/// It has no concept of a Kafka topic id, only the Kafka-side name
`TopicMapping` resolves. v1
+/// is the crate's own floor (`kafka_protocol` does not implement v0 for this
message).
+pub const RANGE: ApiVersionRange = ApiVersionRange {
+ api_key: API_KEY_DELETE_TOPICS,
+ min_version: 1,
+ max_version: 5,
+};
+
+/// Cap on topic names one `DeleteTopics` request may address through the
bridge.
+///
+/// Same rationale as `create_topics::MAX_BRIDGE_BACKED_TOPICS`, but counted
differently: every
+/// name here costs its own bridge round trip against the single lockstep
`IggyClient` every
+/// Kafka connection on this gateway shares, duplicates included, so this
charges the raw count
+/// rather than the distinct one `CreateTopics` charges.
+const MAX_BRIDGE_BACKED_TOPICS: usize = 100;
+
+/// Same bounds as `create_topics::clamp_request_timeout` - this request
carries the identical
+/// KIP-4 `timeout_ms` field for the identical reason (a client-supplied
deadline that becomes
+/// the aggregate bridge-work budget, otherwise unchecked).
+const MIN_REQUEST_TIMEOUT: Duration = Duration::from_millis(1_000);
+const MAX_REQUEST_TIMEOUT: Duration = Duration::from_secs(30);
+
+fn clamp_request_timeout(timeout_ms: i32) -> Duration {
+ let requested =
Duration::from_millis(u64::try_from(timeout_ms).unwrap_or(0));
+ requested.clamp(MIN_REQUEST_TIMEOUT, MAX_REQUEST_TIMEOUT)
+}
+
+pub async fn handle(state: &GatewayState, api_version: i16, body: Bytes) ->
HandleOutcome {
+ let Some(bridge) = &state.bridge else {
+ return handle_versioned_request(
+ API_KEY_DELETE_TOPICS,
+ api_version,
+ body,
+ |v, b| {
+ decode_guarded::<DeleteTopicsRequest>(v, b, |v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ })
+ },
+ encode_response,
+ encode_error_response,
+ "DeleteTopics",
+ );
+ };
+
+ if !is_supported_version(API_KEY_DELETE_TOPICS, api_version) {
+ return unsupported_version_response(API_KEY_DELETE_TOPICS,
api_version, |version| {
+ encode_error_response(version, ERROR_INVALID_REQUEST)
+ });
+ }
+
+ let req = match decode_guarded::<DeleteTopicsRequest>(api_version, body,
|v, b| {
+ validate_delete_topics_shape(v, b, state.max_frame_size)
+ }) {
+ Ok(req) => req,
+ Err(error) => {
+ // debug!, not warn!: attacker-controlled, not operator-actionable.
+ tracing::debug!(%error, "Failed to decode DeleteTopics request");
+ return respond_or_close(
+ encode_error_response(api_version, ERROR_INVALID_REQUEST),
+ "DeleteTopics",
+ );
+ }
+ };
+
+ // RANGE caps at v5, so req.topics (the v6 topic-id shape) is always empty
- topic_names is
+ // the only populated field at any version this bridge advertises.
+ if req.topic_names.len() > MAX_BRIDGE_BACKED_TOPICS {
+ tracing::warn!(
+ requested_topics = req.topic_names.len(),
+ max = MAX_BRIDGE_BACKED_TOPICS,
+ "DeleteTopics request addresses too many topics; rejecting"
+ );
+ let message = StrBytes::from(format!(
+ "this gateway addresses at most {MAX_BRIDGE_BACKED_TOPICS} topic
names per DeleteTopics request"
+ ));
+ let results = req
+ .topic_names
+ .iter()
+ .map(|name| {
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_POLICY_VIOLATION)
+ .with_error_message(Some(message.clone()))
+ })
+ .collect();
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ return respond_or_close(encode_message(&resp, api_version, 256),
"DeleteTopics");
+ }
+
+ let deadline = Instant::now() + clamp_request_timeout(req.timeout_ms);
+ let results = delete_all_topics(bridge, &req.topic_names, deadline).await;
+ let resp = DeleteTopicsResponse::default().with_responses(results);
+ respond_or_close(encode_message(&resp, api_version, 256), "DeleteTopics")
+}
+
+/// Deletes every requested topic, independently of the others.
+///
+/// `deadline` bounds each topic's own bridge work individually
(`timeout_at`), not the whole
+/// loop - same reasoning as `create_topics::create_all_topics`: a single
`timeout` around the
+/// entire call would discard every already-resolved result the moment one
topic's call ran
+/// long, answering a retriable code even for topics that had already deleted
cleanly.
+///
+/// A name repeated within the same request is not specially rejected the way
`CreateTopics`
+/// rejects a duplicate: deleting the same topic twice has no race to protect
against the way
+/// creating it twice does - the first occurrence deletes it, the second then
answers
+/// `UNKNOWN_TOPIC_OR_PARTITION` because by the time it runs, the topic
genuinely doesn't exist
+/// any more. That's the correct answer for a duplicate, not a special case.
+async fn delete_all_topics(
+ bridge: &IggyBridge,
+ topic_names: &[TopicName],
+ deadline: Instant,
+) -> Vec<DeletableTopicResult> {
+ let mut results = Vec::with_capacity(topic_names.len());
+ for name in topic_names {
+ let result = match tokio::time::timeout_at(deadline,
delete_one_topic(bridge, name)).await {
+ Ok(result) => result,
+ Err(_elapsed) => {
+ tracing::warn!(
+ kafka_topic = name.as_str(),
+ "DeleteTopics: this topic's bridge work exceeded the
request deadline; \
+ answering retriable instead of blocking further"
+ );
+ DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_code(ERROR_REQUEST_TIMED_OUT)
+ .with_error_message(None)
+ }
+ };
+ results.push(result);
+ }
+ results
+}
+
+async fn delete_one_topic(bridge: &IggyBridge, name: &TopicName) ->
DeletableTopicResult {
+ let result = DeletableTopicResult::default()
+ .with_name(Some(name.clone()))
+ .with_error_message(None);
+ match bridge.delete_kafka_topic(name.as_str()).await {
+ // The second arm: the write committed on its first attempt, the SDK's
own reconnect path
+ // replayed it, and the server's client-table dedup caught the replay
- not a fault. Same
+ // reasoning as `create_topics::create_one_topic`'s identical arm.
+ Ok(()) | Err(BridgeError::Iggy(IggyError::RequestAlreadyApplied)) => {
+ result.with_error_code(ERROR_NONE)
+ }
+ Err(err) => bridge_error_result(result, name.as_str(), &err),
+ }
+}
+
+/// Maps a bridge failure to a Kafka result, logging the real cause
server-side.
+///
+/// Same caution as `create_topics::bridge_error_result` on forwarding
`err.to_string()`: a
+/// server-side rejection can reconstruct with default-valued fields, so only
the client-caused
+/// variant (an invalid name) forwards its own text; everything else gets a
fixed message.
+fn bridge_error_result(
+ result: DeletableTopicResult,
+ kafka_topic: &str,
+ err: &BridgeError,
+) -> DeletableTopicResult {
+ let error_code = err.to_kafka_error_code();
+ match err {
+ BridgeError::InvalidKafkaTopicName { reason, .. } => {
+ tracing::debug!(
+ kafka_topic,
+ reason,
+ "DeleteTopics rejected an invalid topic name"
+ );
+ result
+ .with_error_code(error_code)
+ .with_error_message(Some(StrBytes::from(reason.clone())))
+ }
+ other => {
+ tracing::error!(kafka_topic, %other, "DeleteTopics failed against
the Iggy bridge");
+ result
+ .with_error_code(error_code)
+ .with_error_message(Some(StrBytes::from(
+ "internal error deleting this topic".to_string(),
Review Comment:
warning: A missing topic and an `Unauthorized` reply both land in this
catch-all. A v5 client then reads `internal error deleting this topic` for a
topic that does not exist, and the log records that routine case at `error!`.
Add arms for the not-found and authorization variants that answer accurate text
and log at `debug!`.
##########
gateways/kafka/src/bridge/iggy_bridge/topics.rs:
##########
@@ -257,6 +257,30 @@ impl IggyBridge {
}
}
+ /// Deletes the Iggy topic backing `kafka_topic`, resolved through the
configured
+ /// [`TopicMapping`](crate::bridge::topic_map::TopicMapping).
+ ///
+ /// Deliberately leaves the backing Iggy stream alone, even when this
delete empties it out:
+ /// `TopicMapping::resolve` sends every *unmapped* Kafka topic to the same
`default_stream`
+ /// (`ensure_stream_and_topic`'s own doc comment flags the same sharing
risk for create), so a
+ /// stream this call happens to empty may still be the live home for a
different Kafka topic a
+ /// moment later - this call has no way to tell the two cases apart, and
real Kafka itself has
+ /// no analogous container object for a `DeleteTopics` to clean up.
+ ///
+ /// # Errors
+ ///
+ /// Returns [`BridgeError::InvalidKafkaTopicName`] if `kafka_topic` fails
Kafka's own
+ /// topic-naming rules. Returns [`BridgeError::Timeout`] if a call takes
longer than
+ /// `REQUEST_TIMEOUT`. Returns [`BridgeError::Iggy`] with
`StreamNameNotFound`/`TopicNameNotFound`
Review Comment:
nit: The doc names `StreamNameNotFound` and `TopicNameNotFound`, but a
rejection from `delete_topic` arrives as `StreamIdNotFound` (1009) or
`TopicIdNotFound` (2010). Name those two variants instead.
##########
gateways/kafka/docs/SCOPE.md:
##########
@@ -227,6 +229,29 @@ below it are still open for the issues that build on top
of it.
request's own wire `timeout_ms` (clamped to `[1s, 30s]`) now bounds the
whole handler's
aggregate bridge work, not just decoded and discarded - a deadline that
fires answers every
topic `REQUEST_TIMED_OUT` rather than continuing to hold the shared
lockstep `IggyClient`.
+- [x] Real DeleteTopics ([#3546](https://github.com/apache/iggy/issues/3546)):
with
+ `IGGY_KAFKA_BRIDGE_ENABLED=true`, deletes the Iggy topic through
+ `IggyBridge::delete_kafka_topic`.
`src/protocol/handlers/delete_topics.rs`,
+ `tests/delete_topics_real_bridge_tests.rs`. With the bridge off, the
stub answers
+ `NOT_CONTROLLER` (41), matching CreateTopics' own stub convention.
+ - **Never deletes the backing Iggy stream**, even when the delete leaves it
with no topics in
+ it. `TopicMapping::resolve` sends every unmapped Kafka topic to the same
`default_stream`
+ (`ensure_stream_and_topic`'s own doc comment flags the identical sharing
risk for create), so
+ a stream this call happens to empty may still be the live home for a
different Kafka topic a
+ moment later - there is no way to tell the two cases apart from here, and
real Kafka itself
+ has no analogous container object for a `DeleteTopics` to clean up. See
+ [`BRIDGE_MAPPING.md`](BRIDGE_MAPPING.md).
+ - Per-topic results are independent: one missing or invalid name in a batch
does not block an
+ existing, validly-named topic elsewhere in the same request from being
deleted.
+ - A name repeated within one request is not specially rejected the way
`CreateTopics` rejects a
+ duplicate - deleting the same topic twice has no race to protect against
the way creating it
+ twice does, so the first occurrence deletes it and the second then
correctly answers
+ `UNKNOWN_TOPIC_OR_PARTITION` (3) because the topic genuinely no longer
exists.
+ - Same bridge-fan-out bounds as CreateTopics: a request naming more than 100
distinct topics is
Review Comment:
nit: This bullet says a request naming more than 100 distinct topics is
rejected, but the handler at `delete_topics.rs:121` counts duplicates too. Say
`more than 100 topic names`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
