Ken Huang created KAFKA-20898:
---------------------------------
Summary: PushTelemetry sets terminating flag on validation
failure, permanently locking out client instance
Key: KAFKA-20898
URL: https://issues.apache.org/jira/browse/KAFKA-20898
Project: Kafka
Issue Type: Bug
Reporter: Ken Huang
Assignee: Ken Huang
[https://github.com/apache/kafka/blob/89ccd6a1260130e1a5ea83a0aa98a7d397482ace/server/src/main/java/org/apache/kafka/server/ClientMetricsManager.java#L209]
[https://cwiki.apache.org/confluence/spaces/KAFKA/pages/173085915/KIP-714+Client+metrics+and+observability]
{quote}*Client termination*
When a client with an active metrics subscription is being shut down, it should
send its final metrics without waiting for the PushIntervalMs time.
To avoid the receiving broker’s metrics rate-limiter discarding this
out-of-profile push, the {{PushTelemetryRequest.Terminating}} field must be set
to true. A broker must only allow one such unthrottled metrics push for each
combination of client instance ID and SubscriptionId.
In the event that the client’s metric subscription has changed and the final
metrics push fails with error code {{UNKNOWN_SUBSCRIPTION_ID}} , the
terminating client can choose to obtain a new subscription ID by sending a
GetTelemetrySubscriptionsRequest and then immediately sending a
PushTelemetryRequest with the Terminating flag set to true, or it can choose to
abandon sending a final metrics push.{quote}
The implementation here differs from the behavior described in the KIP. The
second terminating request will fail.
The flow is as follows:
# The consumer is running normally, and the previous
{{GetTelemetrySubscriptions}} request returns {{{}subscriptionId = X{}}}.
# The admin runs {{{}kafka-client-metrics.sh --alter --interval 60000{}}},
causing the broker to recalculate the subscription ID, which changes from *X*
to {*}Y{*}.
# The consumer begins shutting down and sends its final {{PushTelemetry}}
request with {{terminating=true}} and the old {{{}subscriptionId = X{}}}.
# The broker fails the subscription ID validation and returns
{{{}UNKNOWN_SUBSCRIPTION_ID{}}}, but due to a bug, the {{terminating=true}}
state is still recorded.
# The consumer attempts to recover by calling {{GetTelemetrySubscriptions}}
again, obtains the new {{{}subscriptionId = Y{}}}, and sends another
{{PushTelemetry}} request with {{{}terminating=true{}}}.
# The broker sees that the instance has already been marked as
{{terminating=true}} and returns {{{}INVALID_REQUEST{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
