Hi, Following on from the discussion in the community, and the difficulty of trying to accommodate the client instance ID in a tagged field with acceptable complexity, I have made a significant change to the KIP. Now, it introduces v3 of the request header containing the client instance ID as a proper field.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1313%3A+Client+instance+ID+in+all+request+headers Please review and let me know what you think. Thanks, Andrew On 2026/05/28 21:51:01 "Matthias J. Sax" wrote: > Thanks Andrew. I agree to the issue that tagged fields are optional. > > Nevertheless, I find the proposed compatibility protocol too complex and > unnecessary, and would still prefer a version bump for > `GetTelemetrySubscriptionsRequest/Reponse` to clean it all up. -- For > `PushTelemetryRequest` the argument from above applies, so leaving it > untouched is ok. > > I don't see why `GetTelemetrySubscriptionsRequest` would need to encode > the `clientInstandId` in it's request body. Given that the > `clientInstanceId` is a random UUID, the returned subscription cannot > really depend on it. So why would we need to send it to the broker? > Right now, it's set to ZERO only for the purpose to get the broker to > assign a new UUID. But it has nothing to do with the actual "what > metrics should I send" question. > > To be fair, KIP-714 lists `client_instance_id` as a field that can be > used to define a subscription. But it seems rather useless in practice > to me? I a user defines a subscription, they cannot know the UUID. Thus, > I think we should actually drop supporting `client_instance_id` as a > "subscription matching parameter"? Of course, there is a backward > compatibility question, but I think we can address this: if brokers are > upgraded and they have an existing `client_instance_id` based > subscription defined, they could advertise to only support v0, and they > should log a warning that this feature is deprecated. New brokers would > also not allow to use `client_instance_id` any longer to define a > subscription. > > Similarly, for `GetTelemetrySubscriptionsResponse`, we only need > `clientInstanceId` if the broker computes the UUID. But if we let the > client compute it, this field is not needed any longer. > > > -Matthias > > > > On 5/28/26 1:21 AM, Andrew Schofield wrote: > > Hi Jun, > > Thanks for your response. > > > > JR24: I'll update the KIP shortly. Here's a summary. The various modern > > group protocol heartbeat requests encode the member ID as a string for > > historical reasons, even though it's in practice a UUID. The Apache Kafka > > Java client happens to use org.apache.kafka.common.Uuid to create the > > member ID and convert it into a string. This means that when the > > clientInstanceId is converted into a string, its encoding matches the > > member ID and it all lines up. However, there are other ways to encode a > > UUID into a string and I know for a fact that the librdkafka client does it > > differently. > > > > Thanks, > > Andrew > > > > On 2026/05/27 17:11:25 Jun Rao via dev wrote: > >> Hi, Andrew, > >> > >> Thanks for the reply. One more comment. > >> > >> JR24. "The alignment of other identifiers is by convention (and the Java > >> client will follow the convention) rather than mandate." Could you describe > >> the convention to convert a clientInstanceId (UUID) to a memberId (String)? > >> > >> Jun > >> > >> > >> On Tue, May 19, 2026 at 2:36 AM Andrew Schofield <[email protected]> > >> wrote: > >> > >>> Hi Jun, > >>> Thanks for your response. > >>> > >>> JR23: You are absolutely correct. It seems to me that not sending a > >>> clientInstanceId in the header and explicitly sending a zero UUID as the > >>> clientInstanceId in the header can be treated as semantically equivalent. > >>> I've tweaked the words slightly. > >>> > >>> Thanks, > >>> Andrew > >>> > >>> On 2026/05/19 03:42:16 Jun Rao via dev wrote: > >>>> Hi, Andrew, > >>>> > >>>> Thanks for the reply. > >>>> > >>>> JR23. Our message protocol doc says "Any fields in the message object > >>> that > >>>> are not present in the version that you are deserializing will be reset > >>> to > >>>> default values. Unless a custom default has been set:". Uuid fields > >>>> default to zero uuid. > >>>> So if the server gets header.clientInstanceId=0 in the deserialized > >>> header, > >>>> could it distinguish between the ID not being present (since client is > >>> old) > >>>> and the ID being explicitly set to 0 by the client? > >>>> > >>>> Jun > >>>> > >>>> > >>>> On Mon, May 18, 2026 at 7:45 PM Andrew Schofield <[email protected]> > >>>> wrote: > >>>> > >>>>> Hi Jun, > >>>>> Thanks for your reply. It's tricky squaring a circle. > >>>>> > >>>>> JR23: For GetTelemetrySubscriptions, I have changed it so that a client > >>>>> which omits the ClientInstanceId from the request header is permitted > >>> to > >>>>> specify a zero ClientInstanceId in the request body, following original > >>>>> KIP-714 precedent. However, a client which specifies a > >>> ClientInstanceId in > >>>>> the request header MUST specify the same ClientInstanceId in the > >>> request > >>>>> body. This ensures that the header and telemetry UUIDs are the same. > >>>>> > >>>>> Thanks, > >>>>> Andrew > >>>>> > >>>>> On 2026/05/12 17:48:23 Andrew Schofield wrote: > >>>>>> Hi Jun, > >>>>>> Thanks for the reply and digging into the details. > >>>>>> > >>>>>> JR23: Correct. The client telemetry component will use UUID-B as the > >>>>> client instance ID. > >>>>>> > >>>>>> JR23.1: Yes, I agree. It's not ideal. When I was drawing up the > >>> tables, > >>>>> I was thinking that this might be a possibility, but I'm less convinced > >>>>> now. I think that I should mandate that if a client specifies > >>>>> header.ClientInstanceId on GetTelemetrySubscriptions request, then > >>>>> request.ClientInstanceId must either be zero or equal to > >>>>> header.ClientInstanceId. > >>>>>> > >>>>>> JR23.2: This is perhaps the interesting one. From its original > >>> intent, > >>>>> it should be UUID-B (the telemetry UUID), but then that contradicts the > >>>>> change in signature to remove the timeout. Unless I make the change > >>> above, > >>>>> in which case it will be UUID-H. > >>>>>> > >>>>>> Thanks, > >>>>>> Andrew > >>>>>> > >>>>>> On 2026/05/12 17:23:58 Jun Rao via dev wrote: > >>>>>>> Hi, Andrew, > >>>>>>> > >>>>>>> Thanks for the reply. > >>>>>>> > >>>>>>> JR23. In the new client -> old broker case, we have > >>>>>>> header.ClientInstanceId=UUID-H > >>>>>>> request.ClientInstanceId=UUID-B > >>>>>>> response.ClientInstanceId=0 > >>>>>>> > >>>>>>> On the server side, I guess the telemetry component will use > >>> UUID-B as > >>>>> the > >>>>>>> clientInstanceId? This has a couple of implications. > >>>>>>> JR23.1 On the server side, we have two different clientInstanceIds > >>>>> used in > >>>>>>> different places, UUID-H for request logging and UUID-B in > >>> telemetry. > >>>>> This > >>>>>>> seems confusing since we can't uniquely identify a client on the > >>> server > >>>>>>> side. > >>>>>>> JR23.2 On the client side. what uuid does clientInstanceId(Duration > >>>>>>> timeout) return? If it returns UUID-H, it will be confusing since > >>> it > >>>>>>> doesn't match the ID used for telemetry on the server. > >>>>>>> > >>>>>>> Jun > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Tue, May 12, 2026 at 12:58 AM Andrew Schofield < > >>>>> [email protected]> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Hi Jun, > >>>>>>>> Thanks for your response. > >>>>>>>> > >>>>>>>> JR20: I have improved (I hope) the wording. The client sends > >>>>>>>> request.clientInstanceId = 0 and header.clientInstanceId = > >>> UUID-H, > >>>>> and the > >>>>>>>> broker responds response.clientInstanceId=UUID-H. In this way, > >>> the > >>>>> broker > >>>>>>>> will have taken the UUID-H from the header, and told the client > >>> to > >>>>> use it > >>>>>>>> for client telemetry also. > >>>>>>>> > >>>>>>>> JR21: Done. Look for "henceforth". > >>>>>>>> > >>>>>>>> JR22: Summary table added. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Andrew > >>>>>>>> > >>>>>>>> On 2026/05/11 19:18:24 Jun Rao via dev wrote: > >>>>>>>>> Hi, Andrew, > >>>>>>>>> > >>>>>>>>> Thanks for the reply. > >>>>>>>>> > >>>>>>>>> JR20. "If the client requests a new client instance ID on its > >>>>> initial > >>>>>>>>> GetTelemetrySubscriptions request and it sends a client > >>> instance > >>>>> ID in > >>>>>>>> the > >>>>>>>>> request header, the broker will send back that client instance > >>> ID > >>>>> rather > >>>>>>>>> than generating a new UUID. This will automatically align the > >>> UUID > >>>>> in the > >>>>>>>>> request headers and client telemetry." > >>>>>>>>> > >>>>>>>>> This seems inconsistent with what's in the table. In the > >>> table, for > >>>>>>>>> example, if the client has the following: > >>>>>>>>> GetTelemetrySubscriptions v0 > >>>>>>>>> header.ClientInstanceId = UUID-H > >>>>>>>>> request.ClientInstanceId = UUID-H > >>>>>>>>> > >>>>>>>>> or > >>>>>>>>> > >>>>>>>>> GetTelemetrySubscriptions v0 > >>>>>>>>> header.ClientInstanceId = UUID-H > >>>>>>>>> request.ClientInstanceId = UUID-R > >>>>>>>>> > >>>>>>>>> the broker returns > >>>>>>>>> response.ClientInstanceId = 0. > >>>>>>>>> > >>>>>>>>> JR21. It will be useful to document what the new client does > >>> with > >>>>> the > >>>>>>>>> returned response.ClientInstanceId. Note that return value may > >>> or > >>>>> may not > >>>>>>>>> be 0. > >>>>>>>>> > >>>>>>>>> JR22. It's probably clearer if we could populate the table > >>> with 4 > >>>>>>>>> combinations: old/new clients with old/new brokers. > >>>>>>>>> > >>>>>>>>> Jun > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Fri, May 8, 2026 at 2:49 AM Andrew Schofield < > >>>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi Jun and Chia-Ping, > >>>>>>>>>> I've overhauled part of the KIP to do with alignment of the > >>>>> request > >>>>>>>> header > >>>>>>>>>> client instance ID, client telemetry client instance ID and > >>> group > >>>>>>>> protocol > >>>>>>>>>> member IDs. The alignment is by convention, not mandate > >>> (SHOULD > >>>>> not > >>>>>>>> MUST). > >>>>>>>>>> > >>>>>>>>>> It would be possible to go around the existing RPCs such as > >>>>>>>>>> ConsumerGroupHeartbeat and GetTelemetrySubscriptions, and > >>> remove > >>>>> the > >>>>>>>> fields > >>>>>>>>>> containing the existing identifiers which are intended to be > >>>>> aligned. > >>>>>>>> Doing > >>>>>>>>>> so would be a bad idea though, because we would then have RPC > >>>>> versions > >>>>>>>>>> which essentially depend upon the presence of a tagged field > >>> in > >>>>> the > >>>>>>>> request > >>>>>>>>>> header. This is a protocol-compatibility nightmare. > >>>>>>>>>> > >>>>>>>>>> I have removed the new versions of GetTelemetrySubscriptions > >>> and > >>>>>>>>>> PushTelemetry. I have also explained the behavior of > >>>>>>>>>> GetTelemetrySubscriptions in the presence and absence of a > >>> client > >>>>>>>> instance > >>>>>>>>>> ID in the request header. > >>>>>>>>>> > >>>>>>>>>> Let me know what you think. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Andrew > >>>>>>>>>> > >>>>>>>>>> On 2026/05/07 15:09:31 Andrew Schofield wrote: > >>>>>>>>>>> Hi Jun and Chia-Ping, > >>>>>>>>>>> I've been thinking and discussing the changes to the > >>> KIP-714 > >>>>> RPCs. > >>>>>>>> There > >>>>>>>>>> are too many combinations for my liking at the moment. I > >>> want to > >>>>> take > >>>>>>>>>> another pass at this area and will make an update in a few > >>> days. > >>>>>>>>>>> > >>>>>>>>>>> I intend to start a new vote once we have consensus because > >>>>> the spec > >>>>>>>> has > >>>>>>>>>> changed somewhat since the earliest votes. > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Andrew > >>>>>>>>>>> > >>>>>>>>>>> On 2026/05/06 17:28:27 Chia-Ping Tsai wrote: > >>>>>>>>>>>> hi Andrew > >>>>>>>>>>>> > >>>>>>>>>>>> chia_0: If the consensus is to remove the "duplicate" > >>> field > >>>>> from > >>>>>>>> the > >>>>>>>>>> RPC payloads, the tagged field in the header will essentially > >>>>> become a > >>>>>>>>>> required field. This means the broker needs to handle the > >>> edge > >>>>> case > >>>>>>>> where > >>>>>>>>>> both the header and the request body have no > >>> ClientInstanceId, > >>>>> right? > >>>>>>>> If > >>>>>>>>>> so, would you mind clarifying the expected broker behavior in > >>>>> the KIP? > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> Chia-Ping > >>>>>>>>>>>> > >>>>>>>>>>>> On 2026/04/03 16:17:37 Andrew Schofield wrote: > >>>>>>>>>>>>> Hi, > >>>>>>>>>>>>> I would like to start the discussion on KIP-1313. This > >>>>> adds a > >>>>>>>> unique > >>>>>>>>>> client instance ID to the request header of all Kafka > >>> protocol > >>>>>>>> requests to > >>>>>>>>>> give a unique identifier which can be used to correlate the > >>>>> requests > >>>>>>>> from > >>>>>>>>>> each client for the purposes of problem determination. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>> > >>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1313*3A*Client*instance*ID*in*all*request*headers__;JSsrKysrKys!!Ayb5sqE7!uqWf0-b_X82WmpmCYImD2W2rht_s_q5vHcqB9ToMV4IaeQbZF42eMJyS5XC5b5qE_qJJUj3KTCXcqEvYbwYS$ > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Andrew > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >
