Hi Jun, I've updated the KIP. JS34: I don't believe that we need to introduce any new feature versions for this KIP. For example, we do not need a new ShareVersion since that enabled share groups (SV_1) and controls the versioning of the records on the internal topics (SV_2). I think the same applies for the others. I understand GroupVersion and StreamsVersion, and I think they follow the same reasoning as ShareVersion. I don't know TransactionVersion or EligibleLeaderReplicasVersion, so let me know whether those are different in this regard.
Thanks, Andrew On 2026/06/25 21:32:27 Andrew Schofield wrote: > Hi Jun, > Thanks for the reply. > > JR27: Yes, makes sense. So request header v3 is binary-compatible with > request header v2. > > JS34: Yes, I think so. > > I'll update the KIP tomorrow morning. > > Thanks, > Andrew > > On 2026/06/25 20:23:57 Jun Rao via dev wrote: > > Hi, Andrew, > > > > Thanks for the reply. > > > > JR27. I was thinking of the following, which is slightly different from > > what you described. > > * Bump RequestHeader to v3, and introduce ClientInstanceId as a tagged > > field for versions 3+ and change the semantic of the existing field > > clientId (only need to be populated by the first request). > > * Bump the request versions for all RPCs including ApiVersion and map to > > the new version to use v3 of RequestHeader. > > * On the server side, when a new request version is seen, we enforce the > > presence of ClientInstanceId and reject the request if it is missing. > > Essentially, treat it as a required field. We also cache the clientId from > > the first request and ignore it from subsequent ones. > > * We would introduce OffsetDelete v1 as the first flexible version so that > > it uses the v3 request header. > > * We would update PushTelemetry and GetTelemetrySubscriptions as described > > in the KIP and the ClientInstanceId would move to the request header. > > > > In the future, if we want to add a new truly optional field to > > RequestHeader, we can continue adding it as a tagged field without bumping > > up the request version. To add a required field, we will add it as a tagged > > field but force a version bump to all requests. > > > > The benefit of this approach is that (1) it's compatible with the format of > > the old request header; (2) it avoids duplicating new fields; and (3) there > > is no constraint on deprecating the old request header. It's slightly weird > > to bump up the version when adding a tagged field. But it still seems > > better than the alternatives. > > > > JR34. Since we now have different feature versions gating different RPCs, > > should we bump up GV, TV, etc., too? > > > > Jun > > > > On Thu, Jun 25, 2026 at 10:37 AM Andrew Schofield <[email protected]> > > wrote: > > > > > Hi Jun, > > > Thanks for the reply. > > > > > > JR27.1: What if we want to add non-optional fields to the request header > > > in the future? I think that such fields would be by definition future > > > fields which would not be understood by existing brokers. For > > > serialization > > > reasons, I think we would use version 2+ tagged fields and the requirement > > > for non-optionality would be enforced by code, rather than by the protocol > > > schema. This avoids the field duplication. > > > > > > JR27.2: Yes, I don't like it, but when we inevitably do another KIP-896 to > > > remove support for ancient parts of the protocol, it's going to be another > > > KIP. That's all I meant. > > > > > > If I understand your proposal correctly compared with the existing KIP > > > text, it would look like this: > > > * Remove RequestHeader v3, and introduce ClientInstanceId as a tagged > > > field for versions 2+. > > > * Bump the request versions for all RPCs. We'd update the request JSON > > > schemas with something like "ClientInstanceIdVersions": "X+". This would > > > mean that the request would only be properly formed if it had the > > > ClientInstanceId present, and would also mean that ClientId could be null > > > for non-initial requests on each connection. > > > * We would create ApiVersions v6, simply as a matter of the bumping of > > > request versions for all RPCs, but its schema would be unchanged from v5. > > > * We would introduce OffsetDelete v1 as the first flexible version so that > > > it uses the v2 request header. > > > * We would update PushTelemetry and GetTelemetrySubscriptions as described > > > in the KIP and the ClientInstanceId would move to the request header. > > > > > > I think that works roughly the same as what we have today. I don't really > > > like the mandatory nature of the tagged field, but then I don't really > > > like > > > the way that ApiVersions is pegged on a historical request header version. > > > > > > Do I have this correct? If so, I can update the KIP accordingly. > > > > > > Thanks, > > > Andrew > > > > > > On 2026/06/24 23:40:06 Jun Rao via dev wrote: > > > > Hi, Andrew, > > > > > > > > Thanks for the reply. > > > > > > > > JR27.1 What if we want to add non-optional fields to the request header > > > in > > > > the future? The choice I see is duplicating the field in the new request > > > > header and the new ApiVersion request. But it's not very clean. > > > > JR27.2 Requiring a major version API compatibility KIP is also a bit > > > > awkward. > > > > > > > > Here is yet another proposal. We add the ClientInstanceId to the request > > > > header as a tagged field and also bump up all request versions. This > > > avoids > > > > the header compatibility issue in SocketServer. The bumped-up version > > > > signals that the new field is actually required. On the server side, > > > > when > > > > receiving the new version of a request, we verify that the > > > ClientInstanceId > > > > field exists. If it's not present, we reject the request. This will be > > > the > > > > convention for adding future required fields to the header. Compared to > > > the > > > > current proposal, this avoids duplicating the field between request > > > header > > > > and ApiVersionRequest, and it provides a path for deprecating the older > > > > version of the request header. What do you think? > > > > > > > > Jun > > > > > > > > On Wed, Jun 24, 2026 at 2:15 AM Andrew Schofield <[email protected]> > > > > wrote: > > > > > > > > > Hi Jun, > > > > > Thanks for the reply. > > > > > > > > > > JR27: Yes, I enhanced that section of the KIP. > > > > > JR27.1: This KIP was unusual because we wanted to add a concept which > > > > > existed in the KIP-714 RPCs to the request header, complicating the > > > > > behavior of those RPCs, AND we wanted to change the handling of client > > > ID > > > > > without breaking older brokers. If we simply wanted to add a brand new > > > > > field to the request header, we can do it as a tagged field in request > > > > > header versions 2+. > > > > > JR27.2: Covered this too. We'd need a major version API compatibility > > > KIP. > > > > > > > > > > JR31: Yes. KIP updated. > > > > > > > > > > JR32: Yes. This is a minor inconvenience I feel. KIP updated. > > > > > > > > > > JR33: I don't see the value of a string comparison here. Personally, > > > I'm > > > > > happy with "assume". > > > > > > > > > > Thanks, > > > > > Andrew > > > > > > > > > > On 2026/06/23 22:52:15 Jun Rao via dev wrote: > > > > > > Hi, Andrew, > > > > > > > > > > > > Thanks for the reply. > > > > > > > > > > > > JR27. The code that changed the request header parsing was > > > introduced in > > > > > > 2019 ( > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/kafka/pull/7372__;!!Ayb5sqE7!scbWbTEEm2o-Y3MBGi829H2cpMWPRyrF9g2CW3lmRHyRmgBU2MyoRif2mFJpnfmvQ2x7fF4LrAJXdwH86ERR$ > > > > > ). I agree that fixing it > > > > > > now is too late. We should document the path forward for ApiVersion. > > > > > Could > > > > > > we answer at least the following questions in the Compatibility, > > > > > > Deprecation, and Migration Plan section? > > > > > > JR27.1 What happens to ApiVersion if we change the request header in > > > the > > > > > > future? > > > > > > JR27.2 What happens if we ever want to deprecate v2 of the request > > > > > header? > > > > > > > > > > > > JR28. The new MV needs to be covered in the Compatibility, > > > Deprecation, > > > > > and > > > > > > Migration Plan section. > > > > > > > > > > > > JR31. "If a connection uses the v3 request header in its first > > > request, > > > > > it > > > > > > must specify a non-zero client instance ID and it must specify the > > > same > > > > > > client instance ID for all subsequent requests." > > > > > > Do we do the same if the first request is v6 of ApiVersion? > > > > > > > > > > > > JR32. "For subsequent requests using an earlier version of the > > > request > > > > > > header, the client sends the client ID". This means that subsequent > > > > > > ApiVersion V6 requests will carry the non-null clientID > > > unnecessarily. > > > > > > > > > > > > JR33. "After this KIP, the broker will assume that the client ID > > > from the > > > > > > initial request applies to all subsequent requests on a connection, > > > and > > > > > it > > > > > > will ignore the client ID specified on any subsequent requests." > > > > > > Should the broker verify that the clientId from subsequent request > > > match > > > > > > the first cached one, if not null? > > > > > > > > > > > > Jun > > > > > > > > > > > > On Tue, Jun 23, 2026 at 6:05 AM Andrew Schofield < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Jun, > > > > > > > Thanks for your reply. > > > > > > > > > > > > > > JR27: Unfortunately, I don't think this is acceptable. We would be > > > > > saying > > > > > > > that AK 4.5 clients can only connect to AK 4.5 or later brokers, > > > and > > > > > that > > > > > > > we'd create patch release for AK 4.0.x to AK 4.4.x. so AK 4.5 > > > clients > > > > > could > > > > > > > connect to those too. We'd be preventing connection to AK 3.x > > > brokers > > > > > by AK > > > > > > > 4.5 clients. I don't see how we can break compatibility in a minor > > > > > release. > > > > > > > Here's the existing compatibility table: > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://kafka.apache.org/41/getting-started/compatibility/__;!!Ayb5sqE7!ppWU_jI9f7EY9DYwcbiOESSBG2TmPn9CHfLnVM5TaIMwWqHUbUbIiGre1GGKMr-HmelQlyP6ibUSyORCAgte$ > > > > > > > > > > > > > > JR29: OK, makes sense to me. > > > > > > > > > > > > > > Thanks, > > > > > > > Andrew > > > > > > > > > > > > > > On 2026/06/22 21:50:43 Jun Rao via dev wrote: > > > > > > > > Hi, Andrew, > > > > > > > > > > > > > > > > Thanks for the reply. > > > > > > > > > > > > > > > > JR27. I am not sure I like the approach of pinning the header to > > > v2 > > > > > for > > > > > > > > ApiVersionsRequest. This means that in the future, each change > > > to the > > > > > > > > request header must be duplicated in the request itself. Here is > > > an > > > > > > > > alternative. Note that to detect an unsupported > > > ApiVersionRequest, we > > > > > > > don't > > > > > > > > need to parse the full v2 request header. Parsing the first two > > > > > fields > > > > > > > > ApiKey and ApiVersion is enough. So, we could patch all the 4.x > > > > > releases > > > > > > > > with a change that short-circuits parsing the full request > > > > > > > > header > > > > > based > > > > > > > on > > > > > > > > the first two fields. Then, we recommend upgrading from the > > > latest > > > > > 4.x > > > > > > > > releases to future releases. This way, we can still let > > > > > > > ApiVersionsRequest > > > > > > > > depend on the request header for current and future changes. > > > > > > > > > > > > > > > > JR29. The simplest thing is to upgrade the KRaft RPCs too to be > > > > > > > consistent. > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > On Sat, Jun 20, 2026 at 3:38 AM Andrew Schofield < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Jun, > > > > > > > > > Thanks for these insightful comments. I did prototype the > > > earlier > > > > > v2 > > > > > > > > > request header with tagged fields variant. I have now > > > prototyped > > > > > the v3 > > > > > > > > > request header variant. > > > > > > > > > > > > > > > > > > JR27: You were of course entirely correct that a new version > > > > > > > > > of > > > > > > > > > ApiVersions request cannot use a request header which is not > > > > > > > > > binary-compatible with what has gone before. As a result, I > > > have > > > > > > > introduced > > > > > > > > > a new version of ApiVersions request which includes > > > > > ClientInstanceId > > > > > > > in the > > > > > > > > > request body but continues to use the v2 request header. > > > > > > > > > > > > > > > > > > JR28: Yes, we need a new metadata version, which is > > > provisionally > > > > > > > > > IBP_4_5_IV0. I'll update the KIP with the final value when it > > > is > > > > > > > assigned. > > > > > > > > > > > > > > > > > > JR29: I don't have any strong feelings on this. I suppose for > > > each > > > > > > > inter > > > > > > > > > broker RPC version that is bumped, we need to introduce an MV > > > > > check. We > > > > > > > > > cannot avoid this for RPCs which are also used by clients such > > > as > > > > > > > Fetch. > > > > > > > > > For purely inter-broker RPCs, we could just exempt them from > > > the > > > > > > > version > > > > > > > > > bump. What do you recommend here? > > > > > > > > > > > > > > > > > > JR30: Done. INVALID_REQUEST. I did consider permitting zero > > > client > > > > > > > > > instance ID for inter-broker connections, but I decided that I > > > > > > > preferred > > > > > > > > > not creating another rule. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > On 2026/06/17 22:45:07 Andrew Schofield wrote: > > > > > > > > > > Hi Jun, > > > > > > > > > > Thanks for your response. > > > > > > > > > > > > > > > > > > > > JR27: I am familiar with KIP-511 and the oddness of > > > ApiVersions. > > > > > You > > > > > > > > > make an excellent point. I suspect that ApiVersionsRequest > > > > > > > > > will > > > > > have > > > > > > > to be > > > > > > > > > limited to the v2 request header, just as it is restricted to > > > the > > > > > v0 > > > > > > > > > response header. > > > > > > > > > > > > > > > > > > > > I'll do some experimentation. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > On 2026/06/17 17:35:17 Jun Rao via dev wrote: > > > > > > > > > > > Hi, Andrew, > > > > > > > > > > > > > > > > > > > > > > Thanks for the reply. A few more comments. > > > > > > > > > > > > > > > > > > > > > > JR27. It seems that we have a compatibility issue with > > > > > > > > > ApiVersionRequest v6 > > > > > > > > > > > depending on RequestHeader v3. This happens when a new > > > client > > > > > > > issues > > > > > > > > > > > ApiVersionRequest v6 to an old broker (one that only > > > supports > > > > > > > > > > > ApiVersionRequest v5). The SocketServer in the broker will > > > > > first > > > > > > > try to > > > > > > > > > > > read the request header. It tries to find the header > > > version > > > > > from > > > > > > > the > > > > > > > > > > > request version. Since the broker doesn't know > > > > > ApiVersionRequest > > > > > > > v6, it > > > > > > > > > > > uses v2 to parse the request header. > > > > > > > > > > > > > > > > > > > > > > The v2's wire format for the request header is: ApiKey | > > > > > > > ApiVersion | > > > > > > > > > > > CorrelationId | ClientId | tagged-fields-varint > > > > > > > > > > > The v3's wire format for the request header is: ApiKey | > > > > > > > ApiVersion | > > > > > > > > > > > CorrelationId | ClientId | ClientInstanceId(16 bytes) | > > > > > > > tagged-fields > > > > > > > > > varint > > > > > > > > > > > The first 4 fields are identical, allowing the broker to > > > parse > > > > > > > them. > > > > > > > > > The > > > > > > > > > > > broker then tries to read the tagged-fields-varint using > > > the > > > > > > > serialized > > > > > > > > > > > bytes intended for ClientInstanceId. This could lead to a > > > > > parsing > > > > > > > > > error and > > > > > > > > > > > raise an invalid request error. The broker will close the > > > > > > > connection > > > > > > > > > > > instead of sending a v0 response for ApiVersionRequest. > > > Then > > > > > the > > > > > > > client > > > > > > > > > > > will never be able to detect supported API versions on the > > > old > > > > > > > broker. > > > > > > > > > > > > > > > > > > > > > > The same problem didn't exist when ApiVersionRequest > > > changed > > > > > from > > > > > > > the > > > > > > > > > v1 > > > > > > > > > > > request header to v2 because the wire format for v1 is a > > > > > prefix of > > > > > > > v2. > > > > > > > > > This > > > > > > > > > > > problem exists for this KIP because the wire format of v2 > > > is no > > > > > > > longer > > > > > > > > > a > > > > > > > > > > > prefix of v3. > > > > > > > > > > > > > > > > > > > > > > JR28. Since we are bumping up the version of inter broker > > > > > > > requests, we > > > > > > > > > need > > > > > > > > > > > to gate them by introducing a new MV. > > > > > > > > > > > > > > > > > > > > > > JR29. Could you specify whether the version for KRaft PRCs > > > > > will be > > > > > > > > > bumped > > > > > > > > > > > too? > > > > > > > > > > > > > > > > > > > > > > JR30. "If a connection uses the v3 request header in its > > > first > > > > > > > > > request, it > > > > > > > > > > > must specify a non-zero client instance ID" > > > > > > > > > > > Could we define the broker's behavior when a v3 request > > > header > > > > > > > > > contains a 0 > > > > > > > > > > > client instance ID? > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 16, 2026 at 1:25 AM Andrew Schofield < > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > Thanks for your response. > > > > > > > > > > > > > > > > > > > > > > > > JR26: Yes, I was considering open-ended version ranges. > > > I've > > > > > > > updated > > > > > > > > > the > > > > > > > > > > > > KIP. > > > > > > > > > > > > > > > > > > > > > > > > Now that we have a new request header version, I think > > > it's > > > > > safe > > > > > > > to > > > > > > > > > > > > reintroduce the client ID optimisation. I've added that > > > into > > > > > the > > > > > > > KIP > > > > > > > > > also. > > > > > > > > > > > > > > > > > > > > > > > > Let me know what you think. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > On 2026/06/15 17:46:00 Jun Rao via dev wrote: > > > > > > > > > > > > > Hi, Andrew, > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the reply. Just a minor comment. > > > > > > > > > > > > > > > > > > > > > > > > > > JR26. "The keys of the map are either single versions > > > like > > > > > "0" > > > > > > > or > > > > > > > > > closed > > > > > > > > > > > > > version ranges like "1-2"." > > > > > > > > > > > > > It will be useful to support open-ended versions. That > > > way, > > > > > > > "3": > > > > > > > > > "3" > > > > > > > > > > > > could > > > > > > > > > > > > > be "3+": "3". This part won't need to change in the > > > future > > > > > > > when we > > > > > > > > > bump > > > > > > > > > > > > up > > > > > > > > > > > > > the request version without changing the header. > > > > > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2026 at 2:12 PM Andrew Schofield < > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Jun, > > > > > > > > > > > > > > Thanks for your response. > > > > > > > > > > > > > > > > > > > > > > > > > > > > JR25: I agree. I've changed to a headerVersions map > > > in > > > > > both > > > > > > > the > > > > > > > > > request > > > > > > > > > > > > > > and response schemas. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let me know what you think. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2026/06/12 03:39:39 Jun Rao via dev wrote: > > > > > > > > > > > > > > > Hi, Andrew, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the updated KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > JR25. I have some concerns about adding the > > > > > > > > > clientInstanceIdVersions > > > > > > > > > > > > > > field > > > > > > > > > > > > > > > to the schema. First of all, it only applies to > > > > > requests, > > > > > > > but > > > > > > > > > not > > > > > > > > > > > > > > > responses. So, it's a bit weird to add it to the > > > > > general > > > > > > > > > schema. > > > > > > > > > > > > > > Secondly, > > > > > > > > > > > > > > > it's very specific to the new field added to the > > > > > request > > > > > > > > > header. If > > > > > > > > > > > > we > > > > > > > > > > > > > > > change the request header again in the future, we > > > need > > > > > > > another > > > > > > > > > way > > > > > > > > > > > > to map > > > > > > > > > > > > > > > request versions to the new header version. > > > Ideally, we > > > > > > > should > > > > > > > > > have a > > > > > > > > > > > > > > > generic way to map request versions to all future > > > > > header > > > > > > > > > versions. > > > > > > > > > > > > One > > > > > > > > > > > > > > > possibility is to introduce a headerVersion field > > > in > > > > > the > > > > > > > > > schema. > > > > > > > > > > > > Then, > > > > > > > > > > > > > > each > > > > > > > > > > > > > > > request/response can use this field to map its > > > > > versions to > > > > > > > the > > > > > > > > > header > > > > > > > > > > > > > > > versions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 9, 2026 at 4:55 AM Andrew Schofield < > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Matthias, > > > > > > > > > > > > > > > > Thanks for your response. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have change the version property for > > > introducing > > > > > client > > > > > > > > > instance > > > > > > > > > > > > ID > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > > the request headers to be > > > `clientInstanceIdVersions` > > > > > > > which > > > > > > > > > is now > > > > > > > > > > > > an > > > > > > > > > > > > > > > > open-ended range, such as "3+", more closely > > > matching > > > > > > > what > > > > > > > > > we have > > > > > > > > > > > > for > > > > > > > > > > > > > > > > flexible versions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I also added a change to the > > > > > > > > > RebalanceConsumer.clientInstanceId > > > > > > > > > > > > method > > > > > > > > > > > > > > > > introduced in KIP-1306. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2026/06/03 18:31:06 "Matthias J. Sax" wrote: > > > > > > > > > > > > > > > > > Thanks Andrew. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From a KIP-714 perspective, this is a great > > > > > change, > > > > > > > as it > > > > > > > > > make > > > > > > > > > > > > the > > > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > RPCs and client/broker cross-version > > > compatibility > > > > > > > clean. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One question about your `DeleteGroups` RPC > > > example. > > > > > > > Should > > > > > > > > > the > > > > > > > > > > > > new > > > > > > > > > > > > > > > > > version of the RPC say > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "clientInstanceIdVersion": "3+", > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ie, 3+ instead of 3? Similar for other > > > > > > > > > > > > > > > > > request, > > > > > which > > > > > > > > > don't use > > > > > > > > > > > > `+` > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > > `clientInstanceIdVersion` field. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 6/2/26 7:42 AM, Andrew Schofield wrote: > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > Following on from the discussion in the > > > > > community, > > > > > > > and > > > > > > > > > the > > > > > > > > > > > > > > difficulty > > > > > > > > > > > > > > > > of trying to accommodate the client instance ID > > > in a > > > > > > > tagged > > > > > > > > > field > > > > > > > > > > > > with > > > > > > > > > > > > > > > > acceptable complexity, I have made a significant > > > > > change > > > > > > > to > > > > > > > > > the KIP. > > > > > > > > > > > > > > Now, it > > > > > > > > > > > > > > > > introduces v3 of the request header containing > > > the > > > > > client > > > > > > > > > instance > > > > > > > > > > > > ID > > > > > > > > > > > > > > as a > > > > > > > > > > > > > > > > proper field. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1313*3A*Client*instance*ID*in*all*request*headers__;JSsrKysrKys!!Ayb5sqE7!pUZS25PuTKdhiqvTGcwDKbclqpG8dFOzw2Ax9WxVM6Sb09PVQ0gl1i0gjlr2_Krr66agbuVC2qLGgUcg7vrW$ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please review and let me know what you > > > > > > > > > > > > > > > > > > think. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2026/05/28 21:51:01 "Matthias J. Sax" > > > wrote: > > > > > > > > > > > > > > > > > >> Thanks Andrew. I agree to the issue that > > > tagged > > > > > > > fields > > > > > > > > > are > > > > > > > > > > > > > > optional. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> Nevertheless, I find the proposed > > > compatibility > > > > > > > > > protocol too > > > > > > > > > > > > > > complex > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > >> unnecessary, and would still prefer a > > > version > > > > > bump > > > > > > > for > > > > > > > > > > > > > > > > > >> `GetTelemetrySubscriptionsRequest/Reponse` > > > to > > > > > clean > > > > > > > it > > > > > > > > > all > > > > > > > > > > > > up. -- > > > > > > > > > > > > > > For > > > > > > > > > > > > > > > > > >> `PushTelemetryRequest` the argument from > > > above > > > > > > > applies, > > > > > > > > > so > > > > > > > > > > > > > > leaving it > > > > > > > > > > > > > > > > > >> untouched is ok. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> I don't see why > > > > > `GetTelemetrySubscriptionsRequest` > > > > > > > > > would need > > > > > > > > > > > > to > > > > > > > > > > > > > > > > encode > > > > > > > > > > > > > > > > > >> the `clientInstandId` in it's request body. > > > > > Given > > > > > > > that > > > > > > > > > the > > > > > > > > > > > > > > > > > >> `clientInstanceId` is a random UUID, the > > > > > returned > > > > > > > > > subscription > > > > > > > > > > > > > > cannot > > > > > > > > > > > > > > > > > >> really depend on it. So why would we need > > > > > > > > > > > > > > > > > >> to > > > > > send > > > > > > > it to > > > > > > > > > the > > > > > > > > > > > > > > broker? > > > > > > > > > > > > > > > > > >> Right now, it's set to ZERO only for the > > > > > purpose to > > > > > > > get > > > > > > > > > the > > > > > > > > > > > > > > broker to > > > > > > > > > > > > > > > > > >> assign a new UUID. But it has nothing to do > > > > > with the > > > > > > > > > actual > > > > > > > > > > > > "what > > > > > > > > > > > > > > > > > >> metrics should I send" question. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> To be fair, KIP-714 lists > > > `client_instance_id` > > > > > as a > > > > > > > > > field that > > > > > > > > > > > > > > can be > > > > > > > > > > > > > > > > > >> used to define a subscription. But it seems > > > > > rather > > > > > > > > > useless in > > > > > > > > > > > > > > practice > > > > > > > > > > > > > > > > > >> to me? I a user defines a subscription, > > > > > > > > > > > > > > > > > >> they > > > > > cannot > > > > > > > > > know the > > > > > > > > > > > > UUID. > > > > > > > > > > > > > > > > Thus, > > > > > > > > > > > > > > > > > >> I think we should actually drop supporting > > > > > > > > > > > > `client_instance_id` > > > > > > > > > > > > > > as a > > > > > > > > > > > > > > > > > >> "subscription matching parameter"? Of > > > course, > > > > > there > > > > > > > is a > > > > > > > > > > > > backward > > > > > > > > > > > > > > > > > >> compatibility question, but I think we can > > > > > address > > > > > > > > > this: if > > > > > > > > > > > > > > brokers > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > > >> upgraded and they have an existing > > > > > > > `client_instance_id` > > > > > > > > > based > > > > > > > > > > > > > > > > > >> subscription defined, they could advertise > > > to > > > > > only > > > > > > > > > support > > > > > > > > > > > > v0, and > > > > > > > > > > > > > > > > they > > > > > > > > > > > > > > > > > >> should log a warning that this feature is > > > > > > > deprecated. > > > > > > > > > New > > > > > > > > > > > > brokers > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > >> also not allow to use `client_instance_id` > > > any > > > > > > > longer to > > > > > > > > > > > > define a > > > > > > > > > > > > > > > > > >> subscription. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> Similarly, for > > > > > `GetTelemetrySubscriptionsResponse`, > > > > > > > we > > > > > > > > > only > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > >> `clientInstanceId` if the broker computes > > > the > > > > > UUID. > > > > > > > But > > > > > > > > > if we > > > > > > > > > > > > let > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >> client compute it, this field is not needed > > > any > > > > > > > longer. > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> -Matthias > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> On 5/28/26 1:21 AM, Andrew Schofield wrote: > > > > > > > > > > > > > > > > > >>> Hi Jun, > > > > > > > > > > > > > > > > > >>> Thanks for your response. > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > >>> JR24: I'll update the KIP shortly. Here's > > > > > > > > > > > > > > > > > >>> a > > > > > > > summary. > > > > > > > > > The > > > > > > > > > > > > various > > > > > > > > > > > > > > > > modern group protocol heartbeat requests encode > > > the > > > > > > > member > > > > > > > > > ID as a > > > > > > > > > > > > > > string > > > > > > > > > > > > > > > > for historical reasons, even though it's in > > > practice > > > > > a > > > > > > > UUID. > > > > > > > > > The > > > > > > > > > > > > Apache > > > > > > > > > > > > > > > > Kafka Java client happens to use > > > > > > > > > org.apache.kafka.common.Uuid to > > > > > > > > > > > > > > create the > > > > > > > > > > > > > > > > member ID and convert it into a string. This > > > means > > > > > that > > > > > > > when > > > > > > > > > the > > > > > > > > > > > > > > > > clientInstanceId is converted into a string, its > > > > > encoding > > > > > > > > > matches > > > > > > > > > > > > the > > > > > > > > > > > > > > > > member ID and it all lines up. However, there > > > > > > > > > > > > > > > > are > > > > > other > > > > > > > ways > > > > > > > > > to > > > > > > > > > > > > encode > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > UUID into a string and I know for a fact that > > > > > > > > > > > > > > > > the > > > > > > > librdkafka > > > > > > > > > client > > > > > > > > > > > > > > does it > > > > > > > > > > > > > > > > differently. > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > >>> Thanks, > > > > > > > > > > > > > > > > > >>> Andrew > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > >>> On 2026/05/27 17:11:25 Jun Rao via dev > > > wrote: > > > > > > > > > > > > > > > > > >>>> Hi, Andrew, > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> Thanks for the reply. One more comment. > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> JR24. "The alignment of other identifiers > > > is > > > > > by > > > > > > > > > convention > > > > > > > > > > > > (and > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > Java > > > > > > > > > > > > > > > > > >>>> client will follow the convention) rather > > > than > > > > > > > > > mandate." > > > > > > > > > > > > Could > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > describe > > > > > > > > > > > > > > > > > >>>> the convention to convert a > > > clientInstanceId > > > > > > > (UUID) > > > > > > > > > to a > > > > > > > > > > > > > > memberId > > > > > > > > > > > > > > > > (String)? > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> Jun > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > >>>> On Tue, May 19, 2026 at 2:36 AM Andrew > > > > > Schofield < > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > >>>> wrote: > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > >>>>> Hi Jun, > > > > > > > > > > > > > > > > > >>>>> Thanks for your response. > > > > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > > > > >>>>> JR23: You are absolutely correct. It > > > seems > > > > > to me > > > > > > > > > that not > > > > > > > > > > > > > > sending a > > > > > > > > > > > > > > > > > >>>>> clientInstanceId in the header and > > > explicitly > > > > > > > > > sending a > > > > > > > > > > > > zero > > > > > > > > > > > > > > UUID > > > > > > > > > > > > > > > > as the > > > > > > > > > > > > > > > > > >>>>> clientInstanceId in the header can be > > > > > treated as > > > > > > > > > > > > semantically > > > > > > > > > > > > > > > > equivalent. > > > > > > > > > > > > > > > > > >>>>> I've tweaked the words slightly. > > > > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > > > > >>>>> Thanks, > > > > > > > > > > > > > > > > > >>>>> Andrew > > > > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > > > > >>>>> On 2026/05/19 03:42:16 Jun Rao via dev > > > wrote: > > > > > > > > > > > > > > > > > >>>>>> Hi, Andrew, > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > >>>>>> Thanks for the reply. > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > >>>>>> JR23. Our message protocol doc says > > > > > > > > > > > > > > > > > >>>>>> "Any > > > > > fields > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > message > > > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > > >>>>> that > > > > > > > > > > > > > > > > > >>>>>> are not present in the version that you > > > are > > > > > > > > > deserializing > > > > > > > > > > > > > > will be > > > > > > > > > > > > > > > > reset > > > > > > > > > > > > > > > > > >>>>> to > > > > > > > > > > > > > > > > > >>>>>> default values. Unless a custom > > > default has > > > > > > > been > > > > > > > > > set:". > > > > > > > > > > > > Uuid > > > > > > > > > > > > > > > > fields > > > > > > > > > > > > > > > > > >>>>>> default to zero uuid. > > > > > > > > > > > > > > > > > >>>>>> So if the server gets > > > > > header.clientInstanceId=0 > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > > > > > deserialized > > > > > > > > > > > > > > > > > >>>>> header, > > > > > > > > > > > > > > > > > >>>>>> could it distinguish between the ID not > > > > > being > > > > > > > > > present > > > > > > > > > > > > (since > > > > > > > > > > > > > > > > client is > > > > > > > > > > > > > > > > > >>>>> old) > > > > > > > > > > > > > > > > > >>>>>> and the ID being explicitly set to 0 by > > > the > > > > > > > client? > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > >>>>>> Jun > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > >>>>>> On Mon, May 18, 2026 at 7:45 PM Andrew > > > > > > > Schofield < > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > > > >>>>>> wrote: > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > >>>>>>> Hi Jun, > > > > > > > > > > > > > > > > > >>>>>>> Thanks for your reply. It's tricky > > > > > squaring a > > > > > > > > > circle. > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> JR23: For GetTelemetrySubscriptions, I > > > have > > > > > > > > > changed it so > > > > > > > > > > > > > > that a > > > > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > > >>>>>>> which omits the ClientInstanceId from > > > the > > > > > > > request > > > > > > > > > header > > > > > > > > > > > > is > > > > > > > > > > > > > > > > permitted > > > > > > > > > > > > > > > > > >>>>> to > > > > > > > > > > > > > > > > > >>>>>>> specify a zero ClientInstanceId in the > > > > > request > > > > > > > > > body, > > > > > > > > > > > > > > following > > > > > > > > > > > > > > > > original > > > > > > > > > > > > > > > > > >>>>>>> KIP-714 precedent. However, a client > > > which > > > > > > > > > specifies a > > > > > > > > > > > > > > > > > >>>>> ClientInstanceId in > > > > > > > > > > > > > > > > > >>>>>>> the request header MUST specify the > > > same > > > > > > > > > > > > ClientInstanceId in > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >>>>> request > > > > > > > > > > > > > > > > > >>>>>>> body. This ensures that the header and > > > > > > > telemetry > > > > > > > > > UUIDs > > > > > > > > > > > > are > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > same. > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> Thanks, > > > > > > > > > > > > > > > > > >>>>>>> Andrew > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> On 2026/05/12 17:48:23 Andrew > > > > > > > > > > > > > > > > > >>>>>>> Schofield > > > > > wrote: > > > > > > > > > > > > > > > > > >>>>>>>> Hi Jun, > > > > > > > > > > > > > > > > > >>>>>>>> Thanks for the reply and digging into > > > the > > > > > > > details. > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> JR23: Correct. The client telemetry > > > > > component > > > > > > > > > will use > > > > > > > > > > > > > > UUID-B > > > > > > > > > > > > > > > > as the > > > > > > > > > > > > > > > > > >>>>>>> client instance ID. > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> JR23.1: Yes, I agree. It's not ideal. > > > > > When I > > > > > > > was > > > > > > > > > > > > drawing up > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >>>>> tables, > > > > > > > > > > > > > > > > > >>>>>>> I was thinking that this might be a > > > > > > > possibility, > > > > > > > > > but I'm > > > > > > > > > > > > less > > > > > > > > > > > > > > > > convinced > > > > > > > > > > > > > > > > > >>>>>>> now. I think that I should mandate > > > that if > > > > > a > > > > > > > client > > > > > > > > > > > > specifies > > > > > > > > > > > > > > > > > >>>>>>> header.ClientInstanceId on > > > > > > > > > GetTelemetrySubscriptions > > > > > > > > > > > > request, > > > > > > > > > > > > > > > > then > > > > > > > > > > > > > > > > > >>>>>>> request.ClientInstanceId must either > > > > > > > > > > > > > > > > > >>>>>>> be > > > > > zero or > > > > > > > > > equal to > > > > > > > > > > > > > > > > > >>>>>>> header.ClientInstanceId. > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> JR23.2: This is perhaps the > > > interesting > > > > > one. > > > > > > > From > > > > > > > > > its > > > > > > > > > > > > > > original > > > > > > > > > > > > > > > > > >>>>> intent, > > > > > > > > > > > > > > > > > >>>>>>> it should be UUID-B (the telemetry > > > UUID), > > > > > but > > > > > > > then > > > > > > > > > that > > > > > > > > > > > > > > > > contradicts the > > > > > > > > > > > > > > > > > >>>>>>> change in signature to remove the > > > timeout. > > > > > > > Unless > > > > > > > > > I make > > > > > > > > > > > > the > > > > > > > > > > > > > > > > change > > > > > > > > > > > > > > > > > >>>>> above, > > > > > > > > > > > > > > > > > >>>>>>> in which case it will be UUID-H. > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> Thanks, > > > > > > > > > > > > > > > > > >>>>>>>> Andrew > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> On 2026/05/12 17:23:58 Jun Rao via > > > > > > > > > > > > > > > > > >>>>>>>> dev > > > > > wrote: > > > > > > > > > > > > > > > > > >>>>>>>>> Hi, Andrew, > > > > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>> Thanks for the reply. > > > > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>> JR23. In the new client -> old > > > > > > > > > > > > > > > > > >>>>>>>>> broker > > > > > case, > > > > > > > we > > > > > > > > > have > > > > > > > > > > > > > > > > > >>>>>>>>> header.ClientInstanceId=UUID-H > > > > > > > > > > > > > > > > > >>>>>>>>> request.ClientInstanceId=UUID-B > > > > > > > > > > > > > > > > > >>>>>>>>> response.ClientInstanceId=0 > > > > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>> On the server side, I guess the > > > telemetry > > > > > > > > > component > > > > > > > > > > > > will > > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > > >>>>> UUID-B as > > > > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > > > > >>>>>>>>> clientInstanceId? This has a couple > > > of > > > > > > > > > implications. > > > > > > > > > > > > > > > > > >>>>>>>>> JR23.1 On the server side, we have > > > two > > > > > > > different > > > > > > > > > > > > > > > > clientInstanceIds > > > > > > > > > > > > > > > > > >>>>>>> used in > > > > > > > > > > > > > > > > > >>>>>>>>> different places, UUID-H for request > > > > > logging > > > > > > > and > > > > > > > > > > > > UUID-B in > > > > > > > > > > > > > > > > > >>>>> telemetry. > > > > > > > > > > > > > > > > > >>>>>>> This > > > > > > > > > > > > > > > > > >>>>>>>>> seems confusing since we can't > > > uniquely > > > > > > > identify > > > > > > > > > a > > > > > > > > > > > > client > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >>>>> server > > > > > > > > > > > > > > > > > >>>>>>>>> side. > > > > > > > > > > > > > > > > > >>>>>>>>> JR23.2 On the client side. what uuid > > > does > > > > > > > > > > > > > > > > clientInstanceId(Duration > > > > > > > > > > > > > > > > > >>>>>>>>> timeout) return? If it returns > > > UUID-H, it > > > > > > > will be > > > > > > > > > > > > confusing > > > > > > > > > > > > > > > > since > > > > > > > > > > > > > > > > > >>>>> it > > > > > > > > > > > > > > > > > >>>>>>>>> doesn't match the ID used for > > > telemetry > > > > > on > > > > > > > the > > > > > > > > > server. > > > > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>> Jun > > > > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>> On Tue, May 12, 2026 at 12:58 AM > > > Andrew > > > > > > > > > Schofield < > > > > > > > > > > > > > > > > > >>>>>>> [email protected]> > > > > > > > > > > > > > > > > > >>>>>>>>> wrote: > > > > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>> Hi Jun, > > > > > > > > > > > > > > > > > >>>>>>>>>> Thanks for your response. > > > > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>> JR20: I have improved (I hope) the > > > > > wording. > > > > > > > The > > > > > > > > > client > > > > > > > > > > > > > > sends > > > > > > > > > > > > > > > > > >>>>>>>>>> request.clientInstanceId = 0 and > > > > > > > > > > > > header.clientInstanceId = > > > > > > > > > > > > > > > > > >>>>> UUID-H, > > > > > > > > > > > > > > > > > >>>>>>> and the > > > > > > > > > > > > > > > > > >>>>>>>>>> broker responds > > > > > > > > > response.clientInstanceId=UUID-H. In > > > > > > > > > > > > this > > > > > > > > > > > > > > way, > > > > > > > > > > > > > > > > > >>>>> the > > > > > > > > > > > > > > > > > >>>>>>> broker > > > > > > > > > > > > > > > > > >>>>>>>>>> will have taken the UUID-H from the > > > > > header, > > > > > > > and > > > > > > > > > told > > > > > > > > > > > > the > > > > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > > >>>>> to > > > > > > > > > > > > > > > > > >>>>>>> use it > > > > > > > > > > > > > > > > > >>>>>>>>>> for client telemetry also. > > > > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>> JR21: Done. Look for "henceforth". > > > > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>> JR22: Summary table added. > > > > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>> Thanks, > > > > > > > > > > > > > > > > > >>>>>>>>>> Andrew > > > > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>> On 2026/05/11 19:18:24 Jun Rao via > > > dev > > > > > > > wrote: > > > > > > > > > > > > > > > > > >>>>>>>>>>> Hi, Andrew, > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> Thanks for the reply. > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> JR20. "If the client requests a > > > > > > > > > > > > > > > > > >>>>>>>>>>> new > > > > > client > > > > > > > > > instance > > > > > > > > > > > > ID > > > > > > > > > > > > > > on its > > > > > > > > > > > > > > > > > >>>>>>> initial > > > > > > > > > > > > > > > > > >>>>>>>>>>> GetTelemetrySubscriptions request > > > and > > > > > it > > > > > > > > > sends a > > > > > > > > > > > > client > > > > > > > > > > > > > > > > > >>>>> instance > > > > > > > > > > > > > > > > > >>>>>>> ID in > > > > > > > > > > > > > > > > > >>>>>>>>>> the > > > > > > > > > > > > > > > > > >>>>>>>>>>> request header, the broker will > > > send > > > > > back > > > > > > > that > > > > > > > > > client > > > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > > > >>>>> ID > > > > > > > > > > > > > > > > > >>>>>>> rather > > > > > > > > > > > > > > > > > >>>>>>>>>>> than generating a new UUID. This > > > will > > > > > > > > > automatically > > > > > > > > > > > > > > align the > > > > > > > > > > > > > > > > > >>>>> UUID > > > > > > > > > > > > > > > > > >>>>>>> in the > > > > > > > > > > > > > > > > > >>>>>>>>>>> request headers and client > > > telemetry." > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> This seems inconsistent with > > > what's in > > > > > the > > > > > > > > > table. In > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >>>>> table, for > > > > > > > > > > > > > > > > > >>>>>>>>>>> example, if the client has the > > > > > following: > > > > > > > > > > > > > > > > > >>>>>>>>>>> GetTelemetrySubscriptions v0 > > > > > > > > > > > > > > > > > >>>>>>>>>>> header.ClientInstanceId = UUID-H > > > > > > > > > > > > > > > > > >>>>>>>>>>> request.ClientInstanceId = UUID-H > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> or > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> GetTelemetrySubscriptions v0 > > > > > > > > > > > > > > > > > >>>>>>>>>>> header.ClientInstanceId = UUID-H > > > > > > > > > > > > > > > > > >>>>>>>>>>> request.ClientInstanceId = UUID-R > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> the broker returns > > > > > > > > > > > > > > > > > >>>>>>>>>>> response.ClientInstanceId = 0. > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> JR21. It will be useful to > > > > > > > > > > > > > > > > > >>>>>>>>>>> document > > > > > what > > > > > > > the > > > > > > > > > new > > > > > > > > > > > > client > > > > > > > > > > > > > > does > > > > > > > > > > > > > > > > > >>>>> with > > > > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > > > > >>>>>>>>>>> returned > > > > > > > > > > > > > > > > > >>>>>>>>>>> response.ClientInstanceId. > > > > > Note > > > > > > > that > > > > > > > > > return > > > > > > > > > > > > > > value > > > > > > > > > > > > > > > > may > > > > > > > > > > > > > > > > > >>>>> or > > > > > > > > > > > > > > > > > >>>>>>> may not > > > > > > > > > > > > > > > > > >>>>>>>>>>> be 0. > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> JR22. It's probably clearer if we > > > could > > > > > > > > > populate the > > > > > > > > > > > > > > table > > > > > > > > > > > > > > > > > >>>>> with 4 > > > > > > > > > > > > > > > > > >>>>>>>>>>> combinations: old/new clients with > > > > > old/new > > > > > > > > > brokers. > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> Jun > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> On Fri, May 8, 2026 at 2:49 AM > > > Andrew > > > > > > > > > Schofield < > > > > > > > > > > > > > > > > > >>>>>>> [email protected]> > > > > > > > > > > > > > > > > > >>>>>>>>>>> wrote: > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>> Hi Jun and Chia-Ping, > > > > > > > > > > > > > > > > > >>>>>>>>>>>> I've overhauled part of the KIP > > > to do > > > > > with > > > > > > > > > > > > alignment of > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >>>>>>> request > > > > > > > > > > > > > > > > > >>>>>>>>>> header > > > > > > > > > > > > > > > > > >>>>>>>>>>>> client instance ID, client > > > telemetry > > > > > > > client > > > > > > > > > > > > instance ID > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > >>>>> group > > > > > > > > > > > > > > > > > >>>>>>>>>> protocol > > > > > > > > > > > > > > > > > >>>>>>>>>>>> member IDs. The alignment is by > > > > > > > convention, > > > > > > > > > not > > > > > > > > > > > > mandate > > > > > > > > > > > > > > > > > >>>>> (SHOULD > > > > > > > > > > > > > > > > > >>>>>>> not > > > > > > > > > > > > > > > > > >>>>>>>>>> MUST). > > > > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>> It would be possible to go around > > > the > > > > > > > > > existing RPCs > > > > > > > > > > > > > > such as > > > > > > > > > > > > > > > > > >>>>>>>>>>>> ConsumerGroupHeartbeat and > > > > > > > > > > > > GetTelemetrySubscriptions, > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > > >>>>> remove > > > > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > > > > >>>>>>>>>> fields > > > > > > > > > > > > > > > > > >>>>>>>>>>>> containing the existing > > > identifiers > > > > > which > > > > > > > are > > > > > > > > > > > > intended > > > > > > > > > > > > > > to be > > > > > > > > > > > > > > > > > >>>>>>> aligned. > > > > > > > > > > > > > > > > > >>>>>>>>>> Doing > > > > > > > > > > > > > > > > > >>>>>>>>>>>> so would be a bad idea though, > > > > > because we > > > > > > > > > would then > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > RPC > > > > > > > > > > > > > > > > > >>>>>>> versions > > > > > > > > > > > > > > > > > >>>>>>>>>>>> which essentially depend upon the > > > > > > > presence of > > > > > > > > > a > > > > > > > > > > > > tagged > > > > > > > > > > > > > > field > > > > > > > > > > > > > > > > > >>>>> in > > > > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > > > > >>>>>>>>>> request > > > > > > > > > > > > > > > > > >>>>>>>>>>>> header. This is a > > > > > protocol-compatibility > > > > > > > > > nightmare. > > > > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>> I have removed the new versions > > > > > > > > > > > > > > > > > >>>>>>>>>>>> of > > > > > > > > > > > > > > GetTelemetrySubscriptions > > > > > > > > > > > > > > > > > >>>>> and > > > > > > > > > > > > > > > > > >>>>>>>>>>>> PushTelemetry. I have also > > > explained > > > > > the > > > > > > > > > behavior of > > > > > > > > > > > > > > > > > >>>>>>>>>>>> GetTelemetrySubscriptions in the > > > > > presence > > > > > > > and > > > > > > > > > > > > absence > > > > > > > > > > > > > > of a > > > > > > > > > > > > > > > > > >>>>> client > > > > > > > > > > > > > > > > > >>>>>>>>>> instance > > > > > > > > > > > > > > > > > >>>>>>>>>>>> ID in the request header. > > > > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>> Let me know what you think. > > > > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>> Thanks, > > > > > > > > > > > > > > > > > >>>>>>>>>>>> Andrew > > > > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>> On 2026/05/07 15:09:31 Andrew > > > > > Schofield > > > > > > > wrote: > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> Hi Jun and Chia-Ping, > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> I've been thinking and > > > discussing the > > > > > > > > > changes to > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >>>>> KIP-714 > > > > > > > > > > > > > > > > > >>>>>>> RPCs. > > > > > > > > > > > > > > > > > >>>>>>>>>> There > > > > > > > > > > > > > > > > > >>>>>>>>>>>> are too many combinations for my > > > > > liking > > > > > > > at the > > > > > > > > > > > > moment. I > > > > > > > > > > > > > > > > > >>>>> want to > > > > > > > > > > > > > > > > > >>>>>>> take > > > > > > > > > > > > > > > > > >>>>>>>>>>>> another pass at this area and > > > > > > > > > > > > > > > > > >>>>>>>>>>>> will > > > > > make an > > > > > > > > > update > > > > > > > > > > > > in a > > > > > > > > > > > > > > few > > > > > > > > > > > > > > > > > >>>>> days. > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> I intend to start a new vote > > > once we > > > > > have > > > > > > > > > consensus > > > > > > > > > > > > > > because > > > > > > > > > > > > > > > > > >>>>>>> the spec > > > > > > > > > > > > > > > > > >>>>>>>>>> has > > > > > > > > > > > > > > > > > >>>>>>>>>>>> changed somewhat since the > > > earliest > > > > > votes. > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks, > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> Andrew > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> On 2026/05/06 17:28:27 Chia-Ping > > > Tsai > > > > > > > wrote: > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>> hi Andrew > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>> chia_0: If the consensus is to > > > > > remove > > > > > > > the > > > > > > > > > > > > "duplicate" > > > > > > > > > > > > > > > > > >>>>> field > > > > > > > > > > > > > > > > > >>>>>>> from > > > > > > > > > > > > > > > > > >>>>>>>>>> the > > > > > > > > > > > > > > > > > >>>>>>>>>>>> RPC payloads, the tagged field in > > > the > > > > > > > header > > > > > > > > > will > > > > > > > > > > > > > > > > essentially > > > > > > > > > > > > > > > > > >>>>>>> become a > > > > > > > > > > > > > > > > > >>>>>>>>>>>> required field. This means the > > > broker > > > > > > > needs to > > > > > > > > > > > > handle > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >>>>> edge > > > > > > > > > > > > > > > > > >>>>>>> case > > > > > > > > > > > > > > > > > >>>>>>>>>> where > > > > > > > > > > > > > > > > > >>>>>>>>>>>> both the header and the request > > > body > > > > > have > > > > > > > no > > > > > > > > > > > > > > > > > >>>>> ClientInstanceId, > > > > > > > > > > > > > > > > > >>>>>>> right? > > > > > > > > > > > > > > > > > >>>>>>>>>> If > > > > > > > > > > > > > > > > > >>>>>>>>>>>> so, would you mind clarifying the > > > > > expected > > > > > > > > > broker > > > > > > > > > > > > > > behavior > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > >>>>>>> the KIP? > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>> Best, > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>> Chia-Ping > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>> On 2026/04/03 16:17:37 Andrew > > > > > Schofield > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>>> Hi, > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>>> I would like to start the > > > > > discussion on > > > > > > > > > KIP-1313. > > > > > > > > > > > > > > This > > > > > > > > > > > > > > > > > >>>>>>> adds a > > > > > > > > > > > > > > > > > >>>>>>>>>> unique > > > > > > > > > > > > > > > > > >>>>>>>>>>>> client instance ID to the request > > > > > header > > > > > > > of > > > > > > > > > all > > > > > > > > > > > > Kafka > > > > > > > > > > > > > > > > > >>>>> protocol > > > > > > > > > > > > > > > > > >>>>>>>>>> requests to > > > > > > > > > > > > > > > > > >>>>>>>>>>>> give a unique identifier which > > > can be > > > > > > > used to > > > > > > > > > > > > correlate > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > >>>>>>> requests > > > > > > > > > > > > > > > > > >>>>>>>>>> from > > > > > > > > > > > > > > > > > >>>>>>>>>>>> each client for the purposes of > > > > > problem > > > > > > > > > > > > determination. > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/KAFKA/KIP-1313*3A*Client*instance*ID*in*all*request*headers__;JSsrKysrKys!!Ayb5sqE7!uqWf0-b_X82WmpmCYImD2W2rht_s_q5vHcqB9ToMV4IaeQbZF42eMJyS5XC5b5qE_qJJUj3KTCXcqEvYbwYS$ > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>>> Thanks, > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>>> Andrew > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
