Thanks for the KIP Andrew. I think it will be very useful if we can
identify frameworks, especially our own ones (Connect and Streams).
About Aditya's point: I am wondering where to draw the line. Many
examples seems to be metadata that belongs into he Kafka record
`Headers` at the application level, rather than the lower level request
headers?
There if of course no strict logical dividing line between both. And
yes, the broker does not access application level Kafka record
`Headers`. But if it would be useful to let the broker tap into
application level record `Headers` we should tackle it independently?
Personally, I don't think it would be the right design to push too many
thing into the lower level request headers. So I am in favor of only
added the two new propose `ClientFrameworkName` and
`ClientFrameworkVersion` fields.
-Matthias
On 7/31/26 7:56 AM, Federico Valeri wrote:
Changes look good. Thanks.
On Fri, Jul 31, 2026 at 11:17 AM Andrew Schofield <[email protected]> wrote:
Hi Fede,
Thanks for your response.
FV1: They are set using the regular config properties. Yes, it is possible for
an end user to set arbitrary values, but then that would also be true of a
builder or internal constructor. I've added a bit more information in the KIP
and beefed up the config descriptions to discourage application use.
FV2: I've never encountered such nightmares myself. The framework which sets
the config last would win. Maybe this is a motivation for having a programmatic
way of setting the information. We could support concatenation of framework
information, but that's just pandering to these people. Let me know what you
think.
I've also updated the KIP with a maximum length for these pieces of information
since all identifiers should have defined bounds.
Thanks,
Andrew
On 2026/07/31 08:50:11 Federico Valeri wrote:
Hi Andrew, the motivation looks good. A couple of questions:
FV1: The KIP says frameworks should set them, but it does not specify
the mechanism. If these are ordinary user-facing configs, nothing
prevents an end user from setting arbitrary values, which defeats the
purpose. Should the framework set them programmatically (builder or
internal constructor) or is user override intentional?
FV2: What if we have multiple layers of frameworks? Let's say a custom
framework on top of SpringBoot. I've seen similar nightmares in the
past.
Thanks
Fede
On Tue, Jul 28, 2026 at 8:27 AM Aditya Kousik <[email protected]> wrote:
Hi Andrew,
We’re definitely agreed on the increased use of frameworks over than the client
directly. I’ve cited the different patterns Spring, Micronaut, smallrye and
company-internal frameworks have APIs built on top of the Kafka client, in a
couple of KIPs already.
The project has enough traction that I think of it as its own network client
with distributed log semantics for which frameworks are written, much like gRPC
over netty. People want to just write the business logic and leave the plumbing
and threading to the frameworks.
All of this to say, I’m fine to ship the client framework as the new
identifying parameter for frameworks to set. It will be mighty useful.
My addendum, rather than a pushback is that: guilty as charged, I’m driven by
the otel/DD telemetry/observability of using Apache Kafka in applications. The
client.framework.version config for instance can be used to detect regressions
and isolate root causes. But I feel it is only one of many such facets. You
mentioned that you would use client.id and clientInstanceId to identify a
client but that these do not help with aggregate/fleet-wide issues. As an
example, if I tag an app’s AZ it can help me write alerts on spike in latency
in us-west-2. Or, detect stuck partitions across multiple
client.id/application.names none of whom share the same client.framework.name.
Other tags that come to mind: application.name, az, team, env, rack. All fields
users usually hijack client.id for.
An otel JavaAgent can capture the client metadata registered and attach it as
tags with each resource span. Users who use frameworks but rely on datadog/otel
will get visibility into the client metadata for free.
The KIP as I read it, serves as a foundation for future use. So I don’t want to
shoehorn a new behaviour if it explodes the scope too much.
Best regards,
Aditya
On Jul 27, 2026, at 13:52, Andrew Schofield <[email protected]> wrote:
Hi Aditya,
Thanks for your response.
AK1: I chose framework as the blessed abstraction because my focus was problem
determination for client applications. We often find that users with client
problems have not coded directly to the Kafka client interface because they are
using a framework. As a result, their knowledge of the application code is one
level removed from the Kafka client. Frameworks can override configuration
defaults, introduce different retry behaviour and so on. Lots of companies have
their own internal frameworks, so this KIP can be used by them too. I'm trying
to make it easier to work out when a user is making use of a framework and
knowing what it is.
Sometimes, particularly for non-Java clients, people have overridden the
ClientSoftwareName/Version themselves, which makes those concepts much less
useful than they should be. By providing ClientFrameworkName/Version, there is
no longer any need to do so. That's another motivation here, even though KIPs
don't concern themselves with non-Java clients as such.
We could go for a more flexible key-value approach, but a simple name and
version is sufficient for what I had in mind. Feel free to push back with
additional justification and examples.
AK2: Done. o.a.k.clients.CommonClientConfigs.
AK3: To identify a particular client, I would use client ID and client-instance
ID. I think these are generally more useful concepts than the framework name
and version which are extra information for the person trying to figure out why
a client is not behaving as expected.
AK4: I'm sure you have more experience of OTel/DataDog collectors. You may well
be correct that they would be helpful for the collectors.
Thanks,
Andrew
On 2026/07/26 07:48:20 Aditya Kousik wrote:
Hello Andrew,
I’m reminded of the client.id discussion we had back in KIP-1313 re: client
instance id. After that discussion, I have a WIP KIP that sets a foundation for
shipping client metadata tags to be sent for telemetry. I was hoping we could
discuss if part of that approach could fit this KIP.
AK1. I had a note on the motivation of selecting a “framework” as a blessed
abstraction. The KIP mentions it’s for the client metadata and easier problem
diagnosis. This is akin to an “application id” that non-framework clients
usually tag with.
KIP-606 took an approach like metrics.context.<key>=<val>. If we allow
client.metadata.<key>=<val>, then a framework/application name and version can sit in such
a metadata context and be sent with ApiVersion RPC. Of course, this is an open box approach rather than
just the framework name/version (just two fields) we’re adding to the protocol. But I’m curious about
the tier of importance of framework alone.
AK2. Can you clarify if this config goes into CommonClientConfigs.java,
referenced across producer/consumer/share etc? I know some share props have the
“share.” prefix going.
AK3. In another thread you mentioned that the broker may add it to the request
context. In client logs, clientId is a very useful string to identify WARN logs
when things go sideways like broker disconnected, rebalance in progress etc.
Can you cherry pick and highlight some useful log places that these strings can
go? I suppose adding clientId/framework to the MDC context might be too
voluminous.
AK4. I foresee collectors like Datadog/OTel might find these tags useful in
each span exported.
Looking forward to your thoughts on this.
Regards,
Aditya
On Jul 17, 2026, at 11:53, Andrew Schofield <[email protected]>
wrote:
Hi,
I'd like to open discussion on KIP-1368: Client framework name and version.
Applications often use application frameworks such as Spring to connect to
Kafka. To assist with problem diagnosis, this KIP introduces a way to provide
the framework name and version as part of the metadata the client sends to the
broker when it connects.
Here's the KIP: https://cwiki.apache.org/confluence/x/J4Q_Gg
Thanks,
Andrew
