Re: [DISCUSS] KIP-714: Client metrics and observability

2021-07-22 Thread Feng Min
On Wed, Jul 21, 2021 at 6:17 PM Colin McCabe wrote: > On Tue, Jun 29, 2021, at 07:22, Magnus Edenhill wrote: > > Den tors 17 juni 2021 kl 00:52 skrev Colin McCabe : > > > A few critiques: > > > > > > - As I wrote above, I think this could benefit a lot by being split > into > > > several RPCs. A

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-07-21 Thread Colin McCabe
On Tue, Jun 29, 2021, at 07:22, Magnus Edenhill wrote: > Den tors 17 juni 2021 kl 00:52 skrev Colin McCabe : > > A few critiques: > > > > - As I wrote above, I think this could benefit a lot by being split into > > several RPCs. A registration RPC, a report RPC, and an unregister RPC seem > > like

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-07-09 Thread Xavier Léauté
> > 1. Did you consider using a `default ClientTelemetryReceiver > clientReceiver() { return null; }` method on the existing MetricsReporter > interface, avoiding the need for the ClientTelemetry trait? I did. Part of the motivation was to separate more clearly the MetricsReporter methods which

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-29 Thread Magnus Edenhill
Hey Tom, Den mån 21 juni 2021 kl 21:08 skrev Tom Bentley : > > 1. Did you consider using a `default ClientTelemetryReceiver > clientReceiver() { return null; }` method on the existing MetricsReporter > interface, avoiding the need for the ClientTelemetry trait? > I'll let Xavier answer this one

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-29 Thread Magnus Edenhill
Den fre 18 juni 2021 kl 22:32 skrev Travis Bischel : > H Colin (and Magnus), > > Thanks for the replies! > > I think the biggest concern I have is the cardinality bits. I'm > sympathetic to the aspect of this making it easier for Kafka brokers to > understand *every* aspect of the kafka ecoystem.

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-29 Thread Magnus Edenhill
Den tors 17 juni 2021 kl 00:52 skrev Colin McCabe : > Hi Magnus, > > Thanks for the KIP. This is certainly something I've been wishing for for > a while. > > Maybe we should emphasize more that the metrics that are being gathered > here are Kafka metrics, not general application business logic

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-29 Thread Magnus Edenhill
Thanks for your feedback, Colin, see response below. Den tors 17 juni 2021 kl 00:28 skrev Colin McCabe : > On Sun, Jun 13, 2021, at 21:51, Travis Bischel wrote: > ... > > Another downside is that by dictating the important metrics, this KIP > either > > has two choices: try to choose what is

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-21 Thread Tom Bentley
Hi Magnus, Thanks for the KIP. 1. Did you consider using a `default ClientTelemetryReceiver clientReceiver() { return null; }` method on the existing MetricsReporter interface, avoiding the need for the ClientTelemetry trait? 2. On the metrics naming and format, I wasn't really clear about

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-18 Thread Travis Bischel
H Colin (and Magnus), Thanks for the replies! I think the biggest concern I have is the cardinality bits. I'm sympathetic to the aspect of this making it easier for Kafka brokers to understand *every* aspect of the kafka ecoystem. I am not sure this will 100% solve the need there, though: if

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-18 Thread Colin McCabe
On Thu, Jun 17, 2021, at 12:13, Ryanne Dolan wrote: > Colin, > > > lack of support for collecting client metrics > > ...but kafka is not a metrics collector. There are lots of things kafka > doesn't support. Should it also collect clients' logs for the same reasons? > What other side channels

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-17 Thread Ryanne Dolan
Colin, > lack of support for collecting client metrics ...but kafka is not a metrics collector. There are lots of things kafka doesn't support. Should it also collect clients' logs for the same reasons? What other side channels should it proxy through brokers? > He mentioned the fact that

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-17 Thread Colin McCabe
Hi Ryan, These are not "arguments for observability in general" but descriptions of specific issues that come up due to Kafka's lack of support for collecting client metrics. He mentioned the fact that configuring client metrics usually involves setting up a separate metrics collection

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-16 Thread Ryanne Dolan
Magnus, I think these are arguments for observability in general, but not why kafka should sit between a client and a metics collector. Ryanne On Wed, Jun 16, 2021, 10:27 AM Magnus Edenhill wrote: > Hi Ryanne, > > this proposal stems from a need to improve troubleshooting Kafka issues. > > As

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-16 Thread Colin McCabe
Hi Magnus, Thanks for the KIP. This is certainly something I've been wishing for for a while. Maybe we should emphasize more that the metrics that are being gathered here are Kafka metrics, not general application business logic metrics. That seems like a point of confusion in some of the

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-16 Thread Colin McCabe
On Sun, Jun 13, 2021, at 21:51, Travis Bischel wrote: > Hi! I have a few thoughts on this KIP. First, I'd like to thank you for > the writeup, > clearly a lot of thought has gone into it and it is very thorough. > However, I'm not > convinced it's the right approach from a fundamental level. >

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-16 Thread Magnus Edenhill
Hi Ryanne, this proposal stems from a need to improve troubleshooting Kafka issues. As it currently stands, when an application team is experiencing Kafka service degradation, or the Kafka operator is seeing misbehaving clients, there are plenty of steps that needs to be taken before any

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-16 Thread Magnus Edenhill
Thanks for your feedback, Travis! I believe there are different audiences and uses for application (business logic) and client (infrastructure) metrics. Kafka clients are part of the infrastructure, not the business logic, and should be monitored as such by the organization, sub-organization, or

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-14 Thread Ryanne Dolan
Magnus, I think such a substantial change requires more motivation than is currently provided. As I read it, the motivation boils down to this: you want your clients to phone-home unless they opt-out. As stated in the KIP, "there are plenty of existing solutions [...] to send metrics [...] to a

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-14 Thread Travis Bischel
Apologies for this duplicate reply, I did not notice the success confirmation on the first submission. On 2021/06/14 04:52:11, Travis Bischel wrote: > Hi! I have a few thoughts on this KIP. First, I'd like to thank you for your > work > and writeup, it's clear that a lot of thought went into

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-14 Thread Travis Bischel
Hi! I have a few thoughts on this KIP. First, I'd like to thank you for the writeup, clearly a lot of thought has gone into it and it is very thorough. However, I'm not convinced it's the right approach from a fundamental level. Fundamentally, this KIP seems like somewhat of a solution to an

Re: [DISCUSS] KIP-714: Client metrics and observability

2021-06-14 Thread Travis Bischel
Hi! I have a few thoughts on this KIP. First, I'd like to thank you for your work and writeup, it's clear that a lot of thought went into this and it's very thorough! However, I'm not convinced it's the right approach from a fundamental level. Fundamentally, this KIP seems like somewhat of a

<    1   2