Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Sorry, one more thing occurred to me. Can the names of the time-based metrics include their units? That makes it much easier for people consuming the metrics to interpret them correctly. For example, offset-commit-max-time would become offset-commit-max-time-ms or

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

FWIW, I agree that time metrics have been very useful in the past. The reasoning around perf overhead seems reasonable as well. Can we agree on a subset of time metrics that we feel would be super useful for debugging? On Tue, Sep 12, 2017 at 6:08 PM, Roger Hoover wrote:

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Thanks, Ewen. I agree with you on the overhead of measuring time for SMTs and converters. I'd still argue for keeping other metrics like flush time b/c even small batches should still be small overhead compared to writing to a sink. On Tue, Sep 12, 2017 at 3:06 PM, Ewen Cheslack-Postava

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Regarding the existing rebalance metrics under "kafka.connect:type=connect-coordinator-metrics", I think we should just plan on reusing them rather than duplicating them. On Tue, Sep 12, 2017 at 5:06 PM, Ewen Cheslack-Postava wrote: > Requests are generally substantial

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Requests are generally substantial batches of data, you are not guaranteed that for the processing batches both because source connectors can hand you batches of whatever size they want and consumer's max.poll.records can be overridden. Both SMTs and converters are a concern because they can both

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Ewen, I don't know the details of the perf concern. How is it that the Kafka broker can keep latency stats per request without suffering too much performance? Maybe SMTs are the only concern b/c they are per-message. If so, let's remove those and keep timing info for everything else like

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

On Tue, Sep 12, 2017 at 10:55 AM, Gwen Shapira wrote: > Ewen, you gave a nice talk at Kafka Summit where you warned about the > danger of SMTs that slow down the data pipe. If we don't provide the time > metrics, how will users know when their SMTs are causing performance >

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Okay, I think I've incorporated all feedback except for Gwen and Roger than would like to have timing metrics. Given the deadline and Ewen's concern about degraded performance, I think it's prudent to leave those out of this KIP and proceed as is. On Tue, Sep 12, 2017 at 12:48 PM, Randall Hauch

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Ewen, you gave a nice talk at Kafka Summit where you warned about the danger of SMTs that slow down the data pipe. If we don't provide the time metrics, how will users know when their SMTs are causing performance issues? Gwen On Mon, Sep 11, 2017 at 7:50 PM Ewen Cheslack-Postava

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

On Tue, Sep 12, 2017 at 10:36 AM, Roger Hoover wrote: > Randall/Ewen, > > I think the timing info is still useful even if it's measured since the > last rebalance. How else do you know where time is being spent? > I think Ewen's concern (correct me if I'm wrong) is that

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Hi, James. I was mistaken about how the Kafka metrics are converted to MBeans and attributes. The MBean is constructed from the group and tags, and the metrics show up as attributes on the MBean. I'll update the KIP to reflect this. On Tue, Sep 12, 2017 at 1:43 AM, James Cheng

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Randall/Ewen, I think the timing info is still useful even if it's measured since the last rebalance. How else do you know where time is being spent? The use case for seeing the batch size is that you generally have two knobs to configure - max batch size and max wait time. The batch size

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Thanks for the KIP, Randall. The KIP has one MBean per metric name. Can I suggest an alternate grouping? kafka.connect:type=connector-metrics,connector=([-.\w]+) connector-type connector-class connector-version status

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

re: questions about additional metrics, I think we'll undoubtedly find more that people want in practice, but as I mentioned earlier I think it's better to add the ones we know we need and then fill out the rest as we figure it out. So, e.g., batch size metrics sound like they could be useful, but

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

On Mon, Sep 11, 2017 at 4:50 PM, Randall Hauch wrote: > Thanks, Ewen. Comments inline below. > > On Mon, Sep 11, 2017 at 5:46 PM, Ewen Cheslack-Postava > wrote: > > > Randall, > > > > A couple of questions: > > > > * Some metrics don't seem to have unique

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Based on Roger and Ewen's feedback, I removed the aggregate metrics as they would be difficult to make use of without extra work. This simplified things a great deal, and I took the opportunity to reorganize the groups of metrics. Also, based upon Ewen's concerns regarding measuring

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Thanks, Ewen. Comments inline below. On Mon, Sep 11, 2017 at 5:46 PM, Ewen Cheslack-Postava wrote: > Randall, > > A couple of questions: > > * Some metrics don't seem to have unique names? e.g. > source-record-produce-rate and source-record-produce-total seem like they > are

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Randall, A couple of questions: * Some metrics don't seem to have unique names? e.g. source-record-produce-rate and source-record-produce-total seem like they are duplicated. Looks like maybe just an oversight that the second ones should be changed from "produce" to "write". * I think there's a

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Thanks for reviewing. Responses inline below. On Mon, Sep 11, 2017 at 11:22 AM, Roger Hoover wrote: > Randall, > > Thank you for the KIP. This should improve visibility greatly. I had a > few questions/ideas for more metrics. > > >1. What's the relationship between

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Randall, Thank you for the KIP. This should improve visibility greatly. I had a few questions/ideas for more metrics. 1. What's the relationship between the worker state and the connector status? Does the 'paused' status at the Connector level include the time that worker is

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Thanks, Gwen. That's a great idea, so I've changed the KIP to add those metrics. I've also made a few other changes: 1. The context of all metrics is limited to the activity within the worker. This wasn't clear before, so I changed the motivation and metric descriptions to explicitly

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Thanks for the KIP, Randall. Those are badly needed! Can we have two metrics with record rate per task? One before SMT and one after? We can have cases where we read 5000 rows from JDBC but write 5 to Kafka, or read 5000 records from Kafka and write 5 due to filtering. I think its important to

[DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Hi everyone. I've created a new KIP to add metrics to the Kafka Connect framework: https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework The KIP approval deadline is looming, so if you're interested in Kafka Connect metrics please review and provide