Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-17 Thread Sean Glover
Hi Habib, With regards to your earlier question about timezones, I've updated the KIP to remove the LatencyTime abstraction since it is no longer relevant. I added a note about epoch time as well. Thanks, Sean On Wed, Jan 15, 2020 at 8:28 AM Habib Nahas wrote: > Hi Sean, > > Thats great,

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-17 Thread Sean Glover
Hi Gokul, Thank you for your detailed review. I've summarized the updates I've made to the KIP inline below. Please review the updated KIP when you have time. On Fri, Dec 20, 2019 at 6:56 AM Gokul Ramanan Subramanian < gokul24...@gmail.com> wrote: > Hi Sean. > > Thanks for writing this KIP.

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-15 Thread Habib Nahas
Hi Sean, Thats great, look forward to it. Thanks, Habib On Tue, Jan 14, 2020, at 2:55 PM, Sean Glover wrote: > Hi Habib, > > Thank you for the reminder. I'll update the KIP this week and address the > feedback from you and Gokul. > > Regards, > Sean > > On Tue, Jan 14, 2020 at 9:06 AM Habib

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-14 Thread Sean Glover
Hi Habib, Thank you for the reminder. I'll update the KIP this week and address the feedback from you and Gokul. Regards, Sean On Tue, Jan 14, 2020 at 9:06 AM Habib Nahas wrote: > Any chance of an update on the KIP? We are interested in seeing this move > forward. > > Thanks, > Habib > Sr

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2020-01-14 Thread Habib Nahas
Any chance of an update on the KIP? We are interested in seeing this move forward. Thanks, Habib Sr SDE, AWS On Wed, Dec 18, 2019, at 3:27 PM, Habib Nahas wrote: > Thanks Sean. Look forward to the updated KIP. > > Regards, > Habib > > On Fri, Dec 13, 2019, at 6:22 AM, Sean Glover wrote: > >

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-20 Thread Gokul Ramanan Subramanian
Hi Sean. Thanks for writing this KIP. Sounds like a great addition. Few comments. 1. Currently, I see that you have proposed partition-level records-latency metrics and a global records-latency-max metric across all partitions for a given consumer group. Some Kafka users may organize their

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-18 Thread Habib Nahas
Thanks Sean. Look forward to the updated KIP. Regards, Habib On Fri, Dec 13, 2019, at 6:22 AM, Sean Glover wrote: > Hi, > > After my last reply I had a nagging feeling something wasn't right, and I > remembered that epoch time is UTC. This makes the discussion about > timezone irrelevant, since

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-12 Thread Sean Glover
Hi, After my last reply I had a nagging feeling something wasn't right, and I remembered that epoch time is UTC. This makes the discussion about timezone irrelevant, since we're always using UTC. This makes the need for the LatencyTime interface that I proposed in the design irrelevant as well,

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-12 Thread Sean Glover
Hi Habib, Thanks for question! If the consumer is in a different timezone than the timezone used to produce messages to a partition then you can use an implementation of LatencyTime to return the current time of that timezone. The current design assumes that messages produced to a partition must

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-12 Thread Habib Nahas
Hi Sean, Thanks for the KIP. As I understand it users are free to set their own timestamp on ProducerRecord. What is the recommendation for the proposed metric in a scenario where the user sets this timestamp in timezone A and consumes the record in timezone B. Its not clear to me if a

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-12-11 Thread Sean Glover
Hello again, There has been some interest in this KIP recently. I'm bumping the thread to encourage feedback on the design. Regards, Sean On Mon, Jul 15, 2019 at 9:01 AM Sean Glover wrote: > To hopefully spark some discussion I've copied the motivation section from > the KIP: > > Consumer

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-07-15 Thread Sean Glover
To hopefully spark some discussion I've copied the motivation section from the KIP: Consumer lag is a useful metric to monitor how many records are queued to be processed. We can look at individual lag per partition or we may aggregate metrics. For example, we may want to monitor what the

[DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

2019-07-11 Thread Sean Glover
Hi kafka-dev, I've created KIP-489 as a proposal for adding latency metrics to the Kafka Consumer in a similar way as record-lag metrics are implemented. https://cwiki.apache.org/confluence/display/KAFKA/489%3A+Kafka+Consumer+Record+Latency+Metric Regards, Sean -- Principal Engineer,