Thanks Boyang, I've thought about exposing time via metrics in Streams. The tricky part though is which layer of time we should expose: right now we have task-level and partition-level stream time (what you suggested), and also some processor internally maintain their own observed time. Today we are still trying to get a clear and simple way of exposing a single time concept for users to reason about their application's progress. So before we come up with a good solution I'd postpone adding it in a future KIP.
Guozhang On Thu, Jul 18, 2019 at 1:21 PM Boyang Chen <reluctanthero...@gmail.com> wrote: > I mean the partition time. > > On Thu, Jul 18, 2019 at 11:29 AM Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Boyang, > > > > What do you mean by `per partition latency`? > > > > Guozhang > > > > On Mon, Jul 1, 2019 at 9:28 AM Boyang Chen <reluctanthero...@gmail.com> > > wrote: > > > > > Hey Guozhang, > > > > > > do we plan to add per partition latency in this KIP? > > > > > > On Mon, Jul 1, 2019 at 7:08 AM Bruno Cadonna <br...@confluent.io> > wrote: > > > > > > > Hi Guozhang, > > > > > > > > Thank you for the KIP. > > > > > > > > 1) As far as I understand, the StreamsMetrics interface is there for > > > > user-defined processors. Would it make sense to also add a method to > > > > the interface to specify a sensor that records skipped records? > > > > > > > > 2) What are the semantics of active-task-process and > > standby-task-process > > > > > > > > 3) How do dropped-late-records and expired-window-record-drop relate > > > > to each other? I guess the former is for records that fall outside > the > > > > grace period and the latter is for records that are processed after > > > > the retention period of the window. Is this correct? > > > > > > > > 4) Is there an actual difference between skipped and dropped records? > > > > If not, shall we unify the terminology? > > > > > > > > 5) What happens with removed metrics when the user sets the version > of > > > > "built.in.metrics.version" to 2.2- > > > > > > > > Best, > > > > Bruno > > > > > > > > On Thu, Jun 27, 2019 at 6:11 PM Guozhang Wang <wangg...@gmail.com> > > > wrote: > > > > > > > > > > Hello folks, > > > > > > > > > > As 2.3 is released now, I'd like to bump up this KIP discussion > again > > > for > > > > > your reviews. > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > On Thu, May 23, 2019 at 4:44 PM Guozhang Wang <wangg...@gmail.com> > > > > wrote: > > > > > > > > > > > Hello Patrik, > > > > > > > > > > > > Since we are rolling out 2.3 and everyone is busy with the > release > > > now > > > > > > this KIP does not have much discussion involved yet and will slip > > > into > > > > the > > > > > > next release cadence. > > > > > > > > > > > > This KIP itself contains several parts itself: 1. refactoring the > > > > existing > > > > > > metrics hierarchy to cleanup some redundancy and also get more > > > > clarity; 2. > > > > > > add instance-level metrics like rebalance and state metrics, as > > well > > > as > > > > > > other static metrics. > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > > > > > On Thu, May 23, 2019 at 5:34 AM Patrik Kleindl < > pklei...@gmail.com > > > > > > > wrote: > > > > > > > > > > > >> Hi Guozhang > > > > > >> Thanks for the KIP, this looks very helpful. > > > > > >> Could you please provide more detail on the metrics planned for > > the > > > > state? > > > > > >> We were just considering how to implement this ourselves because > > we > > > > need > > > > > >> to > > > > > >> track the history of stage changes. > > > > > >> The idea was to have an accumulated "seconds in state x" metric > > for > > > > every > > > > > >> state. > > > > > >> The new rebalance metric might solve part of our use case, but > it > > is > > > > > >> interesting what you have planned for the state metric. > > > > > >> best regards > > > > > >> Patrik > > > > > >> > > > > > >> On Fri, 29 Mar 2019 at 18:56, Guozhang Wang <wangg...@gmail.com > > > > > > wrote: > > > > > >> > > > > > >> > Hello folks, > > > > > >> > > > > > > >> > I'd like to propose the following KIP to improve the Kafka > > Streams > > > > > >> metrics > > > > > >> > mechanism to users. This includes 1) a minor change in the > > public > > > > > >> > StreamsMetrics API, and 2) a major cleanup on the Streams' own > > > > built-in > > > > > >> > metrics hierarchy. > > > > > >> > > > > > > >> > Details can be found here: > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams > > > > > >> > > > > > > >> > I'd love to hear your thoughts and feedbacks. Thanks! > > > > > >> > > > > > > >> > -- > > > > > >> > -- Guozhang > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
