For the existing releases, yes (with KIP-447 we are already going to do that anyways), for future release maybe not --- hopefully we only do such metrics refactoring once.
Guozhang On Fri, Aug 2, 2019 at 3:23 PM Matthias J. Sax <matth...@confluent.io> wrote: > Would this imply that we need to update the config in each release to > add a new accepted value? > > > -Matthias > > On 8/2/19 1:07 PM, Guozhang Wang wrote: > > Hello Matthias, > > > > That's a good question. Thinking about a bit more, I'd like to propose > that > > we just list all the version numbers since 0.10 to 2.4 as accepted > values, > > and let Streams to decide if old / new set of metrics can be used > > internally (implementation wise we can reuse the const values for > > `upgrade.from` as well). > > > > And then in the future when we remove the metrics, we can just remove the > > corresponding version values from the accepted list of this config. > > > > > > Guozhang > > > > On Tue, Jul 23, 2019 at 11:55 AM Matthias J. Sax <matth...@confluent.io> > > wrote: > > > >> Thanks for the KIP Guozhang. > >> > >> I just re-read the wiki page and the DISCUSS thread. Overall LGTM. > >> > >> The only nit is the naming of the new config values. With AK 2.3 being > >> released the versions numbers needs to be updated. > >> > >> Additionally, I actually think that "2.2-" and "2.3" are not the best > >> names: the `-` suffix is very subtle IMHO and actually looks more like a > >> typo, and it might be better to be more elaborate. Maybe something like > >> "up-to-2.2" ? > >> > >> For "2.3", this config value would be weird for future releases (ie, > >> 2.4, 2.5, 2.6). Hence, we might want to rename it to "newest" / > >> "current" or something like this? > >> > >> Another alternative may be to rename it to "since-2.3" (or similar) -- > >> however, this may require to rename the config if we change metrics in a > >> future release (hence, it's not my preferred option). > >> > >> Thoughts? > >> > >> > >> -Matthias > >> > >> On 7/22/19 6:33 PM, Guozhang Wang wrote: > >>> Thanks everyone for your inputs, I've updated the wiki page > accordingly. > >>> > >>> @Bruno: please let me know if you have any further thoughts per my > >> replies > >>> above. > >>> > >>> > >>> Guozhang > >>> > >>> > >>> On Mon, Jul 22, 2019 at 6:30 PM Guozhang Wang <wangg...@gmail.com> > >> wrote: > >>> > >>>> Thanks Boyang, > >>>> > >>>> I've thought about exposing time via metrics in Streams. The tricky > part > >>>> though is which layer of time we should expose: right now we have > >>>> task-level and partition-level stream time (what you suggested), and > >> also > >>>> some processor internally maintain their own observed time. Today we > are > >>>> still trying to get a clear and simple way of exposing a single time > >>>> concept for users to reason about their application's progress. So > >> before > >>>> we come up with a good solution I'd postpone adding it in a future > KIP. > >>>> > >>>> > >>>> Guozhang > >>>> > >>>> > >>>> On Thu, Jul 18, 2019 at 1:21 PM Boyang Chen < > reluctanthero...@gmail.com > >>> > >>>> wrote: > >>>> > >>>>> I mean the partition time. > >>>>> > >>>>> On Thu, Jul 18, 2019 at 11:29 AM Guozhang Wang <wangg...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Hi Boyang, > >>>>>> > >>>>>> What do you mean by `per partition latency`? > >>>>>> > >>>>>> Guozhang > >>>>>> > >>>>>> On Mon, Jul 1, 2019 at 9:28 AM Boyang Chen < > >> reluctanthero...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Hey Guozhang, > >>>>>>> > >>>>>>> do we plan to add per partition latency in this KIP? > >>>>>>> > >>>>>>> On Mon, Jul 1, 2019 at 7:08 AM Bruno Cadonna <br...@confluent.io> > >>>>> wrote: > >>>>>>> > >>>>>>>> Hi Guozhang, > >>>>>>>> > >>>>>>>> Thank you for the KIP. > >>>>>>>> > >>>>>>>> 1) As far as I understand, the StreamsMetrics interface is there > for > >>>>>>>> user-defined processors. Would it make sense to also add a method > to > >>>>>>>> the interface to specify a sensor that records skipped records? > >>>>>>>> > >>>>>>>> 2) What are the semantics of active-task-process and > >>>>>> standby-task-process > >>>>>>>> > >>>>>>>> 3) How do dropped-late-records and expired-window-record-drop > relate > >>>>>>>> to each other? I guess the former is for records that fall outside > >>>>> the > >>>>>>>> grace period and the latter is for records that are processed > after > >>>>>>>> the retention period of the window. Is this correct? > >>>>>>>> > >>>>>>>> 4) Is there an actual difference between skipped and dropped > >>>>> records? > >>>>>>>> If not, shall we unify the terminology? > >>>>>>>> > >>>>>>>> 5) What happens with removed metrics when the user sets the > version > >>>>> of > >>>>>>>> "built.in.metrics.version" to 2.2- > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Bruno > >>>>>>>> > >>>>>>>> On Thu, Jun 27, 2019 at 6:11 PM Guozhang Wang <wangg...@gmail.com > > > >>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> Hello folks, > >>>>>>>>> > >>>>>>>>> As 2.3 is released now, I'd like to bump up this KIP discussion > >>>>> again > >>>>>>> for > >>>>>>>>> your reviews. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Guozhang > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, May 23, 2019 at 4:44 PM Guozhang Wang < > wangg...@gmail.com > >>>>>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hello Patrik, > >>>>>>>>>> > >>>>>>>>>> Since we are rolling out 2.3 and everyone is busy with the > >>>>> release > >>>>>>> now > >>>>>>>>>> this KIP does not have much discussion involved yet and will > >>>>> slip > >>>>>>> into > >>>>>>>> the > >>>>>>>>>> next release cadence. > >>>>>>>>>> > >>>>>>>>>> This KIP itself contains several parts itself: 1. refactoring > >>>>> the > >>>>>>>> existing > >>>>>>>>>> metrics hierarchy to cleanup some redundancy and also get more > >>>>>>>> clarity; 2. > >>>>>>>>>> add instance-level metrics like rebalance and state metrics, as > >>>>>> well > >>>>>>> as > >>>>>>>>>> other static metrics. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Guozhang > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, May 23, 2019 at 5:34 AM Patrik Kleindl < > >>>>> pklei...@gmail.com > >>>>>>> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Guozhang > >>>>>>>>>>> Thanks for the KIP, this looks very helpful. > >>>>>>>>>>> Could you please provide more detail on the metrics planned for > >>>>>> the > >>>>>>>> state? > >>>>>>>>>>> We were just considering how to implement this ourselves > >>>>> because > >>>>>> we > >>>>>>>> need > >>>>>>>>>>> to > >>>>>>>>>>> track the history of stage changes. > >>>>>>>>>>> The idea was to have an accumulated "seconds in state x" metric > >>>>>> for > >>>>>>>> every > >>>>>>>>>>> state. > >>>>>>>>>>> The new rebalance metric might solve part of our use case, but > >>>>> it > >>>>>> is > >>>>>>>>>>> interesting what you have planned for the state metric. > >>>>>>>>>>> best regards > >>>>>>>>>>> Patrik > >>>>>>>>>>> > >>>>>>>>>>> On Fri, 29 Mar 2019 at 18:56, Guozhang Wang < > >>>>> wangg...@gmail.com> > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hello folks, > >>>>>>>>>>>> > >>>>>>>>>>>> I'd like to propose the following KIP to improve the Kafka > >>>>>> Streams > >>>>>>>>>>> metrics > >>>>>>>>>>>> mechanism to users. This includes 1) a minor change in the > >>>>>> public > >>>>>>>>>>>> StreamsMetrics API, and 2) a major cleanup on the Streams' > >>>>> own > >>>>>>>> built-in > >>>>>>>>>>>> metrics hierarchy. > >>>>>>>>>>>> > >>>>>>>>>>>> Details can be found here: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-444%3A+Augment+metrics+for+Kafka+Streams > >>>>>>>>>>>> > >>>>>>>>>>>> I'd love to hear your thoughts and feedbacks. Thanks! > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> -- Guozhang > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> -- Guozhang > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> -- Guozhang > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> -- Guozhang > >>>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> -- Guozhang > >>>> > >>> > >>> > >> > >> > > > > -- -- Guozhang
