Hey all, I'd like to make one adjustment to the current proposal and go back to the earlier proposal to report the task-level e2e latency metric at the source as well as the sink. I feel this will give users a more complete picture of the e2e latency if they only turn on INFO level metrics.
I also want to quickly clarify one point, which is that these metrics will still be reported at the end of a subtopology that includes no sink. The latency will just be recorded at whatever the terminal operator happens to be. Please let me know if you have any questions or concerns. Thanks! Sophie On Mon, May 18, 2020 at 9:32 AM Guozhang Wang <wangg...@gmail.com> wrote: > Thanks Sophie, I'm +1 on the updated KIP. > > On Mon, May 18, 2020 at 8:32 AM Sophie Blee-Goldman <sop...@confluent.io> > wrote: > > > @Bruno > > No, this would be after inserting in the cache/state store/etc -- > basically > > once > > all operations have completed such that the record would be present if > > queried after that time. > > > > @Guozhang > > I'm glad you suggested the TRACE level. It seems we've been > > heading in that direction for a while so I'm happy to add it as part of > > this KIP. Especially if it unblocks the question of whether to update the > > system time (for now). > > > > On Mon, May 18, 2020 at 1:09 AM Bruno Cadonna <br...@confluent.io> > wrote: > > > > > Hi Sophie, > > > > > > Just a question for my understanding. When you say " t_A be the time > > > when it is finished being processed by the aggregator node", this > > > means before caching (or putting into the state store if caching is > > > disabled), right? > > > > > > Best, > > > Bruno > > > > > > On Sat, May 16, 2020 at 6:57 PM Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > > > > We do have a big JIRA for making sensor recording to be sufficient. > But > > > it > > > > is a bit hard to try to reduce the num.system clock calls in general > > (but > > > > admittedly, in nowadays OS they are supposed to be very cheap as > well), > > > > I've just recently merged a small PR trying to make less > > > > SystemTime.currentMilliseconds calls whenever possible, but it is > hard > > to > > > > enforce that for all callers. > > > > > > > > Guozhang > > > > > > > > On Sat, May 16, 2020 at 9:46 AM John Roesler <vvcep...@apache.org> > > > wrote: > > > > > > > > > FWIW, I’d be fine with TRACE, if it makes the processor-level > metrics > > > more > > > > > palatable to everyone. > > > > > > > > > > Ultimately, I think this is just kicking the can down the road, and > > we > > > > > should really just refactor our sensors/metrics to be lower > overhead. > > > But > > > > > that’s too much work to consider right before the release so for > now, > > > we > > > > > can kick that can just a little farther. > > > > > > > > > > Thanks for proposing a solution, Guozhang! > > > > > > > > > > -John > > > > > > > > > > On Sat, May 16, 2020, at 11:39, Guozhang Wang wrote: > > > > > > Cool, that's good points -- we should not let our current metrics > > > > > proposal > > > > > > to depend on future roadmaps, since they can still be removed > later > > > if > > > > > ever > > > > > > becoming not valuable after all. > > > > > > > > > > > > On the other hand, what do you think about introducing TRACE > level > > > > > metrics > > > > > > recording and make the the processor-node level metrics on TRACE? > > > > > > > > > > > > Guozhang > > > > > > > > > > > > On Fri, May 15, 2020 at 7:42 PM Sophie Blee-Goldman < > > > sop...@confluent.io > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Yeah, I had proposed to replace the "source + sink" task-level > > > metric > > > > > > > with "sink only" task-level metrics plus the stateful > node-level > > > ones. > > > > > > > > > > > > > > I completely agree that this proposal should and would look > > > different > > > > > > > once we complete those two proposals, ie decouple caching and > > > > > > > consolidate suppression. But as it is right now, caching and > > > > > suppression > > > > > > > cause the actual time-of-processing of records to vary greatly > > > > > throughout > > > > > > > a subtopology such that the delta between a source node and an > > > > > intermediate > > > > > > > one is potentially quite large. > > > > > > > > > > > > > > For this exact reason I would argue that this metric is still > > > valuable > > > > > at > > > > > > > the node-level > > > > > > > even if the "current time" is actually only updated once for a > > > task. > > > > > > > Obviously this > > > > > > > would not convey any information about the actual processing > > > latency, > > > > > but > > > > > > > as you > > > > > > > stated, this is likely to be a small delta on the scale of > > > > > milliseconds. > > > > > > > But the latency > > > > > > > due to a suppression can be on the order of minutes, hours, or > > even > > > > > days! > > > > > > > If > > > > > > > there is more than one suppression or caching in a subtopology, > > the > > > > > source > > > > > > > and sink > > > > > > > e2e latency could be wildly misrepresentative of the e2e > latency > > > of a > > > > > node > > > > > > > in the middle. > > > > > > > > > > > > > > Anyways, I removed this last point from the current KIP > document > > > > > because I > > > > > > > felt > > > > > > > it was ultimately an implementation detail. But the larger > point > > I > > > > > would > > > > > > > argue is that: > > > > > > > if we are truly worried about the performance hit of fetching > the > > > > > system > > > > > > > time at every > > > > > > > node, we should just not fetch the system time at every node > > > instead of > > > > > > > dropping the > > > > > > > metric altogether. It may be slightly off but only on the order > > of > > > ms > > > > > and > > > > > > > not on the order > > > > > > > of hours, and it's still more useful than no metric at all > > > > > > > > > > > > > > > > > > > > > On Fri, May 15, 2020 at 6:40 PM Guozhang Wang < > > wangg...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi Sophie, > > > > > > > > > > > > > > > > I think your motivation makes sense, in fact before KIP-444 > we > > > used > > > > > to > > > > > > > have > > > > > > > > processor-node-level process latency as DEBUG which would > > > > > holistically > > > > > > > > address the motivation but we removed it since it is indeed > > quite > > > > > > > expensive > > > > > > > > (I do not have concrete numbers at hand, but basically you'd > > > have to > > > > > call > > > > > > > > system-wall-clock time a couple of time for each record, each > > > > > processor > > > > > > > > node), and in the long run I feel such scenarios would be > less > > > > > common if > > > > > > > we > > > > > > > > 1) decouple caching from emitting and 2) always put > suppression > > > right > > > > > > > > before sink only, and that's also why I've proposed to remove > > > them in > > > > > > > > KIP-444. > > > > > > > > > > > > > > > > I missed an update in your wiki that right now you've updated > > the > > > > > > > > task-level metric to be recorded at the sink node, and > thought > > > it is > > > > > > > still > > > > > > > > at the source node, but assuming it is still the case, then > > > > > practically > > > > > > > any > > > > > > > > state store's staleness of the task could roughly be measured > > as > > > that > > > > > > > > task-level staleness plus a small delta (usually tens of > > mills), > > > and > > > > > the > > > > > > > > down stream task B's staleness - up stream task A's staleness > > > would > > > > > > > capture > > > > > > > > the suppression effect at the end of A plus the intermediate > > > topic > > > > > > > > producing -> consuming latency, which usually would not > require > > > to be > > > > > > > > distinguished. > > > > > > > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 15, 2020 at 6:09 PM Sophie Blee-Goldman < > > > > > sop...@confluent.io > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hey Guozhang, > > > > > > > > > > > > > > > > > > Thanks for the response. I meant to ask this earlier and > > > forgot, > > > > > but do > > > > > > > > you > > > > > > > > > have any data or benchmarking results on hand for the > > > performance > > > > > hit > > > > > > > we > > > > > > > > > took due to the metrics? Or maybe better yet, on the > > > performance we > > > > > > > > gained > > > > > > > > > back by dropping some metrics? I think it would help us to > > > better > > > > > > > > > understand > > > > > > > > > the tradeoffs with some concrete numbers to compare. > > > > > > > > > > > > > > > > > > I guess I'm inclined to want to include these metrics at > the > > > > > operator > > > > > > > > level > > > > > > > > > on > > > > > > > > > DEBUG since, in my mind, that's exactly what DEBUG is for: > > > > > performance- > > > > > > > > > focused users will presumably run at INFO level and even > > those > > > who > > > > > do > > > > > > > > > use DEBUG may only do so temporarily, when they actually > have > > > > > something > > > > > > > > > to debug. > > > > > > > > > > > > > > > > > > Of course, I'm not running a Streams app myself so all this > > is > > > just > > > > > > > > > conjecture. > > > > > > > > > But it seems reasonable to interpret the two metrics levels > > as > > > > > "minimal > > > > > > > > > useful > > > > > > > > > information" and "all information that provides insight > into > > > the > > > > > > > system" > > > > > > > > > > > > > > > > > > If we can agree there, the question becomes "does this > > provide > > > any > > > > > > > > > additional > > > > > > > > > insight?" I would still argue that it does. To give a > > concrete > > > > > example, > > > > > > > > > let's say > > > > > > > > > the user has a basic > > > > > > > > > > > > > > > > > > SOURCE -> TRANSFORM -> AGGREGATE -> SUPPRESSION -> SINK > > > > > > > > > > > > > > > > > > subtopology, the application performing IQ on the > > transformer's > > > > > state. > > > > > > > > The > > > > > > > > > user > > > > > > > > > wants to know how long it's taking records to be reflected > in > > > the > > > > > > > results > > > > > > > > > of a query. > > > > > > > > > With only task-level metrics, they can only guess based on > > the > > > e2e > > > > > > > > latency > > > > > > > > > of the > > > > > > > > > previous task and of this one. But neither of those is a > good > > > > > > > > > representation of the > > > > > > > > > time it will take a record to be processed by the > > transformer: > > > the > > > > > > > > previous > > > > > > > > > task's > > > > > > > > > latency will not include the time for the record to go > > through > > > the > > > > > > > > > repartitioning, > > > > > > > > > and the current task's latency is measured after a > > > suppression. The > > > > > > > > > suppression > > > > > > > > > itself will introduce significant latency and doesn't > provide > > > an > > > > > > > accurate > > > > > > > > > estimate. > > > > > > > > > Caching would also do what it does best, messing with our > > > > > understanding > > > > > > > > of > > > > > > > > > time :) > > > > > > > > > > > > > > > > > > It also means you can't measure the consumption latency, so > > > there > > > > > would > > > > > > > > be > > > > > > > > > no way to tell if the e2e latency is so high because the > very > > > first > > > > > > > > > subtopology is > > > > > > > > > extremely slow, or the record was severely delayed in > > reaching > > > the > > > > > > > input > > > > > > > > > topic. > > > > > > > > > I guess my earlier proposal to include both source and sink > > > metrics > > > > > > > could > > > > > > > > > address this issue, but it seems to solve the problem more > > > > > holistically > > > > > > > > -- > > > > > > > > > and for > > > > > > > > > more complex subtopologies -- by including metrics at the > > > processor > > > > > > > level > > > > > > > > > > > > > > > > > > On Fri, May 15, 2020 at 5:44 PM Guozhang Wang < > > > wangg...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hey folks, > > > > > > > > > > > > > > > > > > > > I want to make a final fight on processor-node-level > > metrics > > > :) > > > > > Post > > > > > > > > > > KIP-444 we've actually removed a lot of node-level > metrics > > > since > > > > > it > > > > > > > is > > > > > > > > > too > > > > > > > > > > expensive to record at that level and their values are > not > > > proven > > > > > > > worth > > > > > > > > > it. > > > > > > > > > > Again I'd use the "we have to either enable all or none > of > > > DEBUG > > > > > > > > metrics" > > > > > > > > > > card, e.g. if we want to look into per-task > process-latency > > > > > which is > > > > > > > > > DEBUG, > > > > > > > > > > we'd have to record all process-node / state-store / > cache > > > level > > > > > > > > metrics. > > > > > > > > > > On the other hand, within a task we usually do not have a > > > lot of > > > > > > > > stateful > > > > > > > > > > operator nodes and if we want to find if certain stateful > > > nodes > > > > > are > > > > > > > the > > > > > > > > > > bottleneck of process latency, that is usually > discoverable > > > from > > > > > the > > > > > > > > > > state-store level read / write latencies already. So I'd > > > imagine > > > > > that > > > > > > > > > > suppose there's a sub-topology with multiple state stores > > in > > > it, > > > > > if > > > > > > > we > > > > > > > > > > found the task-level process latency is high, we can > still > > > tell > > > > > which > > > > > > > > > state > > > > > > > > > > stores within it is the main bottleneck from the state > > store > > > > > read / > > > > > > > > write > > > > > > > > > > latency (I'm assuming the IO latency is always dominant > > > compared > > > > > with > > > > > > > > > > others). > > > > > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 15, 2020 at 3:22 PM Sophie Blee-Goldman < > > > > > > > > sop...@confluent.io > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > @Matthias > > > > > > > > > > > > And that's what he says about the 95th percentile! > > > Imagine > > > > > what > > > > > > > he > > > > > > > > > > would > > > > > > > > > > > say about the 50th :P > > > > > > > > > > > > > > > > > > > > > > I should have kept watching another 10 minutes. He gets > > > around > > > > > to > > > > > > > > > > covering > > > > > > > > > > > the 50th, and let's > > > > > > > > > > > just say he is not a fan: > > > https://youtu.be/lJ8ydIuPFeU?t=786 > > > > > > > > > > > > > > > > > > > > > > On Fri, May 15, 2020 at 3:15 PM John Roesler < > > > > > vvcep...@apache.org> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks, Sophie! > > > > > > > > > > > > > > > > > > > > > > > > I think this makes perfect sense. It will be much > more > > > > > intuitive > > > > > > > to > > > > > > > > > use > > > > > > > > > > > > the metric for the stated motivation this way. I’d > be > > in > > > > > favor > > > > > > > of > > > > > > > > > the > > > > > > > > > > > > proposal after this update. > > > > > > > > > > > > > > > > > > > > > > > > Thanks again for taking this on, > > > > > > > > > > > > -John > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 15, 2020, at 17:07, Sophie Blee-Goldman > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to clarify/modify one aspect of this KIP, > > > which > > > > > is to > > > > > > > > > record > > > > > > > > > > > the > > > > > > > > > > > > > staleness > > > > > > > > > > > > > at the *completion* of the record's processing by > the > > > > > > > > operator/task > > > > > > > > > > in > > > > > > > > > > > > > question, > > > > > > > > > > > > > rather than on the intake. The task-level metrics > > will > > > be > > > > > > > > recorded > > > > > > > > > at > > > > > > > > > > > the > > > > > > > > > > > > > sink > > > > > > > > > > > > > node instead of at the source, and the > operator-level > > > > > metrics > > > > > > > > will > > > > > > > > > be > > > > > > > > > > > > > recorded > > > > > > > > > > > > > at the end of the operation. > > > > > > > > > > > > > > > > > > > > > > > > > > The stated purpose and intended usefulness of this > > KIP > > > is > > > > > to > > > > > > > give > > > > > > > > > > > users a > > > > > > > > > > > > > way > > > > > > > > > > > > > to gauge roughly how long it takes for a record to > be > > > > > reflected > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > > "results", > > > > > > > > > > > > > whether these results are being read from an output > > > topic > > > > > or > > > > > > > > > through > > > > > > > > > > > IQ. > > > > > > > > > > > > To > > > > > > > > > > > > > take the IQ example, the results of a record are > > > obviously > > > > > not > > > > > > > > > > visible > > > > > > > > > > > > until > > > > > > > > > > > > > *after* that node has finished processing it. The > > > > > staleness, > > > > > > > > while > > > > > > > > > > > still > > > > > > > > > > > > > potentially > > > > > > > > > > > > > useful as it can impact the *way* a record is > > > processed in > > > > > a > > > > > > > > > stateful > > > > > > > > > > > and > > > > > > > > > > > > > time- > > > > > > > > > > > > > dependent operator, is not part of the problem this > > KIP > > > > > > > > > specifically > > > > > > > > > > > set > > > > > > > > > > > > out > > > > > > > > > > > > > to solve. > > > > > > > > > > > > > > > > > > > > > > > > > > In light of this I think it's appropriate to revert > > the > > > > > name > > > > > > > > change > > > > > > > > > > > back > > > > > > > > > > > > to > > > > > > > > > > > > > include > > > > > > > > > > > > > latency, since "staleness" as described above only > > > makes > > > > > sense > > > > > > > > when > > > > > > > > > > > > > measuring > > > > > > > > > > > > > relative to the arrival of a record at a task/node. > > I'd > > > > > propose > > > > > > > > to > > > > > > > > > > save > > > > > > > > > > > > the > > > > > > > > > > > > > term > > > > > > > > > > > > > "staleness" for that particular meaning, and adopt > > > > > Matthias's > > > > > > > > > > > suggestion > > > > > > > > > > > > of > > > > > > > > > > > > > "record-e2e-latency" for this. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for hanging in there all. Please let me know > > if > > > you > > > > > have > > > > > > > > any > > > > > > > > > > > > > concerns about > > > > > > > > > > > > > this change! > > > > > > > > > > > > > > > > > > > > > > > > > > Sophie > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 15, 2020 at 2:25 PM Matthias J. Sax < > > > > > > > > mj...@apache.org> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I am also happy with max/min/99/90. And I buy > your > > > naming > > > > > > > > > argument > > > > > > > > > > > > about > > > > > > > > > > > > > > staleness vs latency. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 5/15/20 12:24 PM, Boyang Chen wrote: > > > > > > > > > > > > > > > Hey Sophie, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 90/99/min/max make sense to me. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 15, 2020 at 12:20 PM Sophie > > > Blee-Goldman < > > > > > > > > > > > > > > sop...@confluent.io> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> @Matthias > > > > > > > > > > > > > > >> Regarding tracking the 50th percentile, I'll > > refer > > > > > you to > > > > > > > > the > > > > > > > > > > 4:53 > > > > > > > > > > > > mark > > > > > > > > > > > > > > of > > > > > > > > > > > > > > >> the video > > > > > > > > > > > > > > >> *you* linked: > > https://youtu.be/lJ8ydIuPFeU?t=293 > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> And that's what he says about the 95th > > percentile! > > > > > Imagine > > > > > > > > > what > > > > > > > > > > he > > > > > > > > > > > > would > > > > > > > > > > > > > > >> say about > > > > > > > > > > > > > > >> the 50th :P > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> But seriously, since we can't seem to agree > that > > > the > > > > > mean > > > > > > > or > > > > > > > > > > 50th > > > > > > > > > > > > > > >> percentile is actually > > > > > > > > > > > > > > >> useful I'm inclined to resurrect my original > > > proposal, > > > > > > > > > neither. > > > > > > > > > > > But > > > > > > > > > > > > I > > > > > > > > > > > > > > think > > > > > > > > > > > > > > >> that's a good > > > > > > > > > > > > > > >> argument against the 75th, which I admittedly > > > chose > > > > > > > somewhat > > > > > > > > > > > > > > arbitrarily as > > > > > > > > > > > > > > >> an > > > > > > > > > > > > > > >> intermediate between the 50th and the higher > > > > > percentiles. > > > > > > > > How > > > > > > > > > > > about: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> -max > > > > > > > > > > > > > > >> -p99 > > > > > > > > > > > > > > >> -p90 > > > > > > > > > > > > > > >> -min > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> with p50/mean still up for debate if anyone > > feels > > > > > strongly > > > > > > > > for > > > > > > > > > > > > either of > > > > > > > > > > > > > > >> them. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Regarding the name, I've already flip-flopped > on > > > this > > > > > so > > > > > > > I'm > > > > > > > > > > > > definitely > > > > > > > > > > > > > > >> still open to > > > > > > > > > > > > > > >> further arguments. But the reason for changing > > it > > > from > > > > > > > > > > > > > > end-to-end-latency > > > > > > > > > > > > > > >> (which > > > > > > > > > > > > > > >> is similar to what you propose) is that this > > > metric > > > > > > > > > technically > > > > > > > > > > > > reflects > > > > > > > > > > > > > > >> how old (ie how "stale") > > > > > > > > > > > > > > >> the record is when it's *received* by the > > > operator, > > > > > not > > > > > > > when > > > > > > > > > > it's > > > > > > > > > > > > > > processed > > > > > > > > > > > > > > >> by the operator. > > > > > > > > > > > > > > >> It seemed like there was the potential for > > > confusion > > > > > that > > > > > > > > > > > > > > >> "end-to-end-latency" might > > > > > > > > > > > > > > >> represent the latency from the event creation > to > > > the > > > > > time > > > > > > > > the > > > > > > > > > > > > processor > > > > > > > > > > > > > > is > > > > > > > > > > > > > > >> done > > > > > > > > > > > > > > >> processing it. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> @John > > > > > > > > > > > > > > >> I'd rather err on the side of "not-enough" > > > metrics as > > > > > we > > > > > > > can > > > > > > > > > > > always > > > > > > > > > > > > add > > > > > > > > > > > > > > >> this to the > > > > > > > > > > > > > > >> stateless metrics later on. If we decide to > > > measure > > > > > the > > > > > > > time > > > > > > > > > at > > > > > > > > > > > > every > > > > > > > > > > > > > > node > > > > > > > > > > > > > > >> and don't > > > > > > > > > > > > > > >> find any evidence of a serious performance > > > impact, and > > > > > > > users > > > > > > > > > > > > indicate > > > > > > > > > > > > > > they > > > > > > > > > > > > > > >> would > > > > > > > > > > > > > > >> like to see this metric at all nodes, then we > > can > > > > > easily > > > > > > > > start > > > > > > > > > > > > reporting > > > > > > > > > > > > > > >> them as well. > > > > > > > > > > > > > > >> WDYT? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> That said, sink nodes seem like a reasonable > > > > > exception to > > > > > > > > the > > > > > > > > > > > rule. > > > > > > > > > > > > > > >> Obviously users > > > > > > > > > > > > > > >> should be able to detect the time when the > > record > > > > > reaches > > > > > > > > the > > > > > > > > > > > output > > > > > > > > > > > > > > topic > > > > > > > > > > > > > > >> but that > > > > > > > > > > > > > > >> still leaves a gap in understanding how long > the > > > > > > > production > > > > > > > > > > > latency > > > > > > > > > > > > was. > > > > > > > > > > > > > > >> This mirrors > > > > > > > > > > > > > > >> the consumption latency that is exposed by the > > > > > task-level > > > > > > > > > > metrics, > > > > > > > > > > > > which > > > > > > > > > > > > > > >> are measured > > > > > > > > > > > > > > >> at the source node. For good symmetry what if > we > > > > > actually > > > > > > > > > expose > > > > > > > > > > > > both > > > > > > > > > > > > > > the > > > > > > > > > > > > > > >> source > > > > > > > > > > > > > > >> and sink latency at the task-level? ie report > > both > > > > > sets of > > > > > > > > > > > > statistical > > > > > > > > > > > > > > >> measurements with > > > > > > > > > > > > > > >> the additional tag -source/-sink > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> @Bill > > > > > > > > > > > > > > >> Thanks for the comment about regarding the > min! > > I > > > > > hadn't > > > > > > > > > > > considered > > > > > > > > > > > > that > > > > > > > > > > > > > > >> and it's > > > > > > > > > > > > > > >> quite useful to think about how and what is > > useful > > > > > from a > > > > > > > > > users > > > > > > > > > > > > point of > > > > > > > > > > > > > > >> view. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Regarding your second. point, I'm inclined to > > > leave > > > > > that > > > > > > > as > > > > > > > > an > > > > > > > > > > > > > > >> implementation detail > > > > > > > > > > > > > > >> but my take would be that the user should be > > > allowed > > > > > to > > > > > > > > > control > > > > > > > > > > > the > > > > > > > > > > > > > > record > > > > > > > > > > > > > > >> timestamp > > > > > > > > > > > > > > >> used for this with the timestamp extractor. My > > > > > impression > > > > > > > is > > > > > > > > > > that > > > > > > > > > > > > users > > > > > > > > > > > > > > may > > > > > > > > > > > > > > >> often embed > > > > > > > > > > > > > > >> the actual event time in the payload for > > whatever > > > > > reason, > > > > > > > > and > > > > > > > > > > this > > > > > > > > > > > > > > >> represents the "true" > > > > > > > > > > > > > > >> timestamp as far as the Streams topology is > > > concerned. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> On Fri, May 15, 2020 at 11:05 AM Bill Bejeck < > > > > > > > > > bbej...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >>> Thanks for the KIP, Sophie, this will be a > > useful > > > > > metric > > > > > > > to > > > > > > > > > > add. > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> Regarding tracking min, I think it could be > > > > > valuable for > > > > > > > > > users > > > > > > > > > > > to > > > > > > > > > > > > > > >> discern > > > > > > > > > > > > > > >>> which part of their topologies are more > > efficient > > > > > since > > > > > > > > this > > > > > > > > > > is a > > > > > > > > > > > > > > >>> task-level metric. I realize everyone seems > to > > > be on > > > > > > > board > > > > > > > > > > with > > > > > > > > > > > > > > >> including > > > > > > > > > > > > > > >>> min anyway, but I wanted to add my 2 cents on > > > this > > > > > topic > > > > > > > > > should > > > > > > > > > > > we > > > > > > > > > > > > > > decide > > > > > > > > > > > > > > >>> to revisit adding min or not. > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> I do have a question regarding the > calculation > > of > > > > > > > > staleness. > > > > > > > > > > > > > > >>> Is there going to be a consideration for > > > timestamp > > > > > > > > > extractors? > > > > > > > > > > > > Users > > > > > > > > > > > > > > >> could > > > > > > > > > > > > > > >>> prefer to use a timestamp embedded in the > > > payload, > > > > > and it > > > > > > > > > could > > > > > > > > > > > > skew > > > > > > > > > > > > > > the > > > > > > > > > > > > > > >>> measurements. > > > > > > > > > > > > > > >>> I was wondering if we should specify in the > KIP > > > if > > > > > > > setting > > > > > > > > > the > > > > > > > > > > > > arrival > > > > > > > > > > > > > > >> time > > > > > > > > > > > > > > >>> is always going to come from the record > > > timestamp, > > > > > or is > > > > > > > > this > > > > > > > > > > an > > > > > > > > > > > > > > >>> implementation detail we can cover in the PR? > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> Thanks! > > > > > > > > > > > > > > >>> Bill > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> On Fri, May 15, 2020 at 1:11 AM Matthias J. > > Sax < > > > > > > > > > > > mj...@apache.org> > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>>> Thanks for the KIP Sophie. > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> I think it's not useful to record the > > avg/mean; > > > it > > > > > > > > sensitive > > > > > > > > > > to > > > > > > > > > > > > > > >>>> outliers. We should rather track the median > > > (50th > > > > > > > > > percentile). > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> Not sure if tracking min is useful, but I am > > > also > > > > > ok to > > > > > > > > > track > > > > > > > > > > > it. > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> However, I find it odd to track 75th > > percentile. > > > > > > > Standard > > > > > > > > > > > measures > > > > > > > > > > > > > > >> would > > > > > > > > > > > > > > >>>> the 90th or 95th -- I guess we don't need > > both, > > > so > > > > > maybe > > > > > > > > > > picking > > > > > > > > > > > > 90th > > > > > > > > > > > > > > >>>> might be more useful? > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> About the name: "staleness" wound really > odd, > > > and if > > > > > > > fact > > > > > > > > > the > > > > > > > > > > > > metric > > > > > > > > > > > > > > >>>> does capture "latency" so we should call it > > > > > "latency". I > > > > > > > > > > > > understand > > > > > > > > > > > > > > the > > > > > > > > > > > > > > >>>> issue that we already have a latency metric. > > So > > > > > maybe we > > > > > > > > > could > > > > > > > > > > > > call it > > > > > > > > > > > > > > >>>> `record-e2e-latency-*` ? > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> While I agree that we should include > > > out-or-order > > > > > data > > > > > > > > (the > > > > > > > > > > KIP > > > > > > > > > > > > should > > > > > > > > > > > > > > >>>> talk about `out-of-order` data, not `late` > > data; > > > > > data is > > > > > > > > > only > > > > > > > > > > > > `late` > > > > > > > > > > > > > > if > > > > > > > > > > > > > > >>>> it's out-of-order and if it's dropped), I > > don't > > > > > really > > > > > > > > > > > understand > > > > > > > > > > > > why > > > > > > > > > > > > > > >>>> the new metric would help to configure grace > > > period > > > > > or > > > > > > > > > > retention > > > > > > > > > > > > time? > > > > > > > > > > > > > > >>>> As you mention in the KIP, both are define > as > > > max > > > > > > > > difference > > > > > > > > > > of > > > > > > > > > > > > > > >>>> `event-time - stream-time` and thus the new > > > metric > > > > > that > > > > > > > > > takes > > > > > > > > > > > > > > >>>> system-/wallclock-time into account does not > > > seem to > > > > > > > help > > > > > > > > at > > > > > > > > > > > all. > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> Btw: there is a great talk about "How NOT to > > > Measure > > > > > > > > > Latency" > > > > > > > > > > by > > > > > > > > > > > > Gil > > > > > > > > > > > > > > >>>> Tene: > > > https://www.youtube.com/watch?v=lJ8ydIuPFeU > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> -Matthias > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> On 5/14/20 7:17 PM, John Roesler wrote: > > > > > > > > > > > > > > >>>>> Hi Sophie, > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > >>>>> It seems like there would still be plenty > of > > > use > > > > > cases > > > > > > > > for > > > > > > > > > > > > recording > > > > > > > > > > > > > > >>>>> this metric at all processors and not just > > > stateful > > > > > > > ones, > > > > > > > > > but > > > > > > > > > > > I'm > > > > > > > > > > > > > > >> happy > > > > > > > > > > > > > > >>>>> to suspend my arguments for now. Since > you're > > > > > proposing > > > > > > > > to > > > > > > > > > > keep > > > > > > > > > > > > > > >>>>> them at the processor-node level, it will > be > > > > > seamless > > > > > > > > later > > > > > > > > > > to > > > > > > > > > > > > add > > > > > > > > > > > > > > >>>>> in the stateless processors if we want. As > a > > > wise > > > > > man > > > > > > > > once > > > > > > > > > > > said, > > > > > > > > > > > > > > >>>>> "Adding is always easier than removing." > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > >>>>> Regarding the time measurement, it's an > > > > > implementation > > > > > > > > > detail > > > > > > > > > > > > > > >>>>> we don't need to consider in the KIP. > > > > > Nevertheless, I'd > > > > > > > > > > greatly > > > > > > > > > > > > > > >>>>> prefer to measure the system time again > when > > > > > recording > > > > > > > > the > > > > > > > > > > > > > > >>>>> metric. I don't think we've seen any > evidence > > > that > > > > > > > proves > > > > > > > > > > this > > > > > > > > > > > > > > >>>>> would harm performance, and the amount of > > > > > inaccuracy > > > > > > > > using > > > > > > > > > > > > > > >>>>> the cached system time could incur is > > actually > > > > > > > > substantial. > > > > > > > > > > > But, > > > > > > > > > > > > > > >>>>> if you want to just "not mention this" in > the > > > KIP, > > > > > we > > > > > > > can > > > > > > > > > > defer > > > > > > > > > > > > to > > > > > > > > > > > > > > >>>>> the actual PR discussion, at which time > we're > > > in a > > > > > > > better > > > > > > > > > > > > position > > > > > > > > > > > > > > >>>>> to use benchmarks, etc., to make the call. > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > >>>>> Along the lines of the measurement accuracy > > > > > discussion, > > > > > > > > one > > > > > > > > > > > > > > >>>>> minor thought I had is that maybe we should > > > > > consider > > > > > > > > > > measuring > > > > > > > > > > > > > > >>>>> the task staleness metric at the sink, > rather > > > than > > > > > the > > > > > > > > > > source, > > > > > > > > > > > so > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > >>>>> it includes the processing latency of the > > task > > > > > itself, > > > > > > > > not > > > > > > > > > > just > > > > > > > > > > > > the > > > > > > > > > > > > > > >>>> latency > > > > > > > > > > > > > > >>>>> of everything up to, but not including, the > > > task > > > > > (which > > > > > > > > > seems > > > > > > > > > > > > > > >> confusing > > > > > > > > > > > > > > >>>>> for users). I guess this could also be an > > > > > > > implementation > > > > > > > > > > > detail, > > > > > > > > > > > > > > >>> though. > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > >>>>> Thanks for the update, > > > > > > > > > > > > > > >>>>> -John > > > > > > > > > > > > > > >>>>> > > > > > > > > > > > > > > >>>>> On Thu, May 14, 2020, at 13:31, Sophie > > > Blee-Goldman > > > > > > > > wrote: > > > > > > > > > > > > > > >>>>>> Hey all, > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> After discussing with Bruno I'd like to > > > propose a > > > > > > > small > > > > > > > > > > > > amendment, > > > > > > > > > > > > > > >>>>>> which is to record the > processor-node-level > > > > > metrics > > > > > > > only > > > > > > > > > for > > > > > > > > > > > > > > >>> *stateful* > > > > > > > > > > > > > > >>>>>> *operators*. They would still be > considered > > a > > > > > > > > > > > > "processor-node-level" > > > > > > > > > > > > > > >>>>>> metric and not a "state-store-level" > metric > > > as the > > > > > > > > > staleness > > > > > > > > > > > is > > > > > > > > > > > > > > >> still > > > > > > > > > > > > > > >>>>>> a property of the node rather than of the > > > state > > > > > > > itself. > > > > > > > > > > > > However, it > > > > > > > > > > > > > > >>>> seems > > > > > > > > > > > > > > >>>>>> that this information is primarily useful > > for > > > > > stateful > > > > > > > > > > > operators > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > >>>> might > > > > > > > > > > > > > > >>>>>> be exposing state via IQ or otherwise > > > dependent > > > > > on the > > > > > > > > > > record > > > > > > > > > > > > time > > > > > > > > > > > > > > >>>>>> unlike a stateless operator. > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> It's worth calling out that recent > > performance > > > > > > > > > improvements > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > >>>> metrics > > > > > > > > > > > > > > >>>>>> framework mean that we no longer fetch the > > > system > > > > > time > > > > > > > > at > > > > > > > > > > the > > > > > > > > > > > > > > >> operator > > > > > > > > > > > > > > >>>>>> level, but only once per task. In other > > words > > > the > > > > > > > system > > > > > > > > > > time > > > > > > > > > > > > is not > > > > > > > > > > > > > > >>>> updated > > > > > > > > > > > > > > >>>>>> between each process as a record flows > > > through the > > > > > > > > > > > subtopology, > > > > > > > > > > > > so > > > > > > > > > > > > > > >>>>>> debugging the processor-level latency via > > the > > > > > > > > stateleness > > > > > > > > > > will > > > > > > > > > > > > not > > > > > > > > > > > > > > >> be > > > > > > > > > > > > > > >>>>>> possible.Note that this doesn't mean the > > > > > > > operator-level > > > > > > > > > > > metrics > > > > > > > > > > > > are > > > > > > > > > > > > > > >>> not > > > > > > > > > > > > > > >>>>>> *useful* relative to the task-level > metric. > > > > > Upstream > > > > > > > > > caching > > > > > > > > > > > > and/or > > > > > > > > > > > > > > >>>>>> suppression > > > > > > > > > > > > > > >>>>>> can still cause a record's staleness at > some > > > > > > > downstream > > > > > > > > > > > stateful > > > > > > > > > > > > > > >>>> operator > > > > > > > > > > > > > > >>>>>> to deviate from the task-level staleness > > > > > (recorded at > > > > > > > > the > > > > > > > > > > > source > > > > > > > > > > > > > > >>> node). > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> Please let me know if you have any > concerns > > > about > > > > > this > > > > > > > > > > change. > > > > > > > > > > > > The > > > > > > > > > > > > > > >>>>>> KIP has been updated with the new proposal > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> On Thu, May 14, 2020 at 3:04 AM Bruno > > Cadonna > > > < > > > > > > > > > > > > br...@confluent.io> > > > > > > > > > > > > > > >>>> wrote: > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>>> Hi Sophie, > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> Thank you for the KIP. > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> The KIP looks good to me. > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> 50th percentile: > > > > > > > > > > > > > > >>>>>>> I think we do not need it now. If we need > > > it, we > > > > > can > > > > > > > > add > > > > > > > > > > it. > > > > > > > > > > > > Here > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > >>>>>>> old truism applies: Adding is always > easier > > > than > > > > > > > > > removing. > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> processor-node-level metrics: > > > > > > > > > > > > > > >>>>>>> I think it is good to have the staleness > > > metrics > > > > > also > > > > > > > > on > > > > > > > > > > > > > > >>>>>>> processor-node-level. If we do not want > to > > > record > > > > > > > them > > > > > > > > on > > > > > > > > > > all > > > > > > > > > > > > > > >>>>>>> processor nodes, you could restrict the > > > > > recording to > > > > > > > > > > stateful > > > > > > > > > > > > > > >>>>>>> processor-nodes, since those are the ones > > > that > > > > > would > > > > > > > > > > benefit > > > > > > > > > > > > most > > > > > > > > > > > > > > >>> from > > > > > > > > > > > > > > >>>>>>> the staleness metrics. > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> Best, > > > > > > > > > > > > > > >>>>>>> Bruno > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> On Thu, May 14, 2020 at 4:15 AM Sophie > > > > > Blee-Goldman < > > > > > > > > > > > > > > >>>> sop...@confluent.io> > > > > > > > > > > > > > > >>>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> Yeah, the specific reason was just to > > align > > > > > with the > > > > > > > > > > current > > > > > > > > > > > > > > >>> metrics. > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> Is it better to conform than to be > right? > > > > > History > > > > > > > has > > > > > > > > a > > > > > > > > > > lot > > > > > > > > > > > > to say > > > > > > > > > > > > > > >>> on > > > > > > > > > > > > > > >>>>>>> that > > > > > > > > > > > > > > >>>>>>>> matter > > > > > > > > > > > > > > >>>>>>>> but I'm not sure how much of it applies > to > > > the > > > > > fine > > > > > > > > > > details > > > > > > > > > > > of > > > > > > > > > > > > > > >>> metrics > > > > > > > > > > > > > > >>>>>>>> naming :P > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> More seriously, I figured if people are > > > looking > > > > > at > > > > > > > > this > > > > > > > > > > > metric > > > > > > > > > > > > > > >>> they're > > > > > > > > > > > > > > >>>>>>>> likely to > > > > > > > > > > > > > > >>>>>>>> be looking at all the others. Then > naming > > > this > > > > > one > > > > > > > > > "-mean" > > > > > > > > > > > > would > > > > > > > > > > > > > > >>>> probably > > > > > > > > > > > > > > >>>>>>>> lead some to conclude that the "-avg" > > > suffix in > > > > > the > > > > > > > > > other > > > > > > > > > > > > metrics > > > > > > > > > > > > > > >>> has > > > > > > > > > > > > > > >>>> a > > > > > > > > > > > > > > >>>>>>>> different meaning. > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> As for the percentiles, I actually like > > p99 > > > (and > > > > > > > p75) > > > > > > > > > > > better. > > > > > > > > > > > > I'll > > > > > > > > > > > > > > >>>> swap > > > > > > > > > > > > > > >>>>>>>> that out > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> On Wed, May 13, 2020 at 7:07 PM John > > > Roesler < > > > > > > > > > > > > vvcep...@apache.org > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> Thanks Sophie, > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> I hope this isn't too nit-picky, but is > > > there a > > > > > > > > reason > > > > > > > > > to > > > > > > > > > > > > choose > > > > > > > > > > > > > > >>>> "avg" > > > > > > > > > > > > > > >>>>>>>>> instead > > > > > > > > > > > > > > >>>>>>>>> of "mean"? Maybe this is too paranoid, > > and > > > I > > > > > might > > > > > > > be > > > > > > > > > > > > > > >> oversensitive > > > > > > > > > > > > > > >>>>>>> because > > > > > > > > > > > > > > >>>>>>>>> of the mistake I just made earlier, but > > it > > > > > strikes > > > > > > > me > > > > > > > > > > that > > > > > > > > > > > > "avg" > > > > > > > > > > > > > > >> is > > > > > > > > > > > > > > >>>>>>>>> actually > > > > > > > > > > > > > > >>>>>>>>> ambiguous, as it refers to a family of > > > > > statistics, > > > > > > > > > > whereas > > > > > > > > > > > > "mean" > > > > > > > > > > > > > > >>> is > > > > > > > > > > > > > > >>>>>>>>> specific. > > > > > > > > > > > > > > >>>>>>>>> I see other Kafka metrics with "avg", > but > > > none > > > > > with > > > > > > > > > > "mean"; > > > > > > > > > > > > was > > > > > > > > > > > > > > >>> that > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > >>>>>>>>> reason? If so, I'm +1. > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> Regarding the names of the percentile, > I > > > > > actually > > > > > > > > > > couldn't > > > > > > > > > > > > find > > > > > > > > > > > > > > >>> _any_ > > > > > > > > > > > > > > >>>>>>> other > > > > > > > > > > > > > > >>>>>>>>> metrics that use percentile. Was there > a > > > > > reason to > > > > > > > > > choose > > > > > > > > > > > > "99th" > > > > > > > > > > > > > > >> as > > > > > > > > > > > > > > >>>>>>> opposed > > > > > > > > > > > > > > >>>>>>>>> to "p99" or any other scheme? This is > > not a > > > > > > > > criticism, > > > > > > > > > > I'm > > > > > > > > > > > > just > > > > > > > > > > > > > > >>>>>>> primarily > > > > > > > > > > > > > > >>>>>>>>> asking > > > > > > > > > > > > > > >>>>>>>>> for consistency's sake. > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> Thanks again, > > > > > > > > > > > > > > >>>>>>>>> -John > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> On Wed, May 13, 2020, at 19:19, Sophie > > > > > Blee-Goldman > > > > > > > > > > wrote: > > > > > > > > > > > > > > >>>>>>>>>> Alright, I can get behind adding the > min > > > > > metric > > > > > > > for > > > > > > > > > the > > > > > > > > > > > > sake of > > > > > > > > > > > > > > >>>>>>> pretty > > > > > > > > > > > > > > >>>>>>>>>> graphs > > > > > > > > > > > > > > >>>>>>>>>> (and trivial computation). > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> I'm still on the fence regarding the > > mean > > > (or > > > > > 50th > > > > > > > > > > > > percentile) > > > > > > > > > > > > > > >>> but I > > > > > > > > > > > > > > >>>>>>> can > > > > > > > > > > > > > > >>>>>>>>> see > > > > > > > > > > > > > > >>>>>>>>>> how users might expect it and find it > a > > > bit > > > > > > > > > disorienting > > > > > > > > > > > > not to > > > > > > > > > > > > > > >>>>>>> have. So > > > > > > > > > > > > > > >>>>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>> updated proposed metrics are > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> - record-staleness-max [ms] > > > > > > > > > > > > > > >>>>>>>>>> - record-staleness-99th [ms] *(99th > > > > > > > percentile)* > > > > > > > > > > > > > > >>>>>>>>>> - record-staleness-75th [ms] *(75th > > > > > > > percentile)* > > > > > > > > > > > > > > >>>>>>>>>> - record-staleness-avg [ms] > *(mean)* > > > > > > > > > > > > > > >>>>>>>>>> - record-staleness-min [ms] > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> On Wed, May 13, 2020 at 4:42 PM John > > > Roesler < > > > > > > > > > > > > > > >> vvcep...@apache.org > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>>>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> Oh boy, I never miss an opportunity > to > > > > > embarrass > > > > > > > > > > myself. > > > > > > > > > > > I > > > > > > > > > > > > > > >> guess > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > >>>>>>>>> mean > > > > > > > > > > > > > > >>>>>>>>>>> seems more interesting to me than the > > > > > median, but > > > > > > > > > > neither > > > > > > > > > > > > are > > > > > > > > > > > > > > >> as > > > > > > > > > > > > > > >>>>>>>>>>> interesting as the higher percentiles > > > (99th > > > > > and > > > > > > > > max). > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> Min isn’t really important for any > > SLAs, > > > but > > > > > it > > > > > > > > does > > > > > > > > > > > round > > > > > > > > > > > > out > > > > > > > > > > > > > > >>> the > > > > > > > > > > > > > > >>>>>>>>> mental > > > > > > > > > > > > > > >>>>>>>>>>> picture of the distribution. I’ve > > always > > > > > graphed > > > > > > > > min > > > > > > > > > > > along > > > > > > > > > > > > with > > > > > > > > > > > > > > >>> the > > > > > > > > > > > > > > >>>>>>>>> other > > > > > > > > > > > > > > >>>>>>>>>>> metrics to help me understand how > fast > > > the > > > > > system > > > > > > > > can > > > > > > > > > > be, > > > > > > > > > > > > which > > > > > > > > > > > > > > >>>>>>> helps > > > > > > > > > > > > > > >>>>>>>>> in > > > > > > > > > > > > > > >>>>>>>>>>> optimization decisions. It’s also a > > > > > relatively > > > > > > > > > > > inexpensive > > > > > > > > > > > > > > >> metric > > > > > > > > > > > > > > >>>>>>> to > > > > > > > > > > > > > > >>>>>>>>>>> compute, so it might be nice to just > > > throw > > > > > it in. > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> On Wed, May 13, 2020, at 18:18, > Sophie > > > > > > > Blee-Goldman > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >>>>>>>>>>>> G1: > > > > > > > > > > > > > > >>>>>>>>>>>> I was considering it as the > > "end-to-end > > > > > latency > > > > > > > > *up* > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > >>>>>>> specific > > > > > > > > > > > > > > >>>>>>>>>>> task" > > > > > > > > > > > > > > >>>>>>>>>>>> but > > > > > > > > > > > > > > >>>>>>>>>>>> I'm happy with "record-staleness" if > > > that > > > > > drives > > > > > > > > the > > > > > > > > > > > point > > > > > > > > > > > > > > >> home > > > > > > > > > > > > > > >>>>>>>>> better. > > > > > > > > > > > > > > >>>>>>>>>>> So > > > > > > > > > > > > > > >>>>>>>>>>>> it's the > > > > > > > > > > > > > > >>>>>>>>>>>> "staleness of the record when it is > > > > > received by > > > > > > > > that > > > > > > > > > > > > task" -- > > > > > > > > > > > > > > >>>>>>> will > > > > > > > > > > > > > > >>>>>>>>> update > > > > > > > > > > > > > > >>>>>>>>>>>> the KIP > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>> B1/J: > > > > > > > > > > > > > > >>>>>>>>>>>> I'm struggling to imagine a case > where > > > the > > > > > min > > > > > > > > would > > > > > > > > > > > > actually > > > > > > > > > > > > > > >> be > > > > > > > > > > > > > > >>>>>>>>> useful, > > > > > > > > > > > > > > >>>>>>>>>>>> rather than > > > > > > > > > > > > > > >>>>>>>>>>>> just intellectually interesting. I > > don't > > > > > feel > > > > > > > > > strongly > > > > > > > > > > > > that we > > > > > > > > > > > > > > >>>>>>>>> shouldn't > > > > > > > > > > > > > > >>>>>>>>>>>> add it, but that's > > > > > > > > > > > > > > >>>>>>>>>>>> why I didn't include it from the > > start. > > > Can > > > > > you > > > > > > > > > > > enlighten > > > > > > > > > > > > me > > > > > > > > > > > > > > >>>>>>> with an > > > > > > > > > > > > > > >>>>>>>>>>>> example? > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>> I was also vaguely concerned about > the > > > > > overhead > > > > > > > of > > > > > > > > > > > adding > > > > > > > > > > > > > > >>>>>>> multiple > > > > > > > > > > > > > > >>>>>>>>>>>> percentile > > > > > > > > > > > > > > >>>>>>>>>>>> metrics. Do we have any data to > > indicate > > > > > what > > > > > > > kind > > > > > > > > > of > > > > > > > > > > > > > > >>> performance > > > > > > > > > > > > > > >>>>>>>>> hit we > > > > > > > > > > > > > > >>>>>>>>>>>> take on > > > > > > > > > > > > > > >>>>>>>>>>>> metrics computation? > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>> Also, not to be too pedantic but the > > > 50th > > > > > > > > percentile > > > > > > > > > > > > would be > > > > > > > > > > > > > > >>> the > > > > > > > > > > > > > > >>>>>>>>> median > > > > > > > > > > > > > > >>>>>>>>>>>> not the > > > > > > > > > > > > > > >>>>>>>>>>>> mean. Would you propose to add the > > mean > > > > > *and* > > > > > > > the > > > > > > > > > 50th > > > > > > > > > > > > > > >>>>>>> percentile, or > > > > > > > > > > > > > > >>>>>>>>>>> just > > > > > > > > > > > > > > >>>>>>>>>>>> one > > > > > > > > > > > > > > >>>>>>>>>>>> of the two? > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>> Thanks all! > > > > > > > > > > > > > > >>>>>>>>>>>> Sophie > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>> On Wed, May 13, 2020 at 3:34 PM John > > > > > Roesler < > > > > > > > > > > > > > > >>>>>>> vvcep...@apache.org> > > > > > > > > > > > > > > >>>>>>>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> Hello all, and thanks for the KIP, > > > Sophie, > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> Just some comments on the > discussion > > so > > > > > far: > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> B2/G1: > > > > > > > > > > > > > > >>>>>>>>>>>>> In principle, it shouldn't matter > > > whether > > > > > we > > > > > > > > report > > > > > > > > > > > > "spans" > > > > > > > > > > > > > > >> or > > > > > > > > > > > > > > >>>>>>>>>>>>> "end-to-end" latency. But in > > practice, > > > > > some of > > > > > > > > the > > > > > > > > > > > spans > > > > > > > > > > > > are > > > > > > > > > > > > > > >>>>>>> pretty > > > > > > > > > > > > > > >>>>>>>>>>>>> difficult to really measure (like > > time > > > > > spent > > > > > > > > > waiting > > > > > > > > > > in > > > > > > > > > > > > the > > > > > > > > > > > > > > >>>>>>>>> topics, or > > > > > > > > > > > > > > >>>>>>>>>>>>> time from the event happening to > the > > > ETL > > > > > > > producer > > > > > > > > > > > > choosing to > > > > > > > > > > > > > > >>>>>>> send > > > > > > > > > > > > > > >>>>>>>>> it, > > > > > > > > > > > > > > >>>>>>>>>>>>> or time spent in send/receive > > buffers, > > > > > etc., > > > > > > > etc. > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> In other words, it's practically > > > easier to > > > > > > > > compute > > > > > > > > > > > spans > > > > > > > > > > > > by > > > > > > > > > > > > > > >>>>>>>>> subtracting > > > > > > > > > > > > > > >>>>>>>>>>>>> e2e latencies than it is to compute > > e2e > > > > > > > latencies > > > > > > > > > by > > > > > > > > > > > > adding > > > > > > > > > > > > > > >>>>>>> spans. > > > > > > > > > > > > > > >>>>>>>>> You > > > > > > > > > > > > > > >>>>>>>>>>>>> can even just consider that the > span > > > > > > > computation > > > > > > > > > from > > > > > > > > > > > e2e > > > > > > > > > > > > > > >>>>>>> always > > > > > > > > > > > > > > >>>>>>>>> just > > > > > > > > > > > > > > >>>>>>>>>>>>> involves subtracting two numbers, > > > whereas > > > > > > > > computing > > > > > > > > > > e2e > > > > > > > > > > > > > > >> latency > > > > > > > > > > > > > > >>>>>>>>> from > > > > > > > > > > > > > > >>>>>>>>>>>>> spans involves adding _all_ the > spans > > > > > leading > > > > > > > up > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > > end > > > > > > > > > > > > > > >> you > > > > > > > > > > > > > > >>>>>>>>> care > > > > > > > > > > > > > > >>>>>>>>>>> about. > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> It seems like people really prefer > to > > > have > > > > > > > spans > > > > > > > > > when > > > > > > > > > > > > they > > > > > > > > > > > > > > >> are > > > > > > > > > > > > > > >>>>>>>>>>> debugging > > > > > > > > > > > > > > >>>>>>>>>>>>> latency problems, whereas e2e > latency > > > is a > > > > > more > > > > > > > > > > general > > > > > > > > > > > > > > >>>>>>> measurement > > > > > > > > > > > > > > >>>>>>>>>>>>> that basically every > > person/application > > > > > cares > > > > > > > > about > > > > > > > > > > and > > > > > > > > > > > > > > >> should > > > > > > > > > > > > > > >>>>>>> be > > > > > > > > > > > > > > >>>>>>>>>>>>> monitoring. > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> Altogether, it really seem to > provide > > > more > > > > > > > value > > > > > > > > to > > > > > > > > > > > more > > > > > > > > > > > > > > >>>>>>> people if > > > > > > > > > > > > > > >>>>>>>>> we > > > > > > > > > > > > > > >>>>>>>>>>>>> report > > > > > > > > > > > > > > >>>>>>>>>>>>> e2e latencies. Regarding > > > > > "record-staleness" as > > > > > > > a > > > > > > > > > > name, > > > > > > > > > > > I > > > > > > > > > > > > > > >> think > > > > > > > > > > > > > > >>>>>>> I > > > > > > > > > > > > > > >>>>>>>>> have > > > > > > > > > > > > > > >>>>>>>>>>> no > > > > > > > > > > > > > > >>>>>>>>>>>>> preference, I'd defer to other > > peoples' > > > > > > > > intuition. > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> G2: > > > > > > > > > > > > > > >>>>>>>>>>>>> I think the processor-node metric > is > > > nice, > > > > > > > since > > > > > > > > > the > > > > > > > > > > > > inside > > > > > > > > > > > > > > >> of > > > > > > > > > > > > > > >>>>>>> a > > > > > > > > > > > > > > >>>>>>>>> task > > > > > > > > > > > > > > >>>>>>>>>>> can > > > > > > > > > > > > > > >>>>>>>>>>>>> introduce a significant amount of > > > latency > > > > > in > > > > > > > some > > > > > > > > > > > cases. > > > > > > > > > > > > > > >> Plus, > > > > > > > > > > > > > > >>>>>>>>> it's a > > > > > > > > > > > > > > >>>>>>>>>>> more > > > > > > > > > > > > > > >>>>>>>>>>>>> direct measurement, if you really > > > wanted to > > > > > > > know > > > > > > > > > (for > > > > > > > > > > > the > > > > > > > > > > > > > > >>>>>>> purposes > > > > > > > > > > > > > > >>>>>>>>> of > > > > > > > > > > > > > > >>>>>>>>>>> IQ > > > > > > > > > > > > > > >>>>>>>>>>>>> or something) how long it takes > > source > > > > > events > > > > > > > to > > > > > > > > > > "show > > > > > > > > > > > > up" at > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>> store. > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> I think actually recording it at > > every > > > > > > > processor > > > > > > > > > > could > > > > > > > > > > > be > > > > > > > > > > > > > > >>>>>>>>> expensive, > > > > > > > > > > > > > > >>>>>>>>>>> but we > > > > > > > > > > > > > > >>>>>>>>>>>>> already record a bunch of metrics > at > > > the > > > > > node > > > > > > > > > level. > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> B1: > > > > > > > > > > > > > > >>>>>>>>>>>>> I think 50% could be reasonable to > > > record > > > > > also. > > > > > > > > > Even > > > > > > > > > > if > > > > > > > > > > > > it's > > > > > > > > > > > > > > >> a > > > > > > > > > > > > > > >>>>>>> poor > > > > > > > > > > > > > > >>>>>>>>>>> metric > > > > > > > > > > > > > > >>>>>>>>>>>>> for operational purposes, a lot of > > > people > > > > > might > > > > > > > > > > expect > > > > > > > > > > > > to see > > > > > > > > > > > > > > >>>>>>>>> "mean". > > > > > > > > > > > > > > >>>>>>>>>>>>> Actually, > > > > > > > > > > > > > > >>>>>>>>>>>>> I was surprised not to see "min". > Is > > > there > > > > > a > > > > > > > > reason > > > > > > > > > > to > > > > > > > > > > > > leave > > > > > > > > > > > > > > >> it > > > > > > > > > > > > > > >>>>>>>>> off? > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> I might suggest: > > > > > > > > > > > > > > >>>>>>>>>>>>> min, mean (50th), 75th, 99th, max > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> B3: > > > > > > > > > > > > > > >>>>>>>>>>>>> I agree we should include late > > records > > > > > (though > > > > > > > > not > > > > > > > > > > the > > > > > > > > > > > > ones > > > > > > > > > > > > > > >> we > > > > > > > > > > > > > > >>>>>>>>> drop). > > > > > > > > > > > > > > >>>>>>>>>>>>> It may be spiky, but only when > there > > > are > > > > > > > > > legitimately > > > > > > > > > > > > some > > > > > > > > > > > > > > >>>>>>> records > > > > > > > > > > > > > > >>>>>>>>>>> with a > > > > > > > > > > > > > > >>>>>>>>>>>>> high end-to-end latency, which is > the > > > whole > > > > > > > point > > > > > > > > > of > > > > > > > > > > > > these > > > > > > > > > > > > > > >>>>>>> metrics. > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> That's it! I don't think I have any > > > other > > > > > > > > feedback, > > > > > > > > > > > other > > > > > > > > > > > > > > >> than > > > > > > > > > > > > > > >>>>>>> a > > > > > > > > > > > > > > >>>>>>>>>>> request to > > > > > > > > > > > > > > >>>>>>>>>>>>> also report "min". > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> Thanks, > > > > > > > > > > > > > > >>>>>>>>>>>>> -John > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> On Wed, May 13, 2020, at 16:58, > > > Guozhang > > > > > Wang > > > > > > > > > wrote: > > > > > > > > > > > > > > >>>>>>>>>>>>>> Thanks Sophie for the KIP, a few > > quick > > > > > > > thoughts: > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> 1) The end-to-end latency includes > > > both > > > > > the > > > > > > > > > > processing > > > > > > > > > > > > > > >>>>>>> latency > > > > > > > > > > > > > > >>>>>>>>> of the > > > > > > > > > > > > > > >>>>>>>>>>>>> task > > > > > > > > > > > > > > >>>>>>>>>>>>>> and the latency spent sitting in > > > > > intermediate > > > > > > > > > > topics. > > > > > > > > > > > I > > > > > > > > > > > > have > > > > > > > > > > > > > > >>>>>>> a > > > > > > > > > > > > > > >>>>>>>>>>> similar > > > > > > > > > > > > > > >>>>>>>>>>>>>> feeling as Boyang mentioned above > > > that the > > > > > > > > latency > > > > > > > > > > > > metric of > > > > > > > > > > > > > > >>>>>>> a > > > > > > > > > > > > > > >>>>>>>>> task A > > > > > > > > > > > > > > >>>>>>>>>>>>>> actually measures the latency of > the > > > > > > > > sub-topology > > > > > > > > > > > up-to > > > > > > > > > > > > but > > > > > > > > > > > > > > >>>>>>> not > > > > > > > > > > > > > > >>>>>>>>>>> including > > > > > > > > > > > > > > >>>>>>>>>>>>>> the processing of A, which is a > bit > > > weird. > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> Maybe the my feeling comes from > the > > > name > > > > > > > > "latency" > > > > > > > > > > > > itself, > > > > > > > > > > > > > > >>>>>>> since > > > > > > > > > > > > > > >>>>>>>>>>> today we > > > > > > > > > > > > > > >>>>>>>>>>>>>> already have several "latency" > > metrics > > > > > already > > > > > > > > > which > > > > > > > > > > > are > > > > > > > > > > > > > > >>>>>>>>> measuring > > > > > > > > > > > > > > >>>>>>>>>>>>> elapsed > > > > > > > > > > > > > > >>>>>>>>>>>>>> system-time for processing a > record > > / > > > etc, > > > > > > > while > > > > > > > > > > here > > > > > > > > > > > > we are > > > > > > > > > > > > > > >>>>>>>>>>> comparing > > > > > > > > > > > > > > >>>>>>>>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>>>>> system wallclock time with the > > record > > > > > > > timestamp. > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> Maybe we can consider renaming it > as > > > > > > > > > > > "record-staleness" > > > > > > > > > > > > > > >>>>>>> (note we > > > > > > > > > > > > > > >>>>>>>>>>> already > > > > > > > > > > > > > > >>>>>>>>>>>>>> have a "record-lateness" metric), > in > > > which > > > > > > > case > > > > > > > > > > > > recording at > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>>>>> system-time before we start > > > processing the > > > > > > > > record > > > > > > > > > > > sounds > > > > > > > > > > > > > > >> more > > > > > > > > > > > > > > >>>>>>>>>>> natural. > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> 2) With that in mind, I'm > wondering > > > if the > > > > > > > > > > > > > > >>>>>>> processor-node-level > > > > > > > > > > > > > > >>>>>>>>> DEBUG > > > > > > > > > > > > > > >>>>>>>>>>>>>> metric is worth to add, given that > > we > > > > > already > > > > > > > > > have a > > > > > > > > > > > > > > >>>>>>> task-level > > > > > > > > > > > > > > >>>>>>>>>>>>> processing > > > > > > > > > > > > > > >>>>>>>>>>>>>> latency metric. Basically, a > > specific > > > > > node's > > > > > > > e2e > > > > > > > > > > > > latency is > > > > > > > > > > > > > > >>>>>>>>> similar > > > > > > > > > > > > > > >>>>>>>>>>> to > > > > > > > > > > > > > > >>>>>>>>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>>>>> task-level e2e latency + > task-level > > > > > processing > > > > > > > > > > > latency. > > > > > > > > > > > > > > >>>>>>>>> Personally I > > > > > > > > > > > > > > >>>>>>>>>>>>> think > > > > > > > > > > > > > > >>>>>>>>>>>>>> having a task-level > record-staleness > > > > > metric is > > > > > > > > > > > > sufficient. > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> Guozhang > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> On Wed, May 13, 2020 at 11:46 AM > > > Sophie > > > > > > > > > > Blee-Goldman < > > > > > > > > > > > > > > >>>>>>>>>>>>> sop...@confluent.io> > > > > > > > > > > > > > > >>>>>>>>>>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> 1. I felt that 50% was not a > > > particularly > > > > > > > > useful > > > > > > > > > > > gauge > > > > > > > > > > > > for > > > > > > > > > > > > > > >>>>>>> this > > > > > > > > > > > > > > >>>>>>>>>>>>> specific > > > > > > > > > > > > > > >>>>>>>>>>>>>>> metric, as > > > > > > > > > > > > > > >>>>>>>>>>>>>>> it's presumably most useful at > > > putting an > > > > > > > > *upper > > > > > > > > > > > > *bound on > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>> latency > > > > > > > > > > > > > > >>>>>>>>>>>>> you > > > > > > > > > > > > > > >>>>>>>>>>>>>>> can > > > > > > > > > > > > > > >>>>>>>>>>>>>>> reasonably expect to see. I chose > > > > > percentiles > > > > > > > > > that > > > > > > > > > > > > would > > > > > > > > > > > > > > >>>>>>>>> hopefully > > > > > > > > > > > > > > >>>>>>>>>>>>> give a > > > > > > > > > > > > > > >>>>>>>>>>>>>>> good > > > > > > > > > > > > > > >>>>>>>>>>>>>>> sense of what *most* records will > > > > > experience, > > > > > > > > and > > > > > > > > > > > what > > > > > > > > > > > > > > >>>>>>> *close > > > > > > > > > > > > > > >>>>>>>>> to > > > > > > > > > > > > > > >>>>>>>>>>> all* > > > > > > > > > > > > > > >>>>>>>>>>>>>>> records > > > > > > > > > > > > > > >>>>>>>>>>>>>>> will. > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> However I'm not married to these > > > specific > > > > > > > > numbers > > > > > > > > > > and > > > > > > > > > > > > > > >>>>>>> could be > > > > > > > > > > > > > > >>>>>>>>>>>>> convinced. > > > > > > > > > > > > > > >>>>>>>>>>>>>>> Would be especially interested in > > > hearing > > > > > > > from > > > > > > > > > > users > > > > > > > > > > > on > > > > > > > > > > > > > > >>>>>>> this. > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> 2. I'm inclined to not include > the > > > > > > > "hop-to-hop > > > > > > > > > > > > latency" in > > > > > > > > > > > > > > >>>>>>>>> this KIP > > > > > > > > > > > > > > >>>>>>>>>>>>> since > > > > > > > > > > > > > > >>>>>>>>>>>>>>> users > > > > > > > > > > > > > > >>>>>>>>>>>>>>> can always compute it themselves > by > > > > > > > subtracting > > > > > > > > > the > > > > > > > > > > > > > > >>>>>>> previous > > > > > > > > > > > > > > >>>>>>>>> node's > > > > > > > > > > > > > > >>>>>>>>>>>>>>> end-to-end latency. I guess we > > could > > > do > > > > > it > > > > > > > > either > > > > > > > > > > way > > > > > > > > > > > > since > > > > > > > > > > > > > > >>>>>>>>> you can > > > > > > > > > > > > > > >>>>>>>>>>>>> always > > > > > > > > > > > > > > >>>>>>>>>>>>>>> compute one from the other, but I > > > think > > > > > the > > > > > > > > > > > end-to-end > > > > > > > > > > > > > > >>>>>>> latency > > > > > > > > > > > > > > >>>>>>>>>>> feels > > > > > > > > > > > > > > >>>>>>>>>>>>> more > > > > > > > > > > > > > > >>>>>>>>>>>>>>> valuable as it's main motivation > is > > > not > > > > > to > > > > > > > > debug > > > > > > > > > > > > > > >>>>>>> bottlenecks > > > > > > > > > > > > > > >>>>>>>>> in the > > > > > > > > > > > > > > >>>>>>>>>>>>>>> topology but > > > > > > > > > > > > > > >>>>>>>>>>>>>>> to give users a sense of how long > > it > > > > > takes > > > > > > > > > arecord > > > > > > > > > > to > > > > > > > > > > > > be > > > > > > > > > > > > > > >>>>>>>>> reflected > > > > > > > > > > > > > > >>>>>>>>>>> in > > > > > > > > > > > > > > >>>>>>>>>>>>>>> certain parts > > > > > > > > > > > > > > >>>>>>>>>>>>>>> of the topology. For example this > > > might > > > > > be > > > > > > > > useful > > > > > > > > > > for > > > > > > > > > > > > users > > > > > > > > > > > > > > >>>>>>>>> who are > > > > > > > > > > > > > > >>>>>>>>>>>>>>> wondering > > > > > > > > > > > > > > >>>>>>>>>>>>>>> roughly when a record that was > just > > > > > produced > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > > > > > >>>>>>> included > > > > > > > > > > > > > > >>>>>>>>> in > > > > > > > > > > > > > > >>>>>>>>>>> their > > > > > > > > > > > > > > >>>>>>>>>>>>> IQ > > > > > > > > > > > > > > >>>>>>>>>>>>>>> results. > > > > > > > > > > > > > > >>>>>>>>>>>>>>> Debugging is just a nice side > > effect > > > -- > > > > > but > > > > > > > > > maybe I > > > > > > > > > > > > didn't > > > > > > > > > > > > > > >>>>>>> make > > > > > > > > > > > > > > >>>>>>>>>>> that > > > > > > > > > > > > > > >>>>>>>>>>>>> clear > > > > > > > > > > > > > > >>>>>>>>>>>>>>> enough > > > > > > > > > > > > > > >>>>>>>>>>>>>>> in the KIP's motivation. > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> 3. Good question, I should > address > > > this > > > > > in > > > > > > > the > > > > > > > > > KIP. > > > > > > > > > > > The > > > > > > > > > > > > > > >>>>>>> short > > > > > > > > > > > > > > >>>>>>>>>>> answer is > > > > > > > > > > > > > > >>>>>>>>>>>>>>> "yes", > > > > > > > > > > > > > > >>>>>>>>>>>>>>> we will include late records. I > > > added a > > > > > > > > paragraph > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > >>>>>>> end > > > > > > > > > > > > > > >>>>>>>>> of the > > > > > > > > > > > > > > >>>>>>>>>>>>>>> Proposed > > > > > > > > > > > > > > >>>>>>>>>>>>>>> Changes section explaining the > > > reasoning > > > > > > > here, > > > > > > > > > > please > > > > > > > > > > > > let > > > > > > > > > > > > > > >>>>>>> me > > > > > > > > > > > > > > >>>>>>>>> know > > > > > > > > > > > > > > >>>>>>>>>>> if > > > > > > > > > > > > > > >>>>>>>>>>>>> you > > > > > > > > > > > > > > >>>>>>>>>>>>>>> have > > > > > > > > > > > > > > >>>>>>>>>>>>>>> any concerns. > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> 4. Assuming you're referring to > the > > > > > existing > > > > > > > > > metric > > > > > > > > > > > > > > >>>>>>>>>>> "process-latency", > > > > > > > > > > > > > > >>>>>>>>>>>>> that > > > > > > > > > > > > > > >>>>>>>>>>>>>>> metric > > > > > > > > > > > > > > >>>>>>>>>>>>>>> reflects the time for the literal > > > > > > > Node#process > > > > > > > > > > method > > > > > > > > > > > > to > > > > > > > > > > > > > > >>>>>>> run > > > > > > > > > > > > > > >>>>>>>>>>> whereas > > > > > > > > > > > > > > >>>>>>>>>>>>> this > > > > > > > > > > > > > > >>>>>>>>>>>>>>> metric > > > > > > > > > > > > > > >>>>>>>>>>>>>>> would always be measured relative > > to > > > the > > > > > > > event > > > > > > > > > > > > timestamp. > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> That said, the naming collision > > > there is > > > > > > > pretty > > > > > > > > > > > > confusing > > > > > > > > > > > > > > >>>>>>> so > > > > > > > > > > > > > > >>>>>>>>> I've > > > > > > > > > > > > > > >>>>>>>>>>>>> renamed > > > > > > > > > > > > > > >>>>>>>>>>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>>>>>> metrics in this KIP to > > > > > "end-to-end-latency" > > > > > > > > > which I > > > > > > > > > > > > feel > > > > > > > > > > > > > > >>>>>>> better > > > > > > > > > > > > > > >>>>>>>>>>>>> reflects > > > > > > > > > > > > > > >>>>>>>>>>>>>>> the nature > > > > > > > > > > > > > > >>>>>>>>>>>>>>> of the metric anyway. > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> Thanks for the feedback! > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> On Wed, May 13, 2020 at 10:21 AM > > > Boyang > > > > > Chen > > > > > > > < > > > > > > > > > > > > > > >>>>>>>>>>>>> reluctanthero...@gmail.com> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> Thanks for the KIP Sophie. > Getting > > > the > > > > > E2E > > > > > > > > > latency > > > > > > > > > > > is > > > > > > > > > > > > > > >>>>>>>>> important > > > > > > > > > > > > > > >>>>>>>>>>> for > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> understanding the bottleneck of > > the > > > > > > > > application. > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> A couple of questions and ideas: > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> 1. Could you clarify the > rational > > of > > > > > picking > > > > > > > > 75, > > > > > > > > > > 99 > > > > > > > > > > > > and > > > > > > > > > > > > > > >>>>>>> max > > > > > > > > > > > > > > >>>>>>>>>>>>> percentiles? > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> Normally I see cases where we > use > > > 50, 90 > > > > > > > > > > percentile > > > > > > > > > > > as > > > > > > > > > > > > > > >>>>>>> well > > > > > > > > > > > > > > >>>>>>>>> in > > > > > > > > > > > > > > >>>>>>>>>>>>> production > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> systems. > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> 2. The current latency being > > > computed is > > > > > > > > > > cumulative, > > > > > > > > > > > > I.E > > > > > > > > > > > > > > >>>>>>> if a > > > > > > > > > > > > > > >>>>>>>>>>> record > > > > > > > > > > > > > > >>>>>>>>>>>>> goes > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> through A -> B -> C, then P(C) = > > > > > T(B->C) + > > > > > > > > P(B) > > > > > > > > > = > > > > > > > > > > > > > > >>>>>>> T(B->C) + > > > > > > > > > > > > > > >>>>>>>>>>> T(A->B) + > > > > > > > > > > > > > > >>>>>>>>>>>>>>> T(A) > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> and so on, where P() represents > > the > > > > > captured > > > > > > > > > > > latency, > > > > > > > > > > > > > > >>>>>>> and T() > > > > > > > > > > > > > > >>>>>>>>>>>>> represents > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> the time for transiting the > > records > > > > > between > > > > > > > > two > > > > > > > > > > > nodes, > > > > > > > > > > > > > > >>>>>>>>> including > > > > > > > > > > > > > > >>>>>>>>>>>>>>> processing > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> time. For monitoring purpose, > > maybe > > > > > having > > > > > > > > > T(B->C) > > > > > > > > > > > and > > > > > > > > > > > > > > >>>>>>>>> T(A->B) > > > > > > > > > > > > > > >>>>>>>>>>> are > > > > > > > > > > > > > > >>>>>>>>>>>>> more > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> natural to view as "hop-to-hop > > > latency", > > > > > > > > > otherwise > > > > > > > > > > > if > > > > > > > > > > > > > > >>>>>>> there > > > > > > > > > > > > > > >>>>>>>>> is a > > > > > > > > > > > > > > >>>>>>>>>>>>> spike in > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> T(A->B), both P(B) and P(C) are > > > > > affected in > > > > > > > > the > > > > > > > > > > same > > > > > > > > > > > > > > >>>>>>> time. > > > > > > > > > > > > > > >>>>>>>>> In the > > > > > > > > > > > > > > >>>>>>>>>>>>> same > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> spirit, the E2E latency is > > > meaningful > > > > > only > > > > > > > > when > > > > > > > > > > the > > > > > > > > > > > > > > >>>>>>> record > > > > > > > > > > > > > > >>>>>>>>> exits > > > > > > > > > > > > > > >>>>>>>>>>>>> from the > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> sink as this marks the whole > time > > > this > > > > > > > record > > > > > > > > > > spent > > > > > > > > > > > > > > >>>>>>> inside > > > > > > > > > > > > > > >>>>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>>>> funnel. Do > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> you think we could have separate > > > > > treatment > > > > > > > for > > > > > > > > > > sink > > > > > > > > > > > > > > >>>>>>> nodes and > > > > > > > > > > > > > > >>>>>>>>>>> other > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> nodes, so that other nodes only > > > count > > > > > the > > > > > > > time > > > > > > > > > > > > receiving > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>> record > > > > > > > > > > > > > > >>>>>>>>>>>>> from > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> last hop? I'm not proposing a > > > solution > > > > > here, > > > > > > > > > just > > > > > > > > > > > > want to > > > > > > > > > > > > > > >>>>>>>>> discuss > > > > > > > > > > > > > > >>>>>>>>>>>>> this > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> alternative to see if it is > > > reasonable. > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> 3. As we are going to monitor > late > > > > > arrival > > > > > > > > > records > > > > > > > > > > > as > > > > > > > > > > > > > > >>>>>>> well, > > > > > > > > > > > > > > >>>>>>>>> they > > > > > > > > > > > > > > >>>>>>>>>>>>> would > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> create some really spiky graphs > > > when the > > > > > > > > > > > out-of-order > > > > > > > > > > > > > > >>>>>>>>> records are > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> interleaving with on time > records. > > > > > Should we > > > > > > > > > also > > > > > > > > > > > > supply > > > > > > > > > > > > > > >>>>>>> a > > > > > > > > > > > > > > >>>>>>>>> smooth > > > > > > > > > > > > > > >>>>>>>>>>>>> version > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> of the latency metrics, or user > > > should > > > > > just > > > > > > > > take > > > > > > > > > > > care > > > > > > > > > > > > of > > > > > > > > > > > > > > >>>>>>> it > > > > > > > > > > > > > > >>>>>>>>> by > > > > > > > > > > > > > > >>>>>>>>>>>>> themself? > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> 4. Regarding this new metrics, > we > > > > > haven't > > > > > > > > > > discussed > > > > > > > > > > > > its > > > > > > > > > > > > > > >>>>>>>>> relation > > > > > > > > > > > > > > >>>>>>>>>>>>> with our > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> existing processing latency > > metrics, > > > > > could > > > > > > > you > > > > > > > > > add > > > > > > > > > > > > some > > > > > > > > > > > > > > >>>>>>>>> context > > > > > > > > > > > > > > >>>>>>>>>>> on > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> comparison and a simple `when to > > use > > > > > which` > > > > > > > > > > tutorial > > > > > > > > > > > > for > > > > > > > > > > > > > > >>>>>>> the > > > > > > > > > > > > > > >>>>>>>>>>> best? > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> Boyang > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> On Tue, May 12, 2020 at 7:28 PM > > > Sophie > > > > > > > > > > Blee-Goldman > > > > > > > > > > > < > > > > > > > > > > > > > > >>>>>>>>>>>>> sop...@confluent.io > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> wrote: > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> Hey all, > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> I'd like to kick off discussion > > on > > > > > KIP-613 > > > > > > > > > which > > > > > > > > > > > aims > > > > > > > > > > > > > > >>>>>>> to > > > > > > > > > > > > > > >>>>>>>>> add > > > > > > > > > > > > > > >>>>>>>>>>>>> end-to-end > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> latency metrics to Streams. > > Please > > > > > take a > > > > > > > > look: > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> Cheers, > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> Sophie > > > > > > > > > > > > > > >>>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>>> -- > > > > > > > > > > > > > > >>>>>>>>>>>>>> -- Guozhang > > > > > > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > > > > > >>>>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > -- > -- Guozhang >
