Hmm... if we somehow mark a tuple as a "test" tuple or a "pilot" tuple, I think we can do the same thing on it with the END_WINDOW tuple to calculate the latency.
However, if there are many such tuples, it will for sure incur significant cost and affect the latency itself, kinda like the Heisenberg's principle -- you can't accurately measure an object because doing so will alter the object itself. David On Tue, Nov 17, 2015 at 4:12 PM, Ganelin, Ilya <[email protected]> wrote: > I totally agree - I think any such functionality should be a configurable > setting, and should not necessarily apply to every single tuple but > perhaps be periodically computed for “test” tuples flowing through the > system. > > > On 11/17/15, 4:10 PM, "Pramod Immaneni" <[email protected]> wrote: > > >I am wondering if accounting latency at a tuple level will add too much > >overhead. It might be better to do it on a conditional basis. > > > >On Tue, Nov 17, 2015 at 3:48 PM, Ganelin, Ilya > ><[email protected]> > >wrote: > > > >> Specifically, I¹m interested in being able to track latencies for > >> individual tuples. For example, as they move through the BufferServer or > >> the CircularBuffer. > >> > >> On 11/17/15, 3:12 PM, "David Yan" <[email protected]> wrote: > >> > >> >Hi Ilya, > >> > > >> >The AutoMetric interface is mostly designed for application-specific > >> >metrics within an operator -- things like how many frauds are > >>detected, or > >> >the dollar amount processed. > >> > > >> >We do have BufferServer bytes and queue tuple count (a.k.a. queue > >>size) as > >> >system metrics. If you have specific metrics that pertain to > >>BufferServer > >> >and CircularBuffer that you're interested in, please state them. We > >> >should > >> >probably add them to system metrics since it's probably useful in > >>general. > >> > > >> >David > >> > > >> >On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni > >><[email protected]> > >> >wrote: > >> > > >> >> Some of them are available in system stats like port queue size and > >>are > >> >> exposed via REST and to a stats listener that you can register via > >>the > >> >>API. > >> >> Some of the others you mentioned can be added. > >> >> > >> >> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya < > >> >> [email protected]> > >> >> wrote: > >> >> > >> >> > Hi, all presently Apex supports surfacing metrics via the > >> >>AutoMetrics > >> >> > interface, documented here: > >> >> > http://docs.datatorrent.com/autometrics/ > >> >> > > >> >> > However, this only provides utility to monitor operators. I would > >> >>like to > >> >> > provide a more general latency suite - fFor example, I would like > >>to > >> >> > instrument metrics around the BufferServer or the CircularQueue. > >>Not > >> >> > everything in Apex is an operator, and there are other components > >>of > >> >>the > >> >> > system around which we may want explicit measurements or metrics. > >>Is > >> >>this > >> >> > something that could exist? I know other projects like Spark have > >> >>similar > >> >> > efforts, although they are equally underdeveloped :) > >> >> > ________________________________________________________ > >> >> > > >> >> > The information contained in this e-mail is confidential and/or > >> >> > proprietary to Capital One and/or its affiliates and may only be > >>used > >> >> > solely in performance of work or services for Capital One. The > >> >> information > >> >> > transmitted herewith is intended only for use by the individual or > >> >>entity > >> >> > to which it is addressed. If the reader of this message is not the > >> >> intended > >> >> > recipient, you are hereby notified that any review, retransmission, > >> >> > dissemination, distribution, copying or other use of, or taking of > >>any > >> >> > action in reliance upon this information is strictly prohibited. If > >> >>you > >> >> > have received this communication in error, please contact the > >>sender > >> >>and > >> >> > delete the material from your computer. > >> >> > > >> >> > >> > >> ________________________________________________________ > >> > >> The information contained in this e-mail is confidential and/or > >> proprietary to Capital One and/or its affiliates and may only be used > >> solely in performance of work or services for Capital One. The > >>information > >> transmitted herewith is intended only for use by the individual or > >>entity > >> to which it is addressed. If the reader of this message is not the > >>intended > >> recipient, you are hereby notified that any review, retransmission, > >> dissemination, distribution, copying or other use of, or taking of any > >> action in reliance upon this information is strictly prohibited. If you > >> have received this communication in error, please contact the sender and > >> delete the material from your computer. > >> > >> > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
