Shadi, you are absolutely right. I created a diff ( https://reviews.apache.org/r/36274/) to properly calculate activeMs. Feel free to leave any comments there. Thanks for pointing this out :) ᐧ
On Fri, Jun 26, 2015 at 9:40 AM, Shadi Noghabi < snogh...@linkedin.com.invalid> wrote: > > Thanks for your prompt reply. I just don’t see when will there be an idle > time that is not added to the activeMs but is counted in the totalMs. > Because as far as I can tell, if one of the process,window, or commit > takes more time it will be reflected in the activeMs value as well. > > > > On 6/25/15, 7:47 PM, "Luis Fernando De Pombo" <lfdepo...@gmail.com> wrote: > > >Hi Shadi, > >Thanks for asking. This metric tracks the utilization of the event loop > >< > https://samza.apache.org/learn/documentation/0.8/container/event-loop.htm > >l> > >within a samza container, which uses a single thread, that is in charge of > >reading and writing messages, flushing metrics, checkpointing, and > >windowing. It is important to track the utilization (aka "duty cycle") of > >any event loop, which is the sum of all the timings (activeMs) divided by > >the window length (totalMs). > > > >You are right in that most of the time this value will be close to 1, > >which > >represents complete utilization. However when the event loop starts to > >have > >idle time, this metric will give you an idea of how much headroom you have > >before the job will start to seriously fall behind. > > > >I hope that answers your question! > > > >ᐧ > > > >On Thu, Jun 25, 2015 at 6:39 PM, Shadi Noghabi < > >snogh...@linkedin.com.invalid> wrote: > > > >> Hi, > >> > >> I was wondering what does this utilization metric in the > >> SamzaContainerMetrics show? I am asking this sine in the code it is > >> calculated as below: > >> > >> while (!shutdownNow) { > >> val loopStartTime = clock(); > >> process > >> window > >> commit > >> val totalMs = clock() - loopStartTime > >> metrics.utilization.set(activeMs.toFloat/totalMs) > >> activeMs = 0L > >> } > >> > >> Where the totalMs is the time it takes to start calling process until > >> commit is done which is equal to the time it takes to run process, > >>window, > >> and commit. And they way activeMs is calculated is by summing up the > >>time > >> it takes to call process, window and commit, which means these two > >>values > >> are going to be almost the same and the utilization is always going to > >>be > >> almost 1. > >> > >> I was just wondering what is the point of doing this? > >> > >> > >> > >> > >
