Storm UI retrieves those stats from Nimbus. They are maintained for 10 minutes, 1hr, 3 hr etc. windows. The bolts or any other code can access the same metrics using the Nimbus Thrift API. I recently added a ganglia plugin to collect historical Storm topology stats from Nimbus (as the sliding window stats are not really very helpful for measuring the performance over time). https://github.com/ganglia/gmond_python_modules/tree/master/storm_topology That may be useful to understand how to get the process, execute latency etc. for Storm spout/bolt etc.
On Thu, Apr 23, 2015 at 3:10 AM, Nick R. Katsipoulakis < [email protected]> wrote: > Hello all, > > I have been trying to calculate, as precisely as possible, how much time a > tuple spends waiting in the input queue of a Bolt until it is provided in > the BaseRichBolt.execute() function. > > The way I try to get this wait time (latency) is by adding a timestamp > attribute in the tuple, the time it gets produced (I call Java's > System.currentTimeInMillis()). When it gets picked up from a downstream > Bolt (call of execute() function), I retrieve the timestamp attribute and > calculate the difference in milliseconds from the current timestamp. > However, due to clock skewness in my local cluster (I guess that this is > the cause), I usually retrieve difference values that do not make sense > (sometimes even negative wait times). I have previously tried NTP but I > still do not get values that make sense. > > When I check the Storm UI reported metrics, I see that there two latency > metrics: Process latency and one more. Is there any way I can retrieve > those from the java code inside a Bolt's execute method? Is there a better > way to get a more precise latency metric from inside the execute() function > of a Bolt? > > Thanks, > Nick > > -- > Nikolaos Romanos Katsipoulakis, > University of Pittsburgh, PhD candidate > -- Regards, Nipun Talukdar Bangalore
