>> However, I wanted to ask if you know how does Nimbus retrieve those stats in the first place. If those stats exist in each Bolt, I want to first check if I can get them, before I ask them from Nimbus.
These stats are maintained in some builtion metrices ( https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/daemon/builtin_metrics.clj ) . I am not sure if there is some API to access these stats directly. But from the TopologyContext (which is passed to the open method of spout or prepare of bolt) we may got have access to them ( https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/task/TopologyContext.java ) On Thu, Apr 23, 2015 at 6:56 PM, Nick R. Katsipoulakis < [email protected]> wrote: > Hello and thank you for your answers, > > @Sushant: Thank you for your suggestion. I will try to see if the values > make more sense with nano-time. > > @Ohidiost: Thank you for explaining the way metrics work in Nimbus. I will > try to apply your code and set up ganglia to my cluster. However, I wanted > to ask if you know how does Nimbus retrieve those stats in the first place. > If those stats exist in each Bolt, I want to first check if I can get them, > before I ask them from Nimbus. > > Thank you all again for your responses. > > Cheers, > Nick > > 2015-04-22 22:49 GMT-04:00 Ohidiost Nal <[email protected]>: > > Storm UI retrieves those stats from Nimbus. They are maintained for 10 >> minutes, 1hr, 3 hr etc. windows. The bolts or any other code can access the >> same metrics using the Nimbus Thrift API. >> I recently added a ganglia plugin to collect historical Storm topology >> stats from Nimbus (as the sliding window stats are not really very helpful >> for measuring the performance over time). >> https://github.com/ganglia/gmond_python_modules/tree/master/storm_topology >> >> That may be useful to understand how to get the process, execute latency >> etc. for Storm spout/bolt etc. >> >> On Thu, Apr 23, 2015 at 3:10 AM, Nick R. Katsipoulakis < >> [email protected]> wrote: >> >>> Hello all, >>> >>> I have been trying to calculate, as precisely as possible, how much time >>> a tuple spends waiting in the input queue of a Bolt until it is provided in >>> the BaseRichBolt.execute() function. >>> >>> The way I try to get this wait time (latency) is by adding a timestamp >>> attribute in the tuple, the time it gets produced (I call Java's >>> System.currentTimeInMillis()). When it gets picked up from a downstream >>> Bolt (call of execute() function), I retrieve the timestamp attribute and >>> calculate the difference in milliseconds from the current timestamp. >>> However, due to clock skewness in my local cluster (I guess that this is >>> the cause), I usually retrieve difference values that do not make sense >>> (sometimes even negative wait times). I have previously tried NTP but I >>> still do not get values that make sense. >>> >>> When I check the Storm UI reported metrics, I see that there two latency >>> metrics: Process latency and one more. Is there any way I can retrieve >>> those from the java code inside a Bolt's execute method? Is there a better >>> way to get a more precise latency metric from inside the execute() function >>> of a Bolt? >>> >>> Thanks, >>> Nick >>> >>> -- >>> Nikolaos Romanos Katsipoulakis, >>> University of Pittsburgh, PhD candidate >>> >> >> >> >> -- >> Regards, >> Nipun Talukdar >> Bangalore >> > > > > -- > Nikolaos Romanos Katsipoulakis, > University of Pittsburgh, PhD candidate > -- Regards, Nipun Talukdar Bangalore
