Thanks Juntaek!
So it this is totally my fault. In my metrics consumer class I was not correctly logging the _system components so they never made it to my database. Fixing this gives me loads of useful data! Just to pick up on what you said about the Worker Receive Thread. So there is no receive queue, just Netty's input buffer and the executor's disruptor queues? Thanks again for the help. Thomas Cooper PhD Student Newcastle University, School of Computer Science W: http://www.tomcooper.org.uk | Twitter: @tomncooper<https://twitter.com/tomncooper> ________________________________ From: Jungtaek Lim <[email protected]> Sent: 26 October 2017 16:34:37 To: [email protected] Subject: Re: Worker Process Send and Receive Queues Hi Thomas, Could you find '__transfer' under '__system' component? '__system' denotes SystemBolt which will be launched for each worker, and worker transfer queue is registered to publish metrics in '__system' component. You can also get metrics from '__recv-iconnection' under '__system' component which is a set of simple metrics for Netty server. There's no explicit disruptor queue for receive queue on worker. Receive callback function will route received messages to executors' receive queues. (They're disruptor queues.) Hope this helps. Thanks, Jungtaek Lim (HeartSaVioR) 2017년 10월 26일 (목) 오후 9:12, Thomas Cooper (PGR) <[email protected]<mailto:[email protected]>>님이 작성: Hi, I'm working on modelling the performance of storm topologies and I am interested in the worker processes (WP) send / receive queues and threads. My aim is to model a proposed physical plan (WP and Executor assignment) based on performance metrics from a currently running plan. Storm provides detailed metrics for the spouts and bolts. However, to accurately model the effect of a new physical plan I need to understand the performance of the WP send and receive systems. Obviously if a new plan puts a lot of high traffic executors in a given WP I want to be able to model the effect of this potential bottleneck. As I understand it, in the v1.1.1 code base, the WP receive queue and transfer (send) queues are both instances of org.apache.storm.utils.DisruptorQueue. These like the executor receive and send queues can produce metrics. However, as far as I can tell, the WP queue metrics are not reported (I have my own metrics consumer that records all available metrics). Is this right? I assume this is because the WP queues are not part of a component with a link to a metrics consumer? My question is weather it would be possible to obtain metrics from the WP receive and transfer queues? If this possible, then also how difficult it would be to obtain them (would this entail modifying the worker class). I can infer some of the metrics via other means, the send queue arrival rates for executors within the WP for example. However, it would be more accurate to have the actual metrics. Also, I am assuming metrics on processing times for the actual receive and send threads (moving tuple off queues) will not be available as this functionality seems to be distributed over several different classes? Thanks in advance, Thomas Cooper PhD Student Newcastle University, School of Computer Science W: http://www.tomcooper.org.uk | Twitter: @tomncooper<https://twitter.com/tomncooper>
