There is only one thread associated with an input queue and that is the one the executor runs on. The output queue has a separate thread that all it does is pop messages off of the output queue and routes them to where they need to be. In my experience this runs fairly infrequently. I would be curious to see if it would benefit from being pined to the same core as the executor thread, as it would likely improve cache hits for the data in the queue, but if the spout/bolt is really busy it would compete with the other thread for CPU resources. The output queue itself is created here https://github.com/apache/storm/blob/v1.0.1/storm-core/src/clj/org/apache/storm/daemon/executor.clj#L230-L236
The thread is created here https://github.com/apache/storm/blob/v1.0.1/storm-core/src/clj/org/apache/storm/daemon/executor.clj#L297-L311 The thread does not really run much user generated code so trying to find a simple place to insert the pinning without modifying storm would be difficult. You do have access to the storm-conf, which is the topology config, and the executor-data to help you pick a core to pin it to there, and run similar code when the bolt/spout thread is created https://github.com/apache/storm/blob/v1.0.1/storm-core/src/clj/org/apache/storm/daemon/executor.clj#L547 and https://github.com/apache/storm/blob/v1.0.1/storm-core/src/clj/org/apache/storm/daemon/executor.clj#L743 - Bobby On Thursday, June 23, 2016 3:32 AM, #ZHANG SHUHAO# <[email protected]> wrote: Hi! Thanks for your pointers! I want to make sure the I/O message queue also being pined together with the corresponding executor, is it possible to do in Storm? Follow Bobby's "easier solution" suggestion, in a machine with multiple CPU sockets, if I pin the executor thread to one CPU socket in the open method, what would happen to its input&output buffer (the disruptor queue)? Would it being "pined"? That is to ask, "who" creates the internal buffers and when? Any helps are appreciated. Thank you! Tony. > On 21 Jun 2016, at 10:16 PM, Bobby Evans <[email protected]> wrote: > > If you really want to do this you should be able to do this is the open > method of your spout or in the prepare method of the bolt. They execute on > the same thread so from that you should be able to do what you need to do. > As for the clojure in executor.clj we use the async-loop macro for creating > the task threads. > > https://github.com/apache/storm/blob/v1.0.2/storm-core/src/clj/org/apache/storm/util.clj#L472-L514 > > and on line 479 you can see the clojure equivalent of `new Thread(...)` > `(Thread. ...)` - Bobby > > On Tuesday, June 21, 2016 9:09 AM, #ZHANG SHUHAO# <[email protected]> >wrote: > > > Hi all, > > I want to manage the binding of Storm executor thread (NOT worker process) to > CPU cores (NOT host nodes). > I assume if I would know how the executor thread are initiated, I can manage > the binding of it by relying on things like JNI. > I have tried my best to look into the related source code of Storm. > Unfortunately, I am very new to Clojure, and I don't get how the executor > thread are initiated, in particular, I'm looking for something like "Thread > thread = new Thread(obj);", which is one of the most common way of creating > threads in Java. > > Any helps are appreciated. Thank you! > > Best regards. > Shuhao Zhang (Tony) > +65-86166722 > >
