Can you provide a jstack dump of all the threads in one worker ? -roshan
On 3/16/17, 8:40 PM, "S G" <[email protected]> wrote: I did not say 3 thds per spout/bolt. I said "When 60 bolts are run per node, there are 1200 total threads/node" That gets me 20 threads/bolt. And I am not creating any new threads inside my bolts. On Thu, Mar 16, 2017 at 6:12 PM, Roshan Naik <[email protected]> wrote: > I assume you mean ..3 thds per spout/bolt is on the high side ? > Currently there is an executor thd, a xsfer thd and flusher thd. The plan > (in the redesign – STORM-2284 subtask 2) is to have 1 thread per spout/bolt. > Also have some (untested) thoughts on reducing the number of remaining > threads in the worker. > -roshan > > > On 3/16/17, 5:07 PM, "S G" <[email protected]> wrote: > > Thanks for sharing the doc Roshan. > It is very informative. > > I think number of threads per bolt is on the high side (When 60 bolts > are > run per node, there are 1200 total threads/node). > > Some of these would be essential for the worker's book-keeping but > still it > seems we can get a lot of performance boost if we can somehow reduce > these > threads and the associated context switching between them. > > > On Thu, Mar 16, 2017 at 12:53 PM, Roshan Naik <[email protected]> > wrote: > > > Typically there are 3 threads per spout or bolt instance (unless the > > spout or bolt is spawning its own threads). There are the acker > bolts, > > event loggers and system bolt running there too. > > > > Then there are several more per worker. See a summary of it in > section 3.1 > > of this document > > > > https://docs.google.com/document/d/1EzeHL3d7EE- > > RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing > > > > > > > > > > Get Outlook for iOS<https://aka.ms/o0ukef> > > > > > > > > > > On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" < > [email protected]< > > mailto:[email protected]>> wrote: > > > > Hi, > > > > I am trying to make sense of number of threads seen in JConsole. > > It seems like a very high number of threads are launched per bolt > thread. > > > > > > [cid:ii_j0crtbtm0_15ad8874d0924ae2] > > > > Experiment 1 > > topology.workers.val=16 > > spout.parallelism.val=1 > > bolt1.parallelism.val=900 > > bolt2.parallelism.val=160 > > observation: > > Threads seen per node in jconsole = 1200 > > Bolt threads per node = (900 + 160)/16 = 66 > > Threads per bolt = 1200/66 = 18 > > > > > > Experiment 2 > > topology.workers.val=16 > > spout.parallelism.val=1 > > bolt1.parallelism.val=16 > > bolt2.parallelism.val=16 > > observation: > > Threads seen per node in jconsole = 61 > > Bolt threads per node = (16+16)/16 = 2 > > Threads per bolt = 61/2 = 30 > > > > There are no other topologies running in my cluster. > > There are no other spouts/bolts running in my cluster except the ones > > mentioned above. > > I am running only one worker process per machine. > > > > > > So the question is how many threads per bolt are launched by storm? > > I am not interested in the exact number, but concerned about the high > > number of extra threads (18+ thread) for running a single bolt. > > > > Can we limit or optimize it somehow? > > Or if all of them are required, it would be good to document them > > somewhere. > > > > Thanks > > SG > > > > > > > > > > >
