Yes that is basically right. If indeed your executors are processing enough data to be utilizing nearly all of cpu then you should consider 1 executor per core. May be a good idea to budget 10-20% cpu per machine for other stuff including interactive login.
Sent from Yahoo Mail for iPhone On Tuesday, April 9, 2019, 2:25 PM, Jayant Sharma <[email protected]> wrote: Hi, I was going through P. Taylor Goetz's talk on scaling storm cluster (2014). He made an interesting point of computing available parallelism units in a cluster. For CPU bound applications this limit was 1 executor/CPU core. I just wanted to be sure my understanding about this is clear: If I have 3 supervisor machines, each having 5 workers or JVMs and each machine has 16 CPU cores. I have 3*16 = 48 parallelism units to distribute among all of my topologies? Which means the sum of all spouts and bolts executors across all the topologies should be 48. What are the implications if I keep my executors more or less than this value? If my understanding is incorrect, can someone please explain how to compute parallelism units and relate them to number of executors. Thanks,Jayant Sharma
