I will agree with Javier, one JVM per node should eliminate the number of messages that need to be serialized.
For tuning Storm topologies you may find the following links useful: https://gist.github.com/mrflip/5958028 https://wassermelonemann.wordpress.com/2014/01/22/tuning-storm-topologies/ Talk: http://demo.ooyala.com/player.html?width=640&height=360&embedCode=Q1eXg5NzpKqUUzBm5WTIb6bXuiWHrRMi&videoPcode=9waHc6zKpbJKt9byfS7l4O4sn7Qn Cheers, Michael @mvogiatzis <https://twitter.com/mvogiatzis> On Sat, 3 Oct 2015 at 14:04 Javier Gonzalez <[email protected]> wrote: > I would suggest sticking with a single worker per machine. It makes memory > allocation easier and it makes inter-component communication much more > efficient. Configure the executors with your parallelism hints to take > advantage of all your availabe CPU cores. > > Regards, > JG > > On Sat, Oct 3, 2015 at 12:10 AM, Kashyap Mhaisekar <[email protected]> > wrote: > >> Hi, >> I was trying to come up with an approach to evaluate the parallelism >> needed for a topology. >> >> Assuming I have 5 machines with 8 cores and 32 gb. And my topology has >> one spout and 5 bolts. >> >> 1. Define one worker port per CPU to start off. (= 8 workers per machine >> ie 40 workers over all) >> 2. Each worker spawns one executor per component per worker, it >> translates to 6 executors per worker which is 40x6= 240 executors. >> 3. Of this, if the bolt logic is CPU intensive, then leave parallelism >> hint at 40 (total workers), else increase parallelism hint beyond 40 till >> you hit a number beyond which there is no more visible performance. >> >> Does this look right? >> >> Thanks >> Kashyap >> > > > > -- > Javier González Nicolini > -- Michael Vogiatzis Twitter: @mvogiatzis http://micvog.com/
