Hello Pracheer, Storm Version 1.1.0 and higher contains a new scheduling strategy for the Resource Aware Scheduler that will only pack executors from a single topology as tightly as it can, but it will spread the scheduling of topologies across all nodes so what you described shouldn't occur anymore. You can read more about the updated scheduling strategy at:
http://storm.apache.org/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html hope this helps! Best, Jerry On Wed, Apr 12, 2017 at 9:12 AM, Bobby Evans <[email protected]> wrote: > No there is currently no way to do this with the default strategy. You can > write your own resource aware strategy to place the workers differently > though. You just need to write a class that implements IStrategy. It can > also be selected on a per topology basis. > > The main reason for doing this is because it reduces the network load > associated with a topology and hopefully speeds it up as a local transfer is > much faster than a remote one. > If you are having problems with it then it might mean that we are not > scheduling for the correct resources. One big hole that we are aware of is > network. If you need/want network as a resource then please let us know. We > should be able to at least implement something simple similar to CPU and > Memory, but a fully functional one that takes the complete network topography > into account is going to be difficult to do. > > - Bobby > > On Tuesday, April 11, 2017, 10:32:49 PM CDT, Pracheer Agarwal > <[email protected]> wrote:---------- Forwarded message ---------- > From: Pracheer Agarwal <[email protected]> > Date: Tue, Apr 11, 2017 at 9:07 AM > Subject: Distribution of workers across cluster with Resource Aware > Scheduler > To: [email protected] > > > Hi All, > > We are in the process of migrating to storm version 1.0.3. The pattern that > we observe with resource aware scheduler is that it launches workers on one > single machine till the time machine is resource wise exhausted. So few of > the machines are fully utilized while few are sitting idle. Can I tune > storm(RAS) to spread the workers across the clusters ? Are there parameters > to control this feature ? > > Regards > Pracheer
