If your topology is the only topology running on 20 node cluster, then I think you can reduce slots per supervisor to "1" by setting "supervisor.slots.ports" (mention just 1 port number there) in storm.yaml. If you are running multiple topologies on this 20 node cluster then this solution may not work, need to think of something else like writing our own meta-data aware custom scheduler by implementing backtype.storm.scheduler.IScheduler.
But I think there can be some scope for refactoring that HishTensionBolt in order to reduce memory usage. All the memory used by that bolt remains static for every tuple? Or it's "execute()" method which consumes that much memory each time and discards it once tuple processing is done? On 21 April 2016 at 06:47, <[email protected]> wrote: > > > Hi, > > > > I have setup 4 workers on each machine. In my Topology, there is one bolt > which needs a lot of memory, so ideally, I don’t want it to schedule more > than 1 of that on any machines. In terms of computation it is pretty fast > so it can manage good throughput when running. Lets call it, > BoltHighTension. > > But, my other bolts are very light weight and I can have a lot of > parallelism on that. > > > > How do I ensure that if I have 20 Supervisors, I don’t have more than 1 > ‘BoltHighTension’ on each machine? I want to give parallelism hint of 20 to > this bolt. > > But, I notice that sometimes, more than 1 such instance gets allocated on > same machine. (Machine can handle 2, but the performance due to paging > becomes a problem). > > > > Thanks for your help/advice/hints. > > > > Thanks > > -Abhishek > > > > Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for > Windows 10 > > > > >
