It's not recommended to run more than one topology on the same server cluster as it can make it difficult to monitor and optimize. I usually only use one slot (JVM) per node. I remember reading somewhere where that is recommended too.
On Wed, Jul 20, 2016 at 7:22 AM, Navin Ipe <[email protected]> wrote: > From what I know, you can figure out how much memory your application > needs and allocate the memory as shown below. The below code allocates 2GiB > of memory for each worker of the topology. > > Config stormConfig = new Config(); > stormConfig.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx2g"); > > On Tue, Jul 19, 2016 at 10:20 PM, Joaquin Menchaca <[email protected]> > wrote: > >> How can one calculate how much memory is needed? >> >> On Tue, Jul 19, 2016 at 1:05 AM, Navin Ipe < >> [email protected]> wrote: >> >>> I've figured out the answer to this. A slot is used by a worker. A >>> worker is a JVM. So each JVM would require a clump of heap memory of its >>> own. So a default of 4 slots would use 4*x amount of memory, where x is the >>> memory used by a worker JVM. >>> Now obviously if you declare more than 4 ports, it'll take up that much >>> more memory. >>> The problem with taking up too much memory, is that your topologies will >>> suddenly crash with a GC overhead limit exceeded exception and the spout or >>> bolt will get re-started constantly. >>> As I understand, you'd be better off with increasing the number of >>> servers or RAM on the existing server, if you want to have many >>> workers/topologies. >>> >>> >>> On Mon, Jul 18, 2016 at 3:43 PM, Navin Ipe < >>> [email protected]> wrote: >>> >>>> Ok, if there's an answer to the first question, then anyone who knows >>>> details about Storm's design, please help in this thread. For the second >>>> question, I'll be starting a separate thread, since there would be people >>>> who'd have experience with running multiple topologies. >>>> >>>> On Mon, Jul 18, 2016 at 12:12 PM, Navin Ipe < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I was initially surprised that Storm couldn't run more than 4 >>>>> topologies by default. On increasing the number of supervisor slots, I was >>>>> able to run more topologies. But there are two things I don't understand: >>>>> >>>>> 1. Was Storm designed to support only 4 default slots because it has >>>>> to allocate memory for each slot and supporting 10 slots by default would >>>>> have eaten up too much memory? >>>>> 2. Since I have to go to each supervisor and alter each storm.yaml >>>>> file to support more than 4 slots, then if I assign 5 slots to 5 >>>>> supervisors, will I be able to run 5*5=25 topologies? (I've tried it only >>>>> on my local system until now). >>>>> >>>>> -- >>>>> Regards, >>>>> Navin >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Navin >>>> >>> >>> >>> >>> -- >>> Regards, >>> Navin >>> >> >> >> >> -- >> >> 是故勝兵先勝而後求戰，敗兵先戰而後求勝。 >> > > > > -- > Regards, > Navin > -- Brian Taylor Resolving Architecture .:. 330-812-7098 [email protected] http://resolvingarchitecture.com www.linkedin.com/in/leanenterprisearchitect/ <http://www.linkedin.com/in/javadevops/>
