I’m just curious, if the recommendation is to use 1 worker per node, then why is the default 4?
From: Brian Taylor [mailto:[email protected]] Sent: woensdag 20 juli 2016 16:06 To: [email protected] Subject: Re: Is there a reason storm has a default of 4 ports? It's not recommended to run more than one topology on the same server cluster as it can make it difficult to monitor and optimize. I usually only use one slot (JVM) per node. I remember reading somewhere where that is recommended too. On Wed, Jul 20, 2016 at 7:22 AM, Navin Ipe <[email protected]<mailto:[email protected]>> wrote: From what I know, you can figure out how much memory your application needs and allocate the memory as shown below. The below code allocates 2GiB of memory for each worker of the topology. Config stormConfig = new Config(); stormConfig.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx2g"); On Tue, Jul 19, 2016 at 10:20 PM, Joaquin Menchaca <[email protected]<mailto:[email protected]>> wrote: How can one calculate how much memory is needed? On Tue, Jul 19, 2016 at 1:05 AM, Navin Ipe <[email protected]<mailto:[email protected]>> wrote: I've figured out the answer to this. A slot is used by a worker. A worker is a JVM. So each JVM would require a clump of heap memory of its own. So a default of 4 slots would use 4*x amount of memory, where x is the memory used by a worker JVM. Now obviously if you declare more than 4 ports, it'll take up that much more memory. The problem with taking up too much memory, is that your topologies will suddenly crash with a GC overhead limit exceeded exception and the spout or bolt will get re-started constantly. As I understand, you'd be better off with increasing the number of servers or RAM on the existing server, if you want to have many workers/topologies. On Mon, Jul 18, 2016 at 3:43 PM, Navin Ipe <[email protected]<mailto:[email protected]>> wrote: Ok, if there's an answer to the first question, then anyone who knows details about Storm's design, please help in this thread. For the second question, I'll be starting a separate thread, since there would be people who'd have experience with running multiple topologies. On Mon, Jul 18, 2016 at 12:12 PM, Navin Ipe <[email protected]<mailto:[email protected]>> wrote: Hi, I was initially surprised that Storm couldn't run more than 4 topologies by default. On increasing the number of supervisor slots, I was able to run more topologies. But there are two things I don't understand: 1. Was Storm designed to support only 4 default slots because it has to allocate memory for each slot and supporting 10 slots by default would have eaten up too much memory? 2. Since I have to go to each supervisor and alter each storm.yaml file to support more than 4 slots, then if I assign 5 slots to 5 supervisors, will I be able to run 5*5=25 topologies? (I've tried it only on my local system until now). -- Regards, Navin -- Regards, Navin -- Regards, Navin -- 是故勝兵先勝而後求戰，敗兵先戰而後求勝。 -- Regards, Navin -- Brian Taylor Resolving Architecture .:. 330-812-7098 [email protected]<mailto:[email protected]> http://resolvingarchitecture.com<http://resolvingarchitecture.com/> www.linkedin.com/in/leanenterprisearchitect/<http://www.linkedin.com/in/javadevops/>
