We will run one topology per customer, so trying to figure out optimal configuration. Ultimately, I can add n-number of supervisors, and then there's the option to increase number of workers per supervisor. Trying to figure out best optimization patterns. Ultimately, I am thinking the supervisors will be put into a container and spun up with Kubernetes.
For memory, I was wondering if topology made an impact, such as topology A vs. topology B. At least for different versions of a topology, I would wonder how to assess memory usage of that topology, and how to adjust it. I am guessing the data size doesn't make a difference for memory, as it just trickles through as tuples. On Thu, Jul 21, 2016 at 1:34 AM, Jungtaek Lim <kabh...@gmail.com> wrote: > That was presentation of Taylor's, and it's 1 worker per node "per > topology". > That suggestion is for reducing inter-JVM communications. So having > multiple ports for multiple topology is OK. > > 2016년 7월 21일 (목) 오후 5:11, Sinnema, Remon <remon.sinn...@emc.com>님이 작성: > >> I’m just curious, if the recommendation is to use 1 worker per node, then >> why is the default 4? >> >> >> >> *From:* Brian Taylor [mailto:br...@resolvingarchitecture.com] >> *Sent:* woensdag 20 juli 2016 16:06 >> *To:* user@storm.apache.org >> *Subject:* Re: Is there a reason storm has a default of 4 ports? >> >> >> >> It's not recommended to run more than one topology on the same server >> cluster as it can make it difficult to monitor and optimize. I usually only >> use one slot (JVM) per node. I remember reading somewhere where that is >> recommended too. >> >> >> >> On Wed, Jul 20, 2016 at 7:22 AM, Navin Ipe < >> navin....@searchlighthealth.com> wrote: >> >> From what I know, you can figure out how much memory your application >> needs and allocate the memory as shown below. The below code allocates 2GiB >> of memory for each worker of the topology. >> >> Config stormConfig = new Config(); >> stormConfig.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx2g"); >> >> >> >> On Tue, Jul 19, 2016 at 10:20 PM, Joaquin Menchaca <jmench...@gobalto.com> >> wrote: >> >> How can one calculate how much memory is needed? >> >> >> >> On Tue, Jul 19, 2016 at 1:05 AM, Navin Ipe < >> navin....@searchlighthealth.com> wrote: >> >> I've figured out the answer to this. A slot is used by a worker. A worker >> is a JVM. So each JVM would require a clump of heap memory of its own. So a >> default of 4 slots would use 4*x amount of memory, where x is the memory >> used by a worker JVM. >> >> Now obviously if you declare more than 4 ports, it'll take up that much >> more memory. >> >> The problem with taking up too much memory, is that your topologies will >> suddenly crash with a GC overhead limit exceeded exception and the spout or >> bolt will get re-started constantly. >> >> As I understand, you'd be better off with increasing the number of >> servers or RAM on the existing server, if you want to have many >> workers/topologies. >> >> >> >> >> >> On Mon, Jul 18, 2016 at 3:43 PM, Navin Ipe < >> navin....@searchlighthealth.com> wrote: >> >> Ok, if there's an answer to the first question, then anyone who knows >> details about Storm's design, please help in this thread. For the second >> question, I'll be starting a separate thread, since there would be people >> who'd have experience with running multiple topologies. >> >> >> >> On Mon, Jul 18, 2016 at 12:12 PM, Navin Ipe < >> navin....@searchlighthealth.com> wrote: >> >> Hi, >> >> I was initially surprised that Storm couldn't run more than 4 topologies >> by default. On increasing the number of supervisor slots, I was able to run >> more topologies. But there are two things I don't understand: >> >> 1. Was Storm designed to support only 4 default slots because it has to >> allocate memory for each slot and supporting 10 slots by default would have >> eaten up too much memory? >> >> 2. Since I have to go to each supervisor and alter each storm.yaml file >> to support more than 4 slots, then if I assign 5 slots to 5 supervisors, >> will I be able to run 5*5=25 topologies? (I've tried it only on my local >> system until now). >> >> >> -- >> >> Regards, >> >> Navin >> >> >> >> -- >> >> Regards, >> >> Navin >> >> >> >> -- >> >> Regards, >> >> Navin >> >> >> >> -- >> >> >> 是故勝兵先勝而後求戰，敗兵先戰而後求勝。 >> >> >> >> -- >> >> Regards, >> >> Navin >> >> >> >> >> >> -- >> >> Brian Taylor >> >> Resolving Architecture .:. >> >> 330-812-7098 >> >> br...@resolvingarchitecture.com >> >> http://resolvingarchitecture.com >> >> www.linkedin.com/in/leanenterprisearchitect/ >> <http://www.linkedin.com/in/javadevops/> >> > -- 是故勝兵先勝而後求戰，敗兵先戰而後求勝。
