Thanks Satish, but that didn't work either. Although 4 of the topologies got 1088MB assigned, the 5th topology got 0MB.
Num workers Num executors Assigned Mem (MB) 1 12 1088 1 4 1088 1 27 1088 1 8 1088 0 0 0 On Wed, Jul 13, 2016 at 3:00 PM, Satish Duggana <[email protected]> wrote: > Config.TOPOLOGY_WORKER_CHILDOPTS: Options which can override > WORKER_CHILDOPTS for a topology. You can configure any java options like > memory, gc etc > > In your case it can be > config.put(Config.TOPOLOGY_WORKER_CHILDOPTS, "-Xmx1g"); > > Thanks, > Satish. > > > On Wed, Jul 13, 2016 at 1:45 PM, Navin Ipe < > [email protected]> wrote: > >> Using *stormConfig.put("topology.**worker.childopts",1024)* gave me this >> error: >> *java.lang.IllegalArgumentException: Field TOPOLOGY_WORKER_CHILDOPTS must >> be an Iterable but was a class java.lang.Integer* >> >> A bit of looking around showed me that this might be the right syntax: >> *config.put(Config.TOPOLOGY_WORKER_CHILDOPTS, >> SOME_OPTS);* >> >> But couldn't find any example on what Storm expects as SOME_OPTS. >> >> On Wed, Jul 13, 2016 at 12:38 PM, Spico Florin <[email protected]> >> wrote: >> >>> Hello! >>> For the the topology that you have 0MB allocated, for me it seems that >>> you don't have enough slots available. Check out the storm.yaml file (on >>> your worker machines) how many slots you have allocated. >>> (by default the are 4 slots available supervisor.slots.ports: >>> - 6700 >>> - 6701 >>> - 6702 >>> - 6703) You have 5 topologies, therefore one is not ran. >>> >>> Regarding the memory allocation, you allocate memory per each worker >>> (slot available), not per topology. If you set up for your topology a >>> number of workers equal to 1, then you topology will run on a single worker >>> (available slot) and will receive the configuration that you gave for your >>> worker. If you configure to spread your spout and bolts to multiple >>> workers,(that are available in as configured slots) then all the workers >>> will receive the same amount of memory configured globally via >>> worker.childopts property in the storm yaml . So you don't configure the >>> meory per topology but per worker. >>> >>> If you want use different memory allocation for workers depending on >>> your topology components memory consumption, then you should use the >>> property >>> >>> stormConfig.put("topology.worker.childopts",1024) >>> >>> I hope it helps. >>> >>> Regards, >>> Florin >>> >>> On Wed, Jul 13, 2016 at 9:23 AM, Navin Ipe < >>> [email protected]> wrote: >>> >>>> I tried setting stormConfig.put(Config.WORKER_HEAP_MEMORY_MB, 1024); >>>> and now *all topologies* Assigned memory is 0MB. >>>> Does anyone know why it works this way? How do we assign memory to >>>> topologies in this situation? >>>> >>>> On Wed, Jul 13, 2016 at 10:39 AM, Navin Ipe < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have a program *MyProg.java* inside which I'm creating 5 topologies >>>>> and using *stormSubmitter* to submit it to Storm. The storm UI shows >>>>> the assigned memory for each topology as: >>>>> *Assigned Mem (MB)* >>>>> 832 >>>>> 0 >>>>> 832 >>>>> 832 >>>>> 832 >>>>> >>>>> One of the topologies was assigned 0MB. Assuming that it was because >>>>> of setting a single Config for all of them, I gave them separate configs >>>>> (as below), but nothing changed. >>>>> >>>>> *StormConfigFactory stormConfigFactory = new StormConfigFactory();* >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *StormSubmitter.submitTopology(upTopologyName, >>>>> stormConfigFactory.GetNewConfig(), >>>>> upBuilder.createTopology());StormSubmitter.submitTopology(viTopologyName, >>>>> stormConfigFactory.GetNewConfig(), >>>>> viBuilder.createTopology());StormSubmitter.submitTopology(prTopologyName, >>>>> stormConfigFactory.GetNewConfig(), >>>>> prBuilder.createTopology());StormSubmitter.submitTopology(opTopologyName, >>>>> stormConfigFactory.GetNewConfig(), >>>>> opBuilder.createTopology());StormSubmitter.submitTopology(poTopologyName, >>>>> stormConfigFactory.GetNewConfig(), poBuilder.createTopology());* >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *And the StormConfigFactory class:public class StormConfigFactory { >>>>> public Config GetNewConfig() { Config stormConfig = new >>>>> Config(); stormConfig.setNumWorkers(1); >>>>> stormConfig.setNumAckers(1); >>>>> stormConfig.put(Config.TOPOLOGY_DEBUG, false); >>>>> stormConfig.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE, 64); >>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, >>>>> 65536); >>>>> stormConfig.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, >>>>> 65536); stormConfig.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, >>>>> 50); stormConfig.put(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS, >>>>> 60); stormConfig.put(Config.STORM_ZOOKEEPER_SERVERS, >>>>> Arrays.asList(new String[]{"localhost"})); >>>>> return stormConfig; }}* >>>>> >>>>> >>>>> *How do I allocate separate memory and workers for each topology?* >>>>> >>>>> -- >>>>> Regards, >>>>> Navin >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Navin >>>> >>> >>> >> >> >> -- >> Regards, >> Navin >> > > -- Regards, Navin
