So you are seeing 65MB on UI. UI only shows assigned memory, not memory usage.
As I mentioned earlier, -Xmx%HEAP-MEM%m in worker.childopts is designed to be replaced with the total memory assigned for the worker. ( https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java#L392 <https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java#L392>). Your config "worker.childopts: "-Xmx2048m -XX:+PrintGCDetails” will make every worker use 2GB. But it will not show up on UI as 2GB because UI doesn’t read “-Xmx2048m” from worker.childopts. The assigned memory on UI is the sum of the assigned memory of all the executors in the worker. The amount of assigned memory for a worker depends on how it’s scheduled and what the executors in the worker are. For example, by default, every instance/executor is configured with 128MB memory (https://github.com/apache/storm/blob/1.x-branch/conf/defaults.yaml#L276 <https://github.com/apache/storm/blob/1.x-branch/conf/defaults.yaml#L276>). If 4 executors are scheduled in one worker, then the assigned memory for that worker is 512MB. Hope that helps. > On Feb 17, 2020, at 8:55 AM, Narasimhan Chengalvarayan > <[email protected]> wrote: > > Hi Ethan Li, > > > Sorry for the late reply. Please find the output where it is showing > -Xmx2048m for worker heap . But in storm ui we are seeing the worker > allocated memory as 65MB for each worker. > > java -server -Dlogging.sensitivity=S3 -Dlogfile.name=worker.log > -Dstorm.home=/opt/storm/apache-storm-1.2.1 > -Dworkers.artifacts=/var/log/storm/workers-artifacts > -Dstorm.id=Topology_334348-43-1580365369 > -Dworker.id=f1e3e060-0b32-4ecd-8c34-c486258264a4 -Dworker.port=6707 > -Dstorm.log.dir=/var/log/storm > -Dlog4j.configurationFile=/opt/storm/apache-storm-1.2.1/log4j2/worker.xml > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > -Dstorm.local.dir=/var/log/storm/tmp -Xmx2048m -XX:+PrintGCDetails > -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 > -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=artifacts/heapdump > -Djava.library.path=/var/log/storm/tmp/supervisor/stormdist/Topology_334348-43-1580365369/resources/Linux-amd64:/var/log/storm/tmp/supervisor/stormdist/Topology_334348-43-1580365369/resources:/usr/local/lib:/opt/local/lib:/usr/lib > -Dstorm.conf.file= -Dstorm.options= > -Djava.io.tmpdir=/var/log/storm/tmp/workers/f1e3e060-0b32-4ecd-8c34-c486258264a4/tmp > -cp > /opt/storm/apache-storm-1.2.1/lib/*:/opt/storm/apache-storm-1.2.1/extlib/*:/opt/storm/apache-storm-1.2.1/conf:/var/log/storm/tmp/supervisor/stormdist/Topology_334348-43-1580365369/stormjar.jar > org.apache.storm.daemon.worker Topology_334348-43-1580365369 > 7fe05c2b-ebcf-491b-a8cc-2565834b5988 6707 > f1e3e060-0b32-4ecd-8c34-c486258264a4 > > On Tue, 4 Feb 2020 at 04:36, Ethan Li <[email protected]> wrote: >> >> This is where the worker launch command is composed: >> >> https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java#L653-L671 >> >> Since your worker.childopts is set, and topology.worker.childopts is empty, >> >> >> worker.childopts: "-Xmx2048m -XX:+PrintGCDetails >> -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps >> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 >> -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError >> -XX:HeapDumpPath=artifacts/heapdump” >> >> >> The command to launch the worker process should have -Xmx2048m. >> >> I don’t see why it would be 65MB. And what do you mean by coming as 65MB >> only? Is it only committed 65MB? Or is the max only 65MB? >> >> Could you submit the topology and show the result of “ps -aux |grep >> --ignore-case worker”? This will show you the JVM parameters of the worker >> process. >> >> >> (BTW, -Xmx%HEAP-MEM%m in worker.childopts is designed to be replaced >> https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java#L392) >> >> >> >> >> On Jan 30, 2020, at 2:12 AM, Narasimhan Chengalvarayan >> <[email protected]> wrote: >> >> Hi Ethan, >> >> >> Please find the configuration detail >> >> ********************************************** >> >> #Licensed to the Apache Software Foundation (ASF) under one >> # or more contributor license agreements. See the NOTICE file >> # distributed with this work for additional information >> # regarding copyright ownership. The ASF licenses this file >> # to you under the Apache License, Version 2.0 (the >> # "License"); you may not use this file except in compliance >> # with the License. You may obtain a copy of the License at >> # >> # http://www.apache.org/licenses/LICENSE-2.0 >> # >> # Unless required by applicable law or agreed to in writing, software >> # distributed under the License is distributed on an "AS IS" BASIS, >> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >> # See the License for the specific language governing permissions and >> # limitations under the License. >> >> ########### These MUST be filled in for a storm configuration >> storm.zookeeper.servers: >> - "ZK1" >> - "ZK2" >> - "ZK3" >> >> nimbus.seeds: ["host1","host2"] >> ui.port : 8081 >> storm.log.dir: "/var/log/storm" >> storm.local.dir: "/var/log/storm/tmp" >> supervisor.slots.ports: >> - 6700 >> - 6701 >> - 6702 >> - 6703 >> - 6704 >> - 6705 >> - 6706 >> - 6707 >> - 6708 >> - 6709 >> - 6710 >> - 6711 >> - 6712 >> - 6713 >> - 6714 >> - 6715 >> - 6716 >> - 6717 >> worker.heap.memory.mb: 1639 >> topology.worker.max.heap.size.mb: 1639 >> worker.childopts: "-Xmx2048m -XX:+PrintGCDetails >> -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps >> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 >> -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError >> -XX:HeapDumpPath=artifacts/heapdump" >> worker.gc.childopts: "" >> >> topology.min.replication.count: 2 >> # >> # >> # ##### These may optionally be filled in: >> # >> ## List of custom serializations >> # topology.kryo.register: >> # - org.mycompany.MyType >> # - org.mycompany.MyType2: org.mycompany.MyType2Serializer >> # >> ## List of custom kryo decorators >> # topology.kryo.decorators: >> # - org.mycompany.MyDecorator >> # >> ## Locations of the drpc servers >> # drpc.servers: >> # - "server1" >> # - "server2" >> >> ## Metrics Consumers >> # topology.metrics.consumer.register: >> # - class: "org.apache.storm.metric.LoggingMetricsConsumer" >> # parallelism.hint: 1 >> # - class: "org.mycompany.MyMetricsConsumer" >> # parallelism.hint: 1 >> # argument: >> # - endpoint: "metrics-collector.mycompany.org" >> >> ********************************************************************************************* >> >> >> On Thu, 30 Jan 2020 at 03:07, Ethan Li <[email protected]> wrote: >> >> >> I am not sure. Can you provide your configs? >> >> >> >> On Jan 28, 2020, at 6:33 PM, Narasimhan Chengalvarayan >> <[email protected]> wrote: >> >> Hi Team, >> >> Do you have any idea, In storm apache 1.1.0 we have set worker size as >> 2 GB , Once we upgrade to 1.2.1 .It was coming as 65MB only. please >> help us .DO we need to follow different configuration setting for >> storm 1.2.1 or it is a bug. >> >> On Mon, 27 Jan 2020 at 16:44, Narasimhan Chengalvarayan >> <[email protected]> wrote: >> >> >> Hi Team, >> >> In storm 1.2.1 version, worker memory is showing as 65MB. But we have >> set worker memory has 2GB. >> >> On Fri, 24 Jan 2020 at 01:25, Ethan Li <[email protected]> wrote: >> >> >> >> 1) What is stored in Workerbeats znode? >> >> >> Worker periodically sends heartbeat to zookeeper under workerbeats node. >> >> 2) Which settings control the frequency of workerbeats update >> >> >> >> https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/Config.java#L1534-L1539 >> task.heartbeat.frequency.secs Default to 3 >> >> 3)What will be the impact if the frequency is reduced >> >> >> Nimbus get the worker status from workerbeat znode to know if executors on >> workers are alive or not. >> https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/Config.java#L595-L601 >> If heartbeat exceeds nimbus.task.timeout.secs (default to 30), nimbus will >> think the certain executor is dead and try to reschedule. >> >> To reduce the issue on zookeeper, a pacemaker component was introduced. >> https://github.com/apache/storm/blob/master/docs/Pacemaker.md >> You might want to use it too. >> >> Thanks >> >> >> On Dec 10, 2019, at 4:36 PM, Surajeet Dev <[email protected]> wrote: >> >> We upgraded Storm version to 1.2.1 , and since then have been consistently >> observing Zookeeper session timeouts . >> >> On analysis , we observed that there is high frequency of updates on >> workerbeats znode with data upto size of 50KB. This causes the Garbage >> Collector to kickoff lasting more than 15 secs , resulting in Zookeper >> session timeout >> >> I understand , increasing the session timeout will alleviate the issue , but >> we have already done that twice >> >> My questions are: >> >> 1) What is stored in Workerbeats znode? >> 2) Which settings control the frequency of workerbeats update >> 3)What will be the impact if the frequency is reduced >> >> >> >> >> >> -- >> Thanks >> C.Narasimhan >> 09739123245 >> >> >> >> >> -- >> Thanks >> C.Narasimhan >> 09739123245 >> >> >> >> >> -- >> Thanks >> C.Narasimhan >> 09739123245 >> >> > > > -- > Thanks > C.Narasimhan > 09739123245
