I also observed that we do need more CPU power since 2.x versions. Try increasing the supervisor CPU and its service memory/RAM. Since you are giving 20Gb XmX for workers, we need supervisors to be in a much beefier state to provide that heap. We are a streaming app and for us, the supervisor CPU utilization can go close to 16 CPU cores+.
On Mon, Aug 28, 2023 at 5:13 PM v.s kadu <virajkadu...@gmail.com> wrote: > Hi All, > I have figured out a way to deal with this. > Just need to increase the parallelism hint of the component which is > taking longer to process or it's Capacity is near to one in storm ui. This > helped me to generate files faster. > > However another question which I have is regarding CPU utilization. > I have executed my topologies with storm 0.9.6 and 2.4.0. > Storm 0.9.6 uses ~30 % of 16 core RHEL 8 server to process. > However storm 2.4.0 uses ~65% to process the same . > The difference is huge. > Please note we are emitting 1000 records from Spout and this is coded in > java. > Storm configurations used in both versions > > topology.max.spout.pending:200 > topology.message.timeout.secs:300 > storm.messaging transport:"backtype.storm.messaging.netty.Context" > nimbus.task.timeout.secs:240 > nimbus.thrift.max_buffer_size:20480000 > nimbus.childopts:"-Xmx2048m -Djava.net.preferIPv4Stack=true" > > worker. childopts:"-Xmx20g -XX:+UseG1GC -Djava.net.preferIPv4Stack=true" > > supervisor.childopts:"-Xmx2048m -Djava.net.preferIPv4Stack=true" > > How we could trace why storm 2.4.0 is taking such high CPU utilization? > Any configuration changes needed for storm 2.4.0 while migrating from > 0.9.6? > > > > Regards, > Viraj Kadu > > On Thu, 24 Aug, 2023, 1:49 am v.s kadu, <virajkadu...@gmail.com> wrote: > >> Hello Team, >> >> I am migrating to apache storm 2.4.0 from apache storm 0.9.6 But the >> performance in apache storm 2.4.0 is slower. >> >> I read that wait strategy plays role in performance Exiting application >> on apache storm 0.9.6 has following wait strategy configuration >> >> topology.spout.wait.strategy: backtype.storm.spout.SleepWaitStrategy >> topology.disruptor.wait.strategy: com.lmax.disruptor.BlockingWaitStrategy >> topology.disruptor.wait.timeout.millis: 1000 >> topology.sleep.spout.wait.strategy.time.ms: 1 >> >> Tried tweaking wait strategy parameter in apache storm 2.4.0. Following >> tests i had performed >> 1. >> Kept all defaults as it is >> >> 2. >> topology.bolt.wait.progressive.level1.count:0 >> topology.bolt.wait.progressive.level2.count:0 >> topology.bolt.wait.progressive.level3.sleep.millis:0 >> topology.backpressure.wait.progressive.level1.count:0 >> topology.backpressure.wait.progressive.level2.count:0 >> topology.backpressure.wait.progressive.level3.sleep.millis:0 >> >> 3. >> topology.spout.wait.strategy:org.apache.storm.policy.WaitStrategy >> topology.backpressure.wait.strategy:org.apache.storm.policy.WaitStrategy >> topology.bolt.wait.strategy:org.apache.storm.policy.WaitStrategy >> topology.bolt.wait.microsec:0 >> >> But performance was lot slower in all scenarios. What other factors we >> can check for performance testing while migrating? How to improve >> performance in apache storm 2.4.0? >> >> Regards, >> Viraj Kadu >> >
