TOPOLOGY_DISRUPTOR_WAIT_TIMEOUT_MILLIS is used for defining wait time when queue in disruptor doesn't have item to read (so it is blocking wait with timeout). TOPOLOGY_DISRUPTOR_BATCH_TIMEOUT_MILLIS is used to limit the timeout while constructing logical batch. Storm handles events separately (in other words, 1-by-1) but does some batching in internal places. The default value for TOPOLOGY_DISRUPTOR_BATCH_TIMEOUT_MILLIS is set to so low to avoid putting outstanding latency while batching, but assuming timeout occurs often, increasing the value with consideration of possible additional latency (if you're having latency SLA for handling event) would be good to go.
-Jungtaek Lim (HeartSaVioR) 2018년 7월 31일 (화) 오전 4:32, Nithin Uppalapati <[email protected]>님이 작성: > I am using storm version 1.2.1 and also using storm-kafka-client version > 1.2.1 > > May I know the difference between TOPOLOGY_DISRUPTOR_WAIT_TIMEOUT_MILLIS > and TOPOLOGY_DISRUPTOR_BATCH_TIMEOUT_MILLIS, and configuring which does > what. Also, can we change these values at runtime? > > TOPOLOGY_DISRUPTOR_BATCH_TIMEOUT_MILLIS I played only with this > parameter and it reduced the CPU utilization as I increase the value. > > Thanks, > Nithin > > On Sun, Jul 29, 2018 at 6:16 PM, Jungtaek Lim <[email protected]> wrote: > >> Could you share your version of Storm and storm-kafka (or >> storm-kafka-client, more encouraged)? I'd encourage end users to keep using >> latest, which Storm community concerns about now. And if you are using >> Kafka 0.10 or higher you're encouraged to use storm-kafka-client. >> >> Please read below slide if you didn't, especially from 45 page. There're >> so many code changes after the slide, but basic considerations still apply, >> so you could get much help from it. >> >> https://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014 >> >> Please try to adjust "topology.disruptor.wait.timeout.millis" to higher >> which default value is 1000 (1 sec) if you follow the slide and the slide >> doesn't help much. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >> 2018년 7월 27일 (금) 오후 11:11, Nithin Uppalapati < >> [email protected]>님이 작성: >> >>> Hi, >>> >>> When we are running a topology the CPU utilization is high around 400%. >>> Also, a high percentage of that is being used in >>> com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(long) >>> -> >>> >TimeoutBlockingWaitStrategy.waitFor(long,Sequence,Sequence,SequenceBarrier). >>> >>> KafkaSpout Parallelism: 4 >>> Bolt Parallelism: 20 >>> The number of tuples flowing through the kafka spout is around 1600/sec. >>> >>> Any suggestion on how to reduce the CPU utilization. >>> >>> Thanks, >>> Nithin >>> >>> >
