Thank you for your valuable responses. 1. Can we change these values at run-time? 2. Can we set these values on every bolt/spout individually? 3. Is there a correlation between batch timeout and wait timeout? Because if the batch size/batch timeout is set to high number, then the wait timeout might elapse in the time taken to construct the batch.
On Mon, Jul 30, 2018 at 7:38 PM, Jungtaek Lim <[email protected]> wrote: > TOPOLOGY_DISRUPTOR_WAIT_TIMEOUT_MILLIS is used for defining wait time > when queue in disruptor doesn't have item to read (so it is blocking wait > with timeout). > TOPOLOGY_DISRUPTOR_BATCH_TIMEOUT_MILLIS is used to limit the timeout > while constructing logical batch. Storm handles events separately (in other > words, 1-by-1) but does some batching in internal places. The default value > for TOPOLOGY_DISRUPTOR_BATCH_TIMEOUT_MILLIS is set to so low to avoid > putting outstanding latency while batching, but assuming timeout occurs > often, increasing the value with consideration of possible additional > latency (if you're having latency SLA for handling event) would be good to > go. > > -Jungtaek Lim (HeartSaVioR) > > 2018년 7월 31일 (화) 오전 4:32, Nithin Uppalapati <nithinuppalapati1989@gmail. > com>님이 작성: > >> I am using storm version 1.2.1 and also using storm-kafka-client version >> 1.2.1 >> >> May I know the difference between TOPOLOGY_DISRUPTOR_WAIT_TIMEOUT_MILLIS >> and TOPOLOGY_DISRUPTOR_BATCH_TIMEOUT_MILLIS, and configuring which does >> what. Also, can we change these values at runtime? >> >> TOPOLOGY_DISRUPTOR_BATCH_TIMEOUT_MILLIS I played only with this >> parameter and it reduced the CPU utilization as I increase the value. >> >> Thanks, >> Nithin >> >> On Sun, Jul 29, 2018 at 6:16 PM, Jungtaek Lim <[email protected]> wrote: >> >>> Could you share your version of Storm and storm-kafka (or >>> storm-kafka-client, more encouraged)? I'd encourage end users to keep using >>> latest, which Storm community concerns about now. And if you are using >>> Kafka 0.10 or higher you're encouraged to use storm-kafka-client. >>> >>> Please read below slide if you didn't, especially from 45 page. There're >>> so many code changes after the slide, but basic considerations still apply, >>> so you could get much help from it. >>> https://www.slideshare.net/ptgoetz/scaling-apache-storm- >>> strata-hadoopworld-2014 >>> >>> Please try to adjust "topology.disruptor.wait.timeout.millis" to higher >>> which default value is 1000 (1 sec) if you follow the slide and the slide >>> doesn't help much. >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>> 2018년 7월 27일 (금) 오후 11:11, Nithin Uppalapati < >>> [email protected]>님이 작성: >>> >>>> Hi, >>>> >>>> When we are running a topology the CPU utilization is high around 400%. >>>> Also, a high percentage of that is being used in com.lmax.disruptor. >>>> ProcessingSequenceBarrier.waitFor(long) -> > >>>> TimeoutBlockingWaitStrategy.waitFor(long,Sequence, >>>> Sequence,SequenceBarrier). >>>> >>>> KafkaSpout Parallelism: 4 >>>> Bolt Parallelism: 20 >>>> The number of tuples flowing through the kafka spout is around >>>> 1600/sec. >>>> >>>> Any suggestion on how to reduce the CPU utilization. >>>> >>>> Thanks, >>>> Nithin >>>> >>>> >>
