I've just tried disabling Transparent Huge Pages (THP), and there was no noticeable difference in the CPU usage.
We've been running tests and it seems that RHEL6 is better about respecting the time of the sleep statements in the code; when the code says, sleep 1ms, RHEL4 tends to be more sloppy and wake up late, while RHEL6 will wake up the thread close to the time specified. That explains why increasing the "topology.sleep.spout.wait.strategy.time.ms" from 1 to 25 ms cut the CPU usage in almost half. The comparison is being done using a machine running in VMWare; not sure that matters. Thanks, Dennis On Sun, Mar 15, 2015 at 9:24 PM, Dennis Cammer <[email protected]> wrote: > As for the number of spout threads, we are usually using 5 spouts + DRPC. > This is a 4 core machine, but using 30-50% of one CPU while idle is > suspicious that something is wrong. > > Thanks, I haven't checked or disabled THP. I'll give that a try. > > On Sun, Mar 15, 2015 at 9:05 PM, 임정택 <[email protected]> wrote: > >> Did you disable THP? If you didn't, it would be better to give it a try. >> (Sorry, but I don't find a relationship between these, but THP shows CPU >> issues from many places.) >> https://twitter.com/mattsta/status/529327782351097856 >> >> 2015-03-16 8:03 GMT+09:00 Dennis Cammer <[email protected]>: >> >>> I haven't heard anything back on this. Does anybody else see the worker >>> process using 25-50% of one CPU when the system is completely idle when >>> using RHEL6 or CentOS 6? Is that expected? I'd also appreciate a "no, I >>> don't see that". >>> >>> Thanks, >>> Dennis >>> >>> On Thu, Mar 12, 2015 at 3:34 AM, Dennis Cammer <[email protected]> >>> wrote: >>> >>>> We currently use storm 0.9.2 and upgraded from Redhat Enterprise Linux >>>> 4 to Redhat Enterprise Linux 6. After the upgrade, we saw that the CPU >>>> usage when the system is idle is a lot higher on the new system. It's now >>>> around ~55%, when it was around ~14% on RHEL4 with the same topology. >>>> Looking at the threads using the CPU, there is no single thread using a >>>> lot of CPU, but the CPU usage is distributed between many threads. >>>> Increasing "topology.sleep.spout.wait.strategy.time.ms" from 1 to 25 >>>> ms helped to bring down the CPU from ~55% to ~30% with no single thread >>>> using more than 1.4% CPU (most are using 0.3%). >>>> >>>> Has anybody seen something similar? >>>> If you are using RHEL6, have you seen higher than expected CPU for an >>>> idle worker? Is this expected? >>>> >>>> Thanks, >>>> Dennis >>>> >>> >>> >> >> >> -- >> Name : 임 정택 >> Blog : http://www.heartsavior.net / http://dev.heartsavior.net >> Twitter : http://twitter.com/heartsavior >> LinkedIn : http://www.linkedin.com/in/heartsavior >> > >
