On 11/13/2014 11:13 AM, Mike Zupan wrote: > I’m having a strange issue doing on with 3.10 or 3.17 kernel that I’m not > seeing with 2.6. We are seeing a lot of softirq requests for network cards > even on a mostly idle system. It happens on any server in the cluster if I > deploy the 3.10 or 3.17 kernel > > Using top we noticed this process using a lot of CPU. As soon as I give the > server traffic load spikes to well over 200 for a 1 min average. > > [kworker/u66:2] > > That lead us to install `powertop` and then saw this > > Usage Events/s Category Description > 1110 ms/s 2045.2 Process php-fpm: pool www > 36.0 ms/s 2165.4 Timer tick_sched_timer > 57.7 ms/s 1285.0 Process nginx: worker process > 13.3 ms/s 416.0 Timer hrtimer_wakeup > 39.1 ms/s 350.7 Interrupt [3] net_rx(softirq) > > This is the same on a 2.6 series getting the same amount of traffic > > Usage Events/s Category Description > 1795 ms/s 1654.0 Process php-fpm: pool www > 45.3 ms/s 1110.4 Process nginx: worker process > 562.8 µs/s 122.4 Process /usr/bin/java -Xms200m > -Xmx2000m -Xss256k -XX:MaxDirectMemorySize=516m -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -Dage > 497.1 µs/s 59.3 Process /usr/sbin/gmond > 16.0 ms/s 30.2 Process /usr/bin/redis-server > 127.0.0.1:6379 > 4.7 ms/s 32.8 Process python > /usr/bin/statsd-relay.py > 81.7 ms/s 0.00 Timer tcp_delack_timer > 24.8 ms/s 0.00 Timer tick_sched_timer > 549.4 µs/s 9.2 Process java -Xmx6g -server > -Dfile.encoding=utf-8 -XX:OnOutOfMemoryError=kill -9 %p > -XX:+HeapDumpOnOutOfMemoryError -XX:HeapD > 15.2 ms/s 0.00 Interrupt [3] net_rx(softirq) > > > As you can see the net_rx is 0 on 2.6 but we get as many as 4k/s on 3.10. The > server specs are the same and removed all sysctl settings. I can replicate > the issue just by installing 3.10 on a server. > > the nics we have in are > > 06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > 06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > 06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > 06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > > -- > Mike Zupan
Mike, I would recommend installing the "perf" tool and running "perf top" instead of "powertop" to try and determine what is running on your system. The powertop tool is meant to determine what is waking you up out of sleep states, not what is actually making use of the system. As such with powertop you could see 0 events per second and all that would mean is that the system isn't getting to sleep as it is too busy, which a high count could actually mean your system is going idle resulting in a significant number of wake-ups. For interrupt information you might try watching the rate at which /proc/interrupts increases or you could install sysstat and then run "sar -I XALL 4 500 | grep -v 0.00", to watch for the non zero interrupt rates after figuring out which interrupts belong to your network adapter. Thanks, Alex ------------------------------------------------------------------------------ Comprehensive Server Monitoring with Site24x7. Monitor 10 servers for $9/Month. Get alerted through email, SMS, voice calls or mobile push notifications. Take corrective actions from your mobile device. http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
