Freebsd-10.0-CURRENT problem in the bottom half
Hi,
In Freebsd 10.0-current with Emulex's OCE driver, I observe that the bottom
half is hogging all the CPU which is leading to system sluggishness. I used the
same hardware to check the behavior on 9.1-RELEASE, everything is fine, bottom
half is not taking more than 10% of the CPU even at the line rate speed. But
with half the throughput of line rate in Freebsd-10.0-current all the CPUs peak
and top -aSCHIP shows that it's all bottom half who is eating the CPU. I also
tried with Intel's 10g NIC to see if it has got the same problem and
interestingly it is also having the same problem, where bottom half is hogging
all the CPU.
We did some profiling to check where the problem is. I'm pasting the profiling
results for reference here. Observation from the results is that
uma_zalloc_arg== __mtx_lock_sleep is taking quite a bit of the CPU. We sleep
at a few places in the bottom half and I think the bottom half scheduling is
what is causing the problem. Please let me know if you see something very
obvious from the profiling data.
I'm seeing similar problem with ixgbe driver as well uma_zalloc_arg==
__mtx_lock_sleep is taking quite a bit of the CPU.
Top command results are also pasted below for both 10.0-current (where the
problem is seen) and 9.1-RELEASE (where the problem is not seen).
42.99% [372357] __mtx_lock_sleep @ /boot/kernel/kernel
100.0% [372357]__mtx_lock_flags
80.37% [299245] sosend_generic
100.0% [299245] soo_write
17.27% [64322] uma_zalloc_arg
50.43% [32438] m_copym
45.85% [29490] m_getm2
03.72% [2393]tcp_output
00.00% [1] oce_alloc_rx_bufs @ /boot/kernel/oce.ko
02.35% [8762] _sleep @ /boot/kernel/kernel
100.0% [8762]sbwait
00.00% [11] in_matroute
100.0% [11] rtalloc1_fib
00.00% [10] uma_zfree_arg
60.00% [6] m_freem
40.00% [4] sbdrop_internal
00.00% [3] rtalloc1_fib
100.0% [3] rtalloc_ign_fib
00.00% [3] tcp_output
100.0% [3] tcp_usr_send
00.00% [1] _cv_wait
100.0% [1] usb_process
34.98% [303027] trash_ctor @ /boot/kernel/kernel
87.44% [264956]mb_ctor_clust
100.0% [264956] uma_zalloc_arg
100.0% [264956] m_getm2
12.51% [37921] mb_ctor_mbuf
100.0% [37921] uma_zalloc_arg
48.28% [18308] m_getm2
48.18% [18272] m_copym
03.54% [1341]tcp_output
00.05% [150] mb_ctor_pack
100.0% [150]uma_zalloc_arg
96.67% [145] m_getm2
03.33% [5] oce_alloc_rx_bufs @ /boot/kernel/oce.ko
04.36% [37786]__rw_wlock_hard @ /boot/kernel/kernel
100.0% [37786] _rw_wlock_cookie
99.88% [37741] tcp_usr_send
100.0% [37741] sosend_generic
00.11% [43] in_pcblookup_hash
100.0% [43] tcp_input
Top -aSCHIP with Freebsd-10.0-CURRENT (throughput 4.5 Gbps)
PID USERNAME PRI NICE SIZERES STATE C TIMECPU COMMAND
0 root -920 0K 960K - 11 4:20 100.00% [kernel{oce1
taskq}]
2325 root 1030 70524K 6240K CPU55 0:39 100.00% iperf -c
12.0.238.24 -t 180 -i 3 -P 10{iperf}
0 root -920 0K 960K CPU88 5:17 99.37% [kernel{oce1
taskq}]
0 root -920 0K 960K CPU10 10 4:27 99.37% [kernel{oce1
taskq}]
0 root -920 0K 960K CPU99 3:35 98.88% [kernel{oce1
taskq}]
0 root -920 0K 960K CPU77 7:25 98.29% [kernel{oce1
taskq}]
11 root 155 ki31 0K 256K RUN12 14:55 93.90% [idle{idle:
cpu12}]
11 root 155 ki31 0K 256K CPU13 13 14:48 91.70% [idle{idle:
cpu13}]
11 root 155 ki31 0K 256K CPU14 14 18:35 89.79% [idle{idle:
cpu14}]
11 root 155 ki31 0K 256K RUN15 20:55 87.99% [idle{idle:
cpu15}]
11 root 155 ki31 0K 256K RUN 4 19:35 73.68% [idle{idle:
cpu4}]
11 root 155 ki31 0K 256K RUN 0 19:47 66.89% [idle{idle:
cpu0}]
11 root 155 ki31 0K 256K CPU22 18:53 64.89% [idle{idle:
cpu2}]
11 root 155 ki31 0K 256K CPU66 19:55 61.18% [idle{idle:
cpu6}]
11 root 155 ki31 0K 256K CPU11 18:46 55.66% [idle{idle:
cpu1}]
11 root 155 ki31 0K 256K RUN 5 19:41 42.48% [idle{idle:
cpu5}]
2325 root520 70524K 6240K sbwait 6 0:15 38.77% iperf -c
12.0.238.24 -t 180 -i 3 -P 10{iperf}
11 root 155 ki31 0K 256K CPU33 19:06 36.38% [idle{idle:
cpu3}]
2325 root490 70524K 6240K CPU11 0:14 35.06% iperf -c
12.0.238.24 -t 180 -i 3 -P 10{iperf}
2325 root440 70524K 6240K sbwait 0 0:13 33.59% iperf -c
12.0.238.24 -t 180 -i 3 -P 10{iperf}
2325 root420 70524K 6240K sbwait 2 0:13 33.15% iperf -c
12.0.238.24 -t 180 -i 3 -P 10{iperf}
2325 root410
Re: Freebsd-10.0-CURRENT problem in the bottom half
Hi,
Can you please try 10-STABLE or 11-CURRENT? 10-CURRENT indicates that
you're a little behind in the source tree(s).
There's been a bit of work recently that may improve things in general for you.
Thanks!
-a
On 31 October 2013 07:00, Venkata Duvvuru
venkatkumar.duvv...@emulex.com wrote:
Hi,
In Freebsd 10.0-current with Emulex's OCE driver, I observe that the bottom
half is hogging all the CPU which is leading to system sluggishness. I used
the same hardware to check the behavior on 9.1-RELEASE, everything is fine,
bottom half is not taking more than 10% of the CPU even at the line rate
speed. But with half the throughput of line rate in Freebsd-10.0-current all
the CPUs peak and top -aSCHIP shows that it's all bottom half who is eating
the CPU. I also tried with Intel's 10g NIC to see if it has got the same
problem and interestingly it is also having the same problem, where bottom
half is hogging all the CPU.
We did some profiling to check where the problem is. I'm pasting the
profiling results for reference here. Observation from the results is that
uma_zalloc_arg== __mtx_lock_sleep is taking quite a bit of the CPU. We sleep
at a few places in the bottom half and I think the bottom half scheduling is
what is causing the problem. Please let me know if you see something very
obvious from the profiling data.
I'm seeing similar problem with ixgbe driver as well uma_zalloc_arg==
__mtx_lock_sleep is taking quite a bit of the CPU.
Top command results are also pasted below for both 10.0-current (where the
problem is seen) and 9.1-RELEASE (where the problem is not seen).
42.99% [372357] __mtx_lock_sleep @ /boot/kernel/kernel
100.0% [372357]__mtx_lock_flags
80.37% [299245] sosend_generic
100.0% [299245] soo_write
17.27% [64322] uma_zalloc_arg
50.43% [32438] m_copym
45.85% [29490] m_getm2
03.72% [2393]tcp_output
00.00% [1] oce_alloc_rx_bufs @ /boot/kernel/oce.ko
02.35% [8762] _sleep @ /boot/kernel/kernel
100.0% [8762]sbwait
00.00% [11] in_matroute
100.0% [11] rtalloc1_fib
00.00% [10] uma_zfree_arg
60.00% [6] m_freem
40.00% [4] sbdrop_internal
00.00% [3] rtalloc1_fib
100.0% [3] rtalloc_ign_fib
00.00% [3] tcp_output
100.0% [3] tcp_usr_send
00.00% [1] _cv_wait
100.0% [1] usb_process
34.98% [303027] trash_ctor @ /boot/kernel/kernel
87.44% [264956]mb_ctor_clust
100.0% [264956] uma_zalloc_arg
100.0% [264956] m_getm2
12.51% [37921] mb_ctor_mbuf
100.0% [37921] uma_zalloc_arg
48.28% [18308] m_getm2
48.18% [18272] m_copym
03.54% [1341]tcp_output
00.05% [150] mb_ctor_pack
100.0% [150]uma_zalloc_arg
96.67% [145] m_getm2
03.33% [5] oce_alloc_rx_bufs @ /boot/kernel/oce.ko
04.36% [37786]__rw_wlock_hard @ /boot/kernel/kernel
100.0% [37786] _rw_wlock_cookie
99.88% [37741] tcp_usr_send
100.0% [37741] sosend_generic
00.11% [43] in_pcblookup_hash
100.0% [43] tcp_input
Top -aSCHIP with Freebsd-10.0-CURRENT (throughput 4.5 Gbps)
PID USERNAME PRI NICE SIZERES STATE C TIMECPU COMMAND
0 root -920 0K 960K - 11 4:20 100.00% [kernel{oce1
taskq}]
2325 root 1030 70524K 6240K CPU55 0:39 100.00% iperf -c
12.0.238.24 -t 180 -i 3 -P 10{iperf}
0 root -920 0K 960K CPU88 5:17 99.37% [kernel{oce1
taskq}]
0 root -920 0K 960K CPU10 10 4:27 99.37% [kernel{oce1
taskq}]
0 root -920 0K 960K CPU99 3:35 98.88% [kernel{oce1
taskq}]
0 root -920 0K 960K CPU77 7:25 98.29% [kernel{oce1
taskq}]
11 root 155 ki31 0K 256K RUN12 14:55 93.90% [idle{idle:
cpu12}]
11 root 155 ki31 0K 256K CPU13 13 14:48 91.70% [idle{idle:
cpu13}]
11 root 155 ki31 0K 256K CPU14 14 18:35 89.79% [idle{idle:
cpu14}]
11 root 155 ki31 0K 256K RUN15 20:55 87.99% [idle{idle:
cpu15}]
11 root 155 ki31 0K 256K RUN 4 19:35 73.68% [idle{idle:
cpu4}]
11 root 155 ki31 0K 256K RUN 0 19:47 66.89% [idle{idle:
cpu0}]
11 root 155 ki31 0K 256K CPU22 18:53 64.89% [idle{idle:
cpu2}]
11 root 155 ki31 0K 256K CPU66 19:55 61.18% [idle{idle:
cpu6}]
11 root 155 ki31 0K 256K CPU11 18:46 55.66% [idle{idle:
cpu1}]
11 root 155 ki31 0K 256K RUN 5 19:41 42.48% [idle{idle:
cpu5}]
2325 root520 70524K 6240K sbwait 6 0:15 38.77% iperf -c
12.0.238.24 -t 180 -i 3 -P 10{iperf}
11 root 155 ki31 0K 256K CPU3
Re: Freebsd-10.0-CURRENT problem in the bottom half
On Thu, Oct 31, 2013 at 10:00 AM, Venkata Duvvuru venkatkumar.duvv...@emulex.com wrote: 34.98% [303027] trash_ctor @ /boot/kernel/kernel This indicates that you have INVARIANTS enabled (and you're probably running GENERIC, so you also have WITNESS). These will debugging features will substantially impact performance. To do performance testing on head, you need disable these options from your kernel config: nooptions INVARIANTS nooptions WITNESS nooptions MALLOC_DEBUG_MAXZONES Also, set MALLOC_PRODUCTION=yes in /etc/make.conf. Then rebuild your world and kernel and try again. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
