Public bug reported:
Ubuntu 12.04 LTS was installed on 2 servers on a SM15000-XE system.
Bidirectional iperf traffic was started on 7 nics between the 2 servers.
After a few hours, following stack trace was observed on the dmesg.
Sluggish network performance was observed after that
[15640.958682] sched: RT throttling activated [16924.585541]
[ cut here ]
[16924.585549] WARNING: at
/build/buildd/linux-3.2.0/net/sched/sch_generic.c:255
dev_watchdog+0x25a/0x270() [16924.585551] Hardware name: Sabre2 [16924.585552]
NETDEV WATCHDOG: eth6 (e1000): transmit queue 0 timed out [16924.585553]
Modules linked in: 8021q garp stp nfsd nfs lockd fscache auth_rpcgss nfs_acl
sunrpc msr mac_hid lp parport e1000 [16924.585564] Pid: 0, comm: swapper/7 Not
tainted 3.2.0-23-generic #36-Ubuntu [16924.585566] Call Trace:
> [16924.585567][] warn_slowpath_common+0x7f/0xc0
> [16924.585575] [] warn_slowpath_fmt+0x46/0x50
> [16924.585578] [] dev_watchdog+0x25a/0x270 [16924.585581]
> [] ? perf_rotate_context+0x110/0x220 [16924.585585]
> [] ? __queue_work+0x320/0x320 [16924.585587]
> [] ? qdisc_reset+0x50/0x50 [16924.585589]
> [] ? qdisc_reset+0x50/0x50 [16924.585592]
> [] call_timer_fn+0x46/0x160 [16924.585594]
> [] ? qdisc_reset+0x50/0x50 [16924.585597]
> [] run_timer_softirq+0x132/0x2a0 [16924.585600]
> [] ? ktime_get+0x65/0xe0 [16924.585604]
> [] __do_softirq+0xa8/0x210 [16924.585607]
> [] ? read_tsc+0x9/0x20 [16924.585609] []
> ? tick_program_event+0x24/0x30 [16924.585613] []
> call_softirq+0x1c/0x30 [16924.585616] []
> do_softirq+0x65/0xa0 [16924.585619] [] irq_exit+0x8e/0xb0
> [16924.585622] [] smp_apic_timer_interrupt+0x6e/0x99
> [16924.585625] [] apic_timer_interrupt+0x6e/0x80
> [16924.585626][] ? intel_idle+0xed/0x150
> [16924.585631] [] ? intel_idle+0xcf/0x150 [16924.585634]
> [] cpuidle_idle_call+0xc1/0x280 [16924.585637]
> [] cpu_idle+0xca/0x120 [16924.585640] []
> start_secondary+0xd9/0xdb [16924.585642]
---[ end trace 3c74a3d373267b03 ]---
[16941.890574] BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
[16941.963686] Modules linked in: 8021q garp stp nfsd nfs lockd fscache
auth_rpcgss nfs_acl sunrpc msr mac_hid lp parport e1000
[16941.963697] CPU 0
[16941.963698] Modules linked in: 8021q garp stp nfsd nfs lockd fscache
auth_rpcgss nfs_acl sunrpc msr mac_hid lp parport e1000
[16941.963704]
[16941.963706] Pid: 0, comm: swapper/0 Tainted: GW3.2.0-23-generic
#36-Ubuntu SeaMicro Sabre2/Type2 - Board Product Name1
[16941.963710] RIP: 0010:[] []
__alloc_skb+0xcf/0x240
[16941.963718] RSP: 0018:88043fc03c70 EFLAGS: 00010202
[16941.963719] RAX: RBX: 88043fc03c60 RCX: 000a
[16941.963721] RDX: 88040f876680 RSI: 00cc RDI: 880420dab478
[16941.963722] RBP: 88043fc03cb0 R08: 0680 R09: 0800
[16941.963724] R10: 880420dab500 R11: 0001 R12: 88043fc03be8
[16941.963725] R13: 8166555e R14: 88043fc03cb0 R15: 88040f876000
[16941.963727] FS: () GS:88043fc0()
knlGS:
[16941.963729] CS: 0010 DS: ES: CR0: 8005003b
[16941.963730] CR2: 7f19afb0ad50 CR3: 000210999000 CR4: 000406f0
[16941.963732] DR0: DR1: DR2:
[16941.963734] DR3: DR6: 0ff0 DR7: 0400
[16941.963735] Process swapper/0 (pid: 0, threadinfo 81c0, task
81c0d020)
[16941.963737] Stack:
[16941.987830] 88043fc03ca0 0632813265a4 880424796090
88042120
[16941.987834] 00ff 88040f875840
8804218a7a80
[16941.987837] 88043fc03cd0 81532424 00fe
c90001877fd0
[16941.987840] Call Trace:
[16942.017153]
[16942.042416] [] __netdev_alloc_skb+0x24/0x50
[16942.042423] [] e1000_alloc_rx_buffers+0x291/0x4f0 [e1000]
[16942.042427] [] ? map_single+0x60/0x60
[16942.042430] [] e1000_clean_rx_irq+0x3ba/0x4d0 [e1000]
[16942.042434] [] e1000_clean+0x51/0xc0 [e1000]
[16942.042438] [] net_rx_action+0x134/0x290
[16942.042442] [] __do_softirq+0xa8/0x210
[16942.042446] [] ? eoi_ioapic_irq.isra.23+0x5e/0x70
[16942.042450] [] call_softirq+0x1c/0x30
[16942.042453] [] do_softirq+0x65/0xa0
[16942.042456] [] irq_exit+0x8e/0xb0
[16942.042458] [] do_IRQ+0x63/0xe0
[16942.042462] [] common_interrupt+0x6e/0x6e
[16942.042463]
[16942.067708] [] ? hrtimer_try_to_cancel+0x50/0xc0
[16942.067712] [] ? tick_nohz_restart_sched_tick+0x106/0x130
[16942.067715] [] ? tick_nohz_restart_sched_tick+0x102/0x130
[16942.067719] [] cpu_idle+0x103/0x120
[16942.067723] [] rest_init+0x72/0x74
[16942.067728] [] start_kernel+0x3ba/0x3c7
[16942.067731] [] x86_64_start_reservations+0x132/0x136
[16942.067734] [] ? early_idt_handlers+0x140/0x140
[16942.067737] [] x86_64_start_kernel+0xcd/0xdc
[16942.067739] Code: df be cc 00