Re: reproducable panic eviction work queue

2015-07-18 Thread Johan Schuijt
With attachment this time, also not sure wether this is what you were referring 
to, so let me know if anything else needed!

- Johan


> On 18 Jul 2015, at 17:28, Johan Schuijt-Li  wrote:
> 
> Thx for your looking into this!
> 
>> 
>> Thank you for the report, I will try to reproduce this locally
>> Could you please post the full crash log ?
> 
> Of course, please see attached file.
> 
>> Also could you test
>> with a clean current kernel from Linus' tree or Dave's -net ?
> 
> Will do.
> 
>> These are available at:
>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>> git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
>> respectively.
>> 
>> One last question how many IRQs do you pin i.e. how many cores
>> do you actively use for receive ?
> 
> This varies a bit across our systems, but we’ve managed to reproduce this 
> with IRQs pinned on as many as 2,4,8 or 20 cores.
> 
> I won’t have access to our test-setup till Monday again, so I’ll be testing 3 
> scenario’s then:
> - Your patch
> - Linux tree
> - Dave’s -net tree
> 
> I’ll make sure to keep you posted on all the results then. We have a kernel 
> dump of the panic, so if you need me to extract any data from there just let 
> me know! (Some instructions might be needed)
> 
> - Johan

[28732.285611] general protection fault:  [#1] SMP 
[28732.285665] Modules linked in: vhost_net vhost macvtap macvlan act_police 
cls_u32 sch_ingress cls_fw sch_sfq sch_htb nf_conntrack_ipv6 nf_defrag_ipv6 
xt_conntrack xt_physdev br_netfilter ebt_arp ebt_ip6 ebt_ip ebtable_nat tun 
rpcsec_gss_krb5 nfsv4 dns_resolver ebtable_filter ebtables ip6table_raw 
ip6table_mangle ip6table_filter ip6_tables nfsd auth_rpcgss oid_registry 
nfs_acl nfs lockd grace fscache sunrpc bridge 8021q garp mrp stp llc bonding 
xt_CT xt_DSCP iptable_mangle ipt_REJECT nf_reject_ipv4 xt_pkttype xt_tcpudp 
nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_comment nf_conntrack_ipv4 
nf_defrag_ipv4 xt_state nf_conntrack xt_owner iptable_filter iptable_raw 
ip_tables x_tables loop joydev hid_generic usbhid hid x86_pkg_temp_thermal 
intel_powerclamp coretemp kvm_intel kvm ttm crct10dif_pclmul crc32_pclmul
[28732.286421]  ghash_clmulni_intel aesni_intel drm_kms_helper drm i2c_algo_bit 
aes_x86_64 lrw gf128mul dcdbas ipmi_si i2c_core evdev glue_helper ablk_helper 
tpm_tis mei_me tpm ehci_pci ehci_hcd mei cryptd usbcore iTCO_wdt 
iTCO_vendor_support ipmi_msghandler lpc_ich mfd_core wmi pcspkr usb_common 
shpchp sb_edac edac_core acpi_power_meter acpi_pad button processor thermal_sys 
ext4 crc16 mbcache jbd2 dm_mod sg sd_mod ahci libahci bnx2x libata ptp pps_core 
mdio crc32c_generic megaraid_sas crc32c_intel scsi_mod libcrc32c
[28732.286955] CPU: 9 PID: 56 Comm: kworker/9:0 Not tainted 3.18.7-transip-2.0 
#1
[28732.287023] Hardware name: Dell Inc. PowerEdge M620/0VHRN7, BIOS 2.5.2 
02/03/2015
[28732.287096] Workqueue: events inet_frag_worker
[28732.287139] task: 885f3d9fc210 ti: 885f3da0 task.ti: 
885f3da0
[28732.287205] RIP: 0010:[]  [] 
inet_evict_bucket+0x119/0x180
[28732.287278] RSP: 0018:885f3da03d58  EFLAGS: 00010292
[28732.287318] RAX: 885f3da03d08 RBX: dead001000a8 RCX: 885f3da03d08
[28732.287362] RDX: 0006 RSI: 885f3da03ce8 RDI: dead001000a8
[28732.287406] RBP: 0002 R08: 0286 R09: 88302f401640
[28732.287450] R10: 8000 R11: 88602ec0c138 R12: 81a8d8c0
[28732.287494] R13: 885f3da03d70 R14:  R15: 881d6efe1a00
[28732.287538] FS:  () GS:88602f28() 
knlGS:
[28732.287606] CS:  0010 DS:  ES:  CR0: 80050033
[28732.287647] CR2: 00b11000 CR3: 004f05b24000 CR4: 000427e0
[28732.287691] Stack:
[28732.287722]  81a905e0 81a905e8 814f4599 
881d6efe1a58
[28732.287807]  0246 002e 81a8d8c0 
81a918c0
[28732.287891]  02d3 0019 0240 
8148075a
[28732.287975] Call Trace:
[28732.288013]  [] ? _raw_spin_unlock_irqrestore+0x9/0x10
[28732.288056]  [] ? inet_frag_worker+0x5a/0x250
[28732.288103]  [] ? process_one_work+0x149/0x3f0
[28732.288146]  [] ? worker_thread+0x63/0x490
[28732.288187]  [] ? rescuer_thread+0x290/0x290
[28732.288229]  [] ? kthread+0xce/0xf0
[28732.288269]  [] ? kthread_create_on_node+0x180/0x180
[28732.288313]  [] ? ret_from_fork+0x7c/0xb0
[28732.288353]  [] ? kthread_create_on_node+0x180/0x180
[28732.288396] Code: 8b 04 24 66 83 40 08 01 48 8b 7c 24 18 48 85 ff 74 2a 48 
83 ef 58 75 13 eb 22 0f 1f 84 00 00 00 00 00 48 83 eb 58 48 89 df 74 11 <48> 8b 
5f 58 41 ff 94 24 70 40 00 00 48 85 db 75 e6 48 83 c4 28 
[28732.288827] RIP  [] inet_evict_bucket+0x119/0x180
[28732.288873]  RSP 


Re: reproducable panic eviction work queue

2015-07-18 Thread Johan Schuijt
Thx for your looking into this!

> 
> Thank you for the report, I will try to reproduce this locally
> Could you please post the full crash log ?

Of course, please see attached file.

> Also could you test
> with a clean current kernel from Linus' tree or Dave's -net ?

Will do.

> These are available at:
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
> respectively.
> 
> One last question how many IRQs do you pin i.e. how many cores
> do you actively use for receive ?

This varies a bit across our systems, but we’ve managed to reproduce this with 
IRQs pinned on as many as 2,4,8 or 20 cores.

I won’t have access to our test-setup till Monday again, so I’ll be testing 3 
scenario’s then:
- Your patch
- Linux tree
- Dave’s -net tree

I’ll make sure to keep you posted on all the results then. We have a kernel 
dump of the panic, so if you need me to extract any data from there just let me 
know! (Some instructions might be needed)

- 
JohanN�r��yb�X��ǧv�^�)޺{.n�+���z�^�)w*jg����ݢj/���z�ޖ��2�ޙ&�)ߡ�a�����G���h��j:+v���w��٥

Re: reproducable panic eviction work queue

2015-07-18 Thread Johan Schuijt
Yes, we already found these and are included in our kernel, but even with these 
patches we still receive the panic.

- Johan


> On 18 Jul 2015, at 10:56, Eric Dumazet  wrote:
> 
> On Fri, 2015-07-17 at 21:18 +0000, Johan Schuijt wrote:
>> Hey guys, 
>> 
>> 
>> We’re currently running into a reproducible panic in the eviction work
>> queue code when we pin al our eth* IRQ to different CPU cores (in
>> order to scale our networking performance for our virtual servers).
>> This only occurs in kernels >= 3.17 and is a result of the following
>> change:
>> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-3.18.y&id=b13d3cbfb8e8a8f53930af67d1ebf05149f32c24
>> 
>> 
>> The race/panic we see seems to be the same as, or similar to:
>> https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-3.18.y&id=65ba1f1ec0eff1c25933468e1d238201c0c2cb29
>> 
>> 
>> We can confirm that this is directly exposed by the IRQ pinning since
>> disabling this stops us from being able to reproduce this case :)
>> 
>> 
>> How te reproduce: in our test-setup we have 4 machines generating UDP
>> packets which are send to the vulnerable host. These all have a MTU of
>> 100 (for test purposes) and send UDP packets of a size of 256 bytes.
>> Within half an hour you will see the following panic:
>> 
>> 
>> crash> bt
>> PID: 56 TASK: 885f3d9fc210  CPU: 9   COMMAND: "kworker/9:0"
>> #0 [885f3da03b60] machine_kexec at 8104a1f7
>> #1 [885f3da03bb0] crash_kexec at 810db187
>> #2 [885f3da03c80] oops_end at 81015140
>> #3 [885f3da03ca0] general_protection at 814f6c88
>>[exception RIP: inet_evict_bucket+281]
>>RIP: 81480699  RSP: 885f3da03d58  RFLAGS: 00010292
>>RAX: 885f3da03d08  RBX: dead001000a8  RCX:
>> 885f3da03d08
>>RDX: 0006  RSI: 885f3da03ce8  RDI:
>> dead001000a8
>>RBP: 0002   R8: 0286   R9:
>> 88302f401640
>>R10: 8000  R11: 88602ec0c138  R12:
>> 81a8d8c0
>>R13: 885f3da03d70  R14:   R15:
>> 881d6efe1a00
>>ORIG_RAX:   CS: 0010  SS: 0018
>> #4 [885f3da03db0] inet_frag_worker at 8148075a
>> #5 [885f3da03e10] process_one_work at 8107be19
>> #6 [885f3da03e60] worker_thread at 8107c6e3
>> #7 [885f3da03ed0] kthread at 8108103e
>> #8 [885f3da03f50] ret_from_fork at 814f4d7c
>> 
>> 
>> We would love to receive your input on this matter.
>> 
>> 
>> Thx in advance,
>> 
>> 
>> - Johan
> 
> Check commits 65ba1f1ec0eff1c25933468e1d238201c0c2cb29 &
> d70127e8a942364de8dd140fe73893efda363293
> 
> Also please send your mails in text format, not html, and CC netdev ( I
> did here)
> 
>> 
>> 
> 
>