Hi Nikos, I'm investigating a kernel crash that happens fairly quickly when multiple cryptodev sessions are opened in parallel. I've tested the issue both on intel and power_pc and the traces are similar. There is no support for hardware offloading in kernel - all algorithms in kernel space are done in software.
I've tried to bisect the cryptodev code up to version 0.5-0.6 with no success in pinpointing the issue. Do you have any ideas? Thanks, Cristian S. Some notes about the tests I've done so far: - tested with kernel 3.8 and 3.10 (doesn't make a difference), openssl 1.0.1e and current cryptodev (my recent patches do not seem to touch the root cause). - openssl is built with -DHAVE_CRYPTODEV and -DUSE_CRYPTODEV_DIGESTS (if this one is disabled the crash happens less often and only on the cryptodev _ablckcipher_init route) - web server is nginx (but this should not matter) on a machine without cryptodev, configured to allow TLSv1 with only AES128-SHA and AES256-SHA ciphers.
[ 193.780805] cryptodev: driver 1.6 loaded. [ 202.164822] openssl (5025) used greatest stack depth: 5312 bytes left [ 202.952256] openssl (5401) used greatest stack depth: 5168 bytes left [ 219.537306] openssl (13303) used greatest stack depth: 5032 bytes left [ 329.967891] BUG: unable to handle kernel paging request at 00100104 [ 329.967899] IP: [<c119eee9>] crypto_larval_kill+0x15/0x58 [ 329.967938] *pde = 00000000 [ 329.967940] Oops: 0002 [#1] SMP [ 329.967944] Modules linked in: cryptodev(O) vboxsf(O) vboxguest(O) [ 329.967951] CPU: 1 PID: 21054 Comm: openssl Tainted: G O 3.10.7-gentoo #5 [ 329.967954] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 329.967956] task: f47457c0 ti: f5b56000 task.ti: f5b56000 [ 329.967958] EIP: 0060:[<c119eee9>] EFLAGS: 00210246 CPU: 1 [ 329.967961] EIP is at crypto_larval_kill+0x15/0x58 [ 329.967963] EAX: 00200200 EBX: f5a6d100 ECX: 00001711 EDX: 00100100 [ 329.967965] ESI: fffffffe EDI: c150ec6c EBP: f5b57b64 ESP: f5b57b60 [ 329.967967] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 329.967969] CR0: 8005003b CR2: 00100104 CR3: 3551a000 CR4: 00000690 [ 329.967976] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 329.967978] DR6: ffff0ff0 DR7: 00000400 [ 329.967979] Stack: [ 329.967981] fffffffe f5b57b74 c119f318 c119f2c0 c119f2c0 f5b57b84 c119ee31 f47457c0 [ 329.967986] 00000001 f5b57ba8 c119f52a 00000000 00000000 00000000 f952df8f f66dc59c [ 329.967991] 00000001 f952df8f f5b57bb4 c11a361c 00000000 f5b57bc8 f952bd5d f66dc540 [ 329.967996] Call Trace: [ 329.968001] [<c119f318>] crypto_alg_mod_lookup+0x58/0x5e [ 329.968005] [<c119f2c0>] ? crypto_larval_lookup+0xff/0xff [ 329.968007] [<c119f2c0>] ? crypto_larval_lookup+0xff/0xff [ 329.968010] [<c119ee31>] crypto_find_alg+0x31/0x35 [ 329.968013] [<c119f52a>] crypto_alloc_tfm+0x28/0x6e [ 329.968017] [<c11a361c>] crypto_alloc_ahash+0x10/0x12 [ 329.968023] [<f952bd5d>] cryptodev_hash_init+0x17/0x1d9 [cryptodev] [ 329.968027] [<f952a749>] crypto_create_session+0x56d/0x766 [cryptodev] [ 329.968036] [<c10969be>] ? zone_watermark_ok+0x1e/0x25 [ 329.968040] [<c1098475>] ? get_page_from_freelist+0xdd/0x39e [ 329.968043] [<c10969be>] ? zone_watermark_ok+0x1e/0x25 [ 329.968047] [<c101eff4>] ? kmap_atomic+0xe/0x10 [ 329.968052] [<c10bdb06>] ? __slab_free+0x195/0x268 [ 329.968056] [<c10bdb06>] ? __slab_free+0x195/0x268 [ 329.968060] [<c10be3b9>] ? kfree+0xda/0xe3 [ 329.968063] [<c10be3b9>] ? kfree+0xda/0xe3 [ 329.968066] [<f952a0ef>] ? crypto_destroy_session+0xef/0xf6 [cryptodev] [ 329.968069] [<f952a0ef>] ? crypto_destroy_session+0xef/0xf6 [cryptodev] [ 329.968072] [<f952a0ef>] ? crypto_destroy_session+0xef/0xf6 [cryptodev] [ 329.968076] [<c11cd5a9>] ? _copy_from_user+0x31/0x11c [ 329.968079] [<f952af1c>] cryptodev_ioctl+0xf4/0x558 [cryptodev] [ 329.968083] [<c10c49ab>] ? chrdev_open+0x102/0x119 [ 329.968088] [<c1093c04>] ? find_get_page+0x1d/0x68 [ 329.968092] [<c1094f04>] ? filemap_fault+0x186/0x31b [ 329.968096] [<c1093e11>] ? unlock_page+0x3e/0x41 [ 329.968099] [<c10ac49c>] ? __do_fault+0x2aa/0x2db [ 329.968102] [<c10ace56>] ? handle_pte_fault+0x252/0x5fa [ 329.968105] [<c10ad2ee>] ? handle_mm_fault+0xf0/0x102 [ 329.968108] [<f952ae28>] ? crypto_get_session_by_sid+0x4b/0x4b [cryptodev] [ 329.968112] [<c10cd7f3>] do_vfs_ioctl+0x428/0x467 [ 329.968123] [<c14e552a>] ? __do_page_fault+0x360/0x39b [ 329.968128] [<c10d3e01>] ? set_close_on_exec+0x1e/0x39 [ 329.968131] [<c10ccf7e>] ? do_fcntl+0x102/0x362 [ 329.968134] [<c10c06d6>] ? do_sys_open+0xc6/0xd0 [ 329.968137] [<c10cd87e>] SyS_ioctl+0x4c/0x6c [ 329.968141] [<c14e7efa>] sysenter_do_call+0x12/0x22 [ 329.968142] Code: d9 89 f2 b8 a4 ec 78 c1 e8 e3 20 ea ff 5a 59 8d 65 f8 5b 5e c9 c3 55 89 e5 53 89 c3 b8 94 ec 78 c1 e8 e3 28 34 00 8b 13 8b 43 04 <89> 42 04 89 10 c7 03 00 01 10 00 c7 43 04 00 02 20 00 b8 94 ec [ 329.968170] EIP: [<c119eee9>] crypto_larval_kill+0x15/0x58 SS:ESP 0068:f5b57b60 [ 329.968174] CR2: 0000000000100104 [ 329.968177] ---[ end trace fe4faec5a6be2f22 ]---
root@p4080ds:root# modprobe cryptodev
cryptodev: driver 1.6 loaded.
root@p4080ds:root# ./crash.sh
1
2
3
4
5
6
7
Unable to handle kernel paging request for data at address 0x00100104
Faulting instruction address: 0xc0201b30
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=8 P4080 DS
Modules linked in: cryptodev(O)
NIP: c0201b30 LR: c0201b24 CTR: c02010a0
REGS: eee5fa20 TRAP: 0300 Tainted: G O (3.8.13-rt9)
MSR: 00029002 <CE,EE,ME> CR: 22008482 XER: 20000000
DEAR: 00100104, ESR: 00800000
TASK = ee26b4f0[2616] 'openssl' THREAD: eee5e000 CPU: 6
GPR00: c0201c64 eee5fad0 ee26b4f0 c080e57c 00003a98 00000001 c28fdee8 020fe000
GPR08: c28fdee8 00200200 00100100 00021002 42008482 10089b14 00000000 00000000
GPR16: 00000000 00000000 100f1354 100f1334 10100000 00000000 00000000 00000001
GPR24: f9107234 00000000 00000000 eee5fb48 c0200000 fffff000 c080e57c eebae5c0
NIP [c0201b30] crypto_larval_kill+0x30/0xd0
LR [c0201b24] crypto_larval_kill+0x24/0xd0
Call Trace:
[eee5fad0] [c02011fc] crypto_larval_wait+0x9c/0x130 (unreliable)
[eee5fae0] [c0201c64] crypto_alg_mod_lookup+0x94/0xc0
[eee5faf0] [c02017b0] crypto_alloc_tfm+0x70/0x100
[eee5fb20] [f9103b04] cryptodev_hash_init+0x34/0x260 [cryptodev]
[eee5fb40] [f9101be4] crypto_create_session+0x294/0x870 [cryptodev]
[eee5fd70] [f9102790] cryptodev_ioctl+0x530/0xa20 [cryptodev]
[eee5fea0] [c00f6e0c] do_vfs_ioctl+0x9c/0x7d0
[eee5ff10] [c00f7580] sys_ioctl+0x40/0xa0
[eee5ff40] [c000ec40] ret_from_syscall+0x0/0x3c
--- Exception: c01 at 0x480fb898
LR = 0x48196098
Instruction dump:
9421fff0 7c0802a6 bfc10008 3fc0c081 3bdee57c 7c7f1b78 7fc3f378 90010014
4840e651 815f0000 813f0004 7fc3f378 <912a0004> 91490000 3d200010 61290100
---[ end trace 18a7c01ae50c3796 ]---
crash.sh
Description: crash.sh
get_mpage10.sh
Description: get_mpage10.sh
_______________________________________________ Cryptodev-linux-devel mailing list Cryptodev-linux-devel@gna.org https://mail.gna.org/listinfo/cryptodev-linux-devel
