Re: Git bisected regression for ipsec/aead

2016-08-27 Thread Sowmini Varadhan 
On (08/25/16 16:49), Herbert Xu wrote:
> 
> On Fri, Aug 19, 2016 at 03:21:24PM -0400, Sowmini Varadhan wrote:
> >7271b33cb87e80f3a416fb031ad3ca87f0bea80a is the first bad commit

> This bisection doesn't make much sense as this patch just causes
> cryptd to be used a little more more frequently.  But it does
> point the finger at cryptd.

On additional testing, I think this might be related to some
subtle race/timing issue so that git-bisect may not necessarily
be able to pin-point the correct bad-commit: if I add a few 
printks in other parts of the IPsec stack (and change the timing), 
the problem does not reproduce. Let me try to collect more data
on this. 

Meanwhile, if you can see some bug in the commit above, then
it probably makes sense to fix it upstream anyway.

--Sowmini

--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Git bisected regression for ipsec/aead

2016-08-25 Thread Sowmini Varadhan 
On (08/25/16 16:49), Herbert Xu wrote:
> This bisection doesn't make much sense as this patch just causes
> cryptd to be used a little more more frequently.  But it does
> point the finger at cryptd.
:
> So we have list corruption here, possibly caused by use-after-free.
> I did spot one bug in the AEAD cryptd path where we end up using
> the wrong tfm to track refcnt.
> 
> Does this patch help?

Unfortunately no, I still see the panic.

--Sowmini

--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Git bisected regression for ipsec/aead

2016-08-25 Thread Herbert Xu 
On Fri, Aug 19, 2016 at 03:21:24PM -0400, Sowmini Varadhan wrote:
> 
> Hi Herbert,
> 
> In the process of testing ipsec I ran into panics (details below)
> with  the algorithm
> "aead rfc4106(gcm(aes)) 0x1234567890123456789012345678901234567890 64"
> 
> git-bisect analyzed this down to
> 
>7271b33cb87e80f3a416fb031ad3ca87f0bea80a is the first bad commit
>commit 7271b33cb87e80f3a416fb031ad3ca87f0bea80a
>Author: Herbert Xu 
>Date:   Tue Jun 21 16:55:16 2016 +0800
> 
> crypto: ghash-clmulni - Fix cryptd reordering

This bisection doesn't make much sense as this patch just causes
cryptd to be used a little more more frequently.  But it does
point the finger at cryptd.
 
> [  124.627594] BUG: unable to handle kernel paging request at 000100c5
> [  124.627612] [ cut here ]
> [  124.627620] WARNING: CPU: 3 PID: 0 at lib/list_debug.c:62 
> __list_del_entry+0x
> 86/0xd0
> [  124.627621] list_del corruption. next->prev should be 88085cebd168, 
> but w
> as ff8d
> [  124.627622] Modules linked in:

So we have list corruption here, possibly caused by use-after-free.
I did spot one bug in the AEAD cryptd path where we end up using
the wrong tfm to track refcnt.

Does this patch help?

---8<---
Subject: crypto: cryptd - Use correct tfm object for AEAD tracking

The AEAD code path incorrectly uses the child tfm to track the
cryptd refcnt, and then potentially frees the child tfm.

Fixes: 81760ea6a95a ("crypto: cryptd - Add helpers to check...")
Reported-by: Sowmini Varadhan 
Signed-off-by: Herbert Xu 

diff --git a/crypto/cryptd.c b/crypto/cryptd.c
index cf8037a..77207b4 100644
--- a/crypto/cryptd.c
+++ b/crypto/cryptd.c
@@ -733,13 +733,14 @@ static void cryptd_aead_crypt(struct aead_request *req,
rctx = aead_request_ctx(req);
compl = rctx->complete;
 
+   tfm = crypto_aead_reqtfm(req);
+
if (unlikely(err == -EINPROGRESS))
goto out;
aead_request_set_tfm(req, child);
err = crypt( req );
 
 out:
-   tfm = crypto_aead_reqtfm(req);
ctx = crypto_aead_ctx(tfm);
refcnt = atomic_read(>refcnt);
 
-- 
Email: Herbert Xu 
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
To unsubscribe from this list: send the line "unsubscribe linux-crypto" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Git bisected regression for ipsec/aead

2016-08-19 Thread Sowmini Varadhan 

Hi Herbert,

In the process of testing ipsec I ran into panics (details below)
with  the algorithm
"aead rfc4106(gcm(aes)) 0x1234567890123456789012345678901234567890 64"

git-bisect analyzed this down to

   7271b33cb87e80f3a416fb031ad3ca87f0bea80a is the first bad commit
   commit 7271b33cb87e80f3a416fb031ad3ca87f0bea80a
   Author: Herbert Xu 
   Date:   Tue Jun 21 16:55:16 2016 +0800

crypto: ghash-clmulni - Fix cryptd reordering
 :

Could you please take a look? here are additional details:

To reproduce the panic, I set up ipsec as follows, on 2 test machines

  # #set up laddr to be local interface address, faddr as peer's addres.
  # ip x p add dir out src $laddr dst $faddr proto tcp \
   tmpl proto esp src $laddr dst $faddr spi 0x0001 \
   mode transport reqid 1
  # ip x p add dir in src $laddr dst $faddr proto tcp \
   tmpl proto esp dst $laddr src $faddr spi 0x0001 \
   mode transport reqid 1
  # ip x s add proto esp src $laddr dst $faddr spi 0x0001 \
   mode transport reqid 1 replay-window 32 \
   aead 'rfc4106(gcm(aes))' 0x1234567890123456789012345678901234567890 64 \
   sel src $laddr dst $faddr proto tcp
  # ip x s add proto esp dst $laddr src $faddr spi 0x0001 \
   mode transport reqid 1 replay-window 32 \
   aead 'rfc4106(gcm(aes))' 0x1234567890123456789012345678901234567890 64 \
   sel src $laddr dst $faddr proto tcp

Then run iperf i.e., start "iperf -s" on one node (server), and 
"iperf -c $faddr -P 1" on the on the other (client). The client will
panic with something like this in the dmesg:

[  124.627594] BUG: unable to handle kernel paging request at 000100c5
[  124.627612] [ cut here ]
[  124.627620] WARNING: CPU: 3 PID: 0 at lib/list_debug.c:62 __list_del_entry+0x
86/0xd0
[  124.627621] list_del corruption. next->prev should be 88085cebd168, but w
as ff8d
[  124.627622] Modules linked in:
   :
   :
[  124.627650] CPU: 3 PID: 0 Comm: swapper/3 Tainted: GE   
4.7.0-rc1-ipsec-offload-api2+ #15
[  124.627651] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS 
GRNDSDP1.86B.0046.R00.1502111331 02/11/2015
[  124.627666]  [] dump_stack+0x51/0x78
[  124.627667]  [] ? __list_del_entry+0x86/0xd0
[  124.627673]  [] __warn+0xfd/0x120
[  124.627676]  [] warn_slowpath_fmt+0x49/0x50
[  124.627677]  [] __list_del_entry+0x86/0xd0
[  124.627683]  [] detach_tasks+0x1ab/0x280
[  124.627685]  [] load_balance+0x32b/0x860
[  124.627691]  [] ? enqueue_hrtimer+0x49/0xa0
[  124.627693]  [] ? run_timer_softirq+0x4c/0x300
[  124.627695]  [] rebalance_domains+0x144/0x290
[  124.627696]  [] run_rebalance_domains+0x49/0x60
[  124.627701]  [] __do_softirq+0xeb/0x2d8
[  124.627703]  [] ? hrtimer_interrupt+0xb8/0x170
[  124.627706]  [] irq_exit+0xa5/0xb0
[  124.627708]  [] smp_apic_timer_interrupt+0x46/0x60
[  124.627709]  [] apic_timer_interrupt+0x7f/0x90
[  124.627709]   
[  124.627716]  [] ? cpuidle_enter_state+0xc9/0x2d0
[  124.627718]  [] ? cpuidle_enter_state+0xbb/0x2d0
[  124.627719]  [] ? menu_select+0x103/0x3a0
[  124.627721]  [] cpuidle_enter+0x17/0x20
[  124.627723]  [] call_cpuidle+0x2e/0x40
[  124.627724]  [] cpuidle_idle_call+0x68/0x100
[  124.627725]  [] cpu_idle_loop+0x155/0x240
[  124.627726]  [] cpu_startup_entry+0x21/0x30
[  124.627732]  [] start_secondary+0x73/0x80
[  124.627733] ---[ end trace d9352c1808e65391 ]---
[  124.640240] paging request
[  124.640557]  at 000100c5
 :
[  124.640809] IP: [] account_system_time+0x66/0x130
[  124.641146] PGD 85a8c3067 PUD 0 
[  124.641533] Thread overran stack, or stack corrupted
[  124.641795] Oops:  [#1] SMP
[  124.642049] Modules linked in: seqiv esp4 xfrm4_mode_transport 
sha256_generic drbg ansi_cprng ctr ghash_generic gf128mul ghash_clmulni_intel 
cryptd gcm autofs4 8021q garp stp llc sunrpc cpufreq_ondemand ipv6 iTCO_wdt 
iTCO_vendor_support pcspkr i40e i2c_i801 i2c_core sg lpc_ich mfd_core xhci_pci 
xhci_hcd ixgbe dca hwmon mdio hed wmi ipmi_si ipmi_msghandler acpi_cpufreq 
acpi_pad ext4(E) mbcache(E) jbd2(E) sd_mod(E) sr_mod(E) cdrom(E) ahci(E) 
libahci(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[  124.647568] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRNDSDP1
.86B.0046.R00.1502111331 02/11/2015
[  124.648027] task: 88085f344100 ti: 88085f348000 task.ti: 880855cb
b2e0
[  124.648293] RIP: 0010:[]  [] 
account_system_time+0x66/0x130
[  124.648814] RSP: 0018:88087ec03d68  EFLAGS: 00010086
[  124.649075] RAX: 0001 RBX: 88085f344100 RCX: ff8d
[  124.649342] RDX: 0001 RSI: 0002 RDI: 
[  124.649609] RBP: 88087ec03d88 R08: 0001 R09: 880855cbb2a8
[  124.649877] R10:  R11: 0001 R12: 
[  124.650143] R13: 88085f32edd8 R14: 88087ec0fc80 R15: 001cdb6654dd
[  124.650409] FS:  ()