Re: kTLS in combination with mlx4 is very unstable
On 01. mai 2018 18:09, Dave Watson wrote: On 04/24/18 10:01 AM, Dave Watson wrote: On 04/22/18 11:21 PM, Andre Tomt wrote: The kernel seems to get increasingly unstable as I load it up with client connections. At about 9Gbps and 700 connections, it is okay at least for a while - it might run fine for say 45 minutes. Once it gets to 20 - 30Gbps, the kernel will usually start spewing OOPSes within minutes and the traffic drops. Some bad interaction between mlx4 and kTLS? I tried to repro, but wasn't able to - of course I don't have an mlx4 test setup. If I manually add a tls_write_space call after do_tcp_sendpages, I get a similar stack though. Something like the following should work, can you test? Thanks Thank you! This does indeed seem to have fixed this problem. It has been sustaining ~36Gbps and about 3000 clients for about an hour now without any crashes. Tested on 4.17-rc3 git snapshot as of a few hours ago. As for performance I am very happy with kTLS. This is some very cool stuff. I dig it. I'm getting a bit over 10Gbps per 2.5Ghz Broadwell-DE core on this low power quad core system. Nearly ideal network conditions and all the data is hot in pagecache but still. I'm going to have to add another port. ;-)
Re: kTLS in combination with mlx4 is very unstable
Hi Andre,
On 04/24/18 10:01 AM, Dave Watson wrote:
> On 04/22/18 11:21 PM, Andre Tomt wrote:
> > The kernel seems to get increasingly unstable as I load it up with client
> > connections. At about 9Gbps and 700 connections, it is okay at least for a
> > while - it might run fine for say 45 minutes. Once it gets to 20 - 30Gbps,
> > the kernel will usually start spewing OOPSes within minutes and the traffic
> > drops.
> >
> > Some bad interaction between mlx4 and kTLS?
I tried to repro, but wasn't able to - of course I don't have an mlx4
test setup. If I manually add a tls_write_space call after
do_tcp_sendpages, I get a similar stack though.
Something like the following should work, can you test? Thanks
diff --git a/include/net/tls.h b/include/net/tls.h
index 8c56809..ee78f33 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -187,6 +187,7 @@ struct tls_context {
struct scatterlist *partially_sent_record;
u16 partially_sent_offset;
unsigned long flags;
+ bool in_tcp_sendpages;
u16 pending_open_record_frags;
int (*push_pending_record)(struct sock *sk, int flags);
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index 3aafb87..095af65 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -114,6 +114,7 @@ int tls_push_sg(struct sock *sk,
size = sg->length - offset;
offset += sg->offset;
+ ctx->in_tcp_sendpages = 1;
while (1) {
if (sg_is_last(sg))
sendpage_flags = flags;
@@ -148,6 +149,8 @@ int tls_push_sg(struct sock *sk,
}
clear_bit(TLS_PENDING_CLOSED_RECORD, >flags);
+ ctx->in_tcp_sendpages = 0;
+ ctx->sk_write_space(sk);
return 0;
}
@@ -217,6 +220,9 @@ static void tls_write_space(struct sock *sk)
{
struct tls_context *ctx = tls_get_ctx(sk);
+ if (ctx->in_tcp_sendpages)
+ return;
+
if (!sk->sk_write_pending && tls_is_pending_closed_record(ctx)) {
gfp_t sk_allocation = sk->sk_allocation;
int rc;
Re: kTLS in combination with mlx4 is very unstable
On 04/22/18 11:21 PM, Andre Tomt wrote: > kTLS looks fun, so I decided to play with it. It is quite spiffy - however > with mlx4 I get kernel crashes I'm not seeing when testing on ixgbe. > > For testing I'm using a git build of the "stream reflector" cubemap[1] > configured with kTLS and 8 worker threads running on 4 physical cores, > loading it up with a ~13Mbps MPEG-TS stream pulled from satelite TV. > > The kernel seems to get increasingly unstable as I load it up with client > connections. At about 9Gbps and 700 connections, it is okay at least for a > while - it might run fine for say 45 minutes. Once it gets to 20 - 30Gbps, > the kernel will usually start spewing OOPSes within minutes and the traffic > drops. > > Some bad interaction between mlx4 and kTLS? I'm not familiar with any mlx4 specific issues, but it looks like there is enough information here to fix the stack overflow from recursive callbacks. I'll see if I can come up with something. Thanks for the report. > > [1] > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.sesse.net_-3Fp-3Dcubemap=DwICaQ=5VD0RTtNlTh3ycd41b3MUw=vou6lT5jmE_fWQWZZgNrsMWu4RT87QAB9V07tPHlP5U=nXYfAmb3ozJUT-pI1JGDgMYhxb7Dq4XSorzfyyQeGWk=05SnCOrNbK2DHRub2qPdVxAzXW9e7utxqDMeVaGBd8k= > > First OOPS (from 4.16.3): > > [ 660.467358] BUG: stack guard page was hit at b136e403 (stack is > > ded3f179..835ee6c5) > > [ 660.467422] kernel stack overflow (double-fault): [#1] SMP PTI > > [ 660.467457] Modules linked in: coretemp intel_rapl sb_edac > > x86_pkg_temp_thermal intel_powerclamp iTCO_wdt gpio_ich iTCO_vendor_support > > kvm_intel mxm_wmi xfs libcrc32c kvm crc32c_generic irqbypass nls_iso8859_1 > > crct10dif_pclmul crc32_pclmul nls_cp437 ghash_clmulni_intel vfat fat > > aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_pch_thermal > > mei_me sg mei lpc_ich mfd_core evdev ipmi_si ipmi_devintf ipmi_msghandler > > wmi acpi_pad tls ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 > > hid_generic usbhid hid mlx4_ib mlx4_en ib_core sd_mod ast i2c_algo_bit ttm > > drm_kms_helper syscopyarea sysfillrect ehci_pci xhci_pci sysimgblt > > fb_sys_fops ahci libahci xhci_hcd ehci_hcd libata crc32c_intel nvme drm > > usbcore scsi_mod mlx4_core ixgbe i2c_core mdio usb_common devlink hwmon > > nvme_core rtc_cmos > > [ 660.467856] CPU: 4 PID: 660 Comm: cubemap Not tainted 4.16.0-1 #1 > > [ 660.467890] Hardware name: Supermicro Super Server/X10SDV-4C-TLN2F, BIOS > > 1.2c 09/19/2017 > > [ 660.467939] RIP: 0010:__kmalloc+0x7/0x1f0 > > [ 660.467962] RSP: 0018:abafc27b8000 EFLAGS: 00010206 > > [ 660.467992] RAX: 000d RBX: 0010 RCX: > > abafc27b8070 > > [ 660.468030] RDX: 98a0d0235490 RSI: 01080020 RDI: > > 001d > > [ 660.468069] RBP: 000d R08: 98a0d5be4860 R09: > > 98a0ec299180 > > [ 660.468106] R10: abafc27b80b8 R11: 0010 R12: > > 0010 > > [ 660.468145] R13: 98a0ec299180 R14: 98a0ec299180 R15: > > > > [ 660.468184] FS: 7f8a35ffb700() GS:98a17fd0() > > knlGS: > > [ 660.468227] CS: 0010 DS: ES: CR0: 80050033 > > [ 660.468258] CR2: abafc27b7ff8 CR3: 0004698ee001 CR4: > > 003606e0 > > [ 660.468297] DR0: DR1: DR2: > > > > [ 660.468334] DR3: DR6: fffe0ff0 DR7: > > 0400 > > [ 660.468373] Call Trace: > > [ 660.468401] gcmaes_encrypt.constprop.5+0x137/0x240 [aesni_intel] > > [ 660.468439] ? generic_gcmaes_encrypt+0x5f/0x80 [aesni_intel] > > [ 660.468476] ? gcmaes_wrapper_encrypt+0x36/0x80 [aesni_intel] > > [ 660.468511] ? tls_push_record+0x1d3/0x390 [tls] > > [ 660.468537] ? tls_push_record+0x1d3/0x390 [tls] > > [ 660.468565] ? tls_write_space+0x6a/0x80 [tls] > > [ 660.468593] ? do_tcp_sendpages+0x8d/0x580 > > [ 660.468618] ? tls_push_sg+0x74/0x130 [tls] > > [ 660.468643] ? tls_push_record+0x24a/0x390 [tls] > > [ 660.468671] ? tls_write_space+0x6a/0x80 [tls] > > [ 660.468697] ? do_tcp_sendpages+0x8d/0x580 > > [ 660.468722] ? tls_push_sg+0x74/0x130 [tls] > > [ 660.468748] ? tls_push_record+0x24a/0x390 [tls] > > [ 660.468776] ? tls_write_space+0x6a/0x80 [tls] > > [ 660.468802] ? do_tcp_sendpages+0x8d/0x580 > > [ 660.468826] ? tls_push_sg+0x74/0x130 [tls] > > [ 660.468852] ? tls_push_record+0x24a/0x390 [tls] > > [ 660.468880] ? tls_write_space+0x6a/0x80 [tls] > > [ 660.468906] ? do_tcp_sendpages+0x8d/0x580 > > [ 660.468931] ? tls_push_sg+0x74/0x130 [tls] > > [ 660.468957] ? tls_push_record+0x24a/0x390 [tls] > > [ 660.470165] ? tls_write_space+0x6a/0x80 [tls] > > [ 660.471363] ? do_tcp_sendpages+0x8d/0x580 > > [ 660.472555] ? tls_push_sg+0x74/0x130 [tls] > > [ 660.473713] ? tls_push_record+0x24a/0x390 [tls] > > [ 660.474838] ? tls_write_space+0x6a/0x80 [tls] > > [ 660.475927]
Re: kTLS in combination with mlx4 is very unstable
On 22. april 2018 23:21, Andre Tomt wrote: Hello! kTLS looks fun, so I decided to play with it. It is quite spiffy - however with mlx4 I get kernel crashes I'm not seeing when testing on ixgbe. For testing I'm using a git build of the "stream reflector" cubemap[1] configured with kTLS and 8 worker threads running on 4 physical cores, loading it up with a ~13Mbps MPEG-TS stream pulled from satelite TV. The kernel seems to get increasingly unstable as I load it up with client connections. At about 9Gbps and 700 connections, it is okay at least for a while - it might run fine for say 45 minutes. Once it gets to 20 - 30Gbps, the kernel will usually start spewing OOPSes within minutes and the traffic drops. Some bad interaction between mlx4 and kTLS? Hardware is a quad core Xeon-D 1520 using a dual port Mellanox ConnectX-3 VPI with a single 40Gbps ethernet link configured. Mellanox mlx4 driver settings are kernel.org upstream defaults. Interface is configured with FQ qdisc and sockets are using BBR congestion control. Tested on kernel 4.14.34, 4.15.17, and 4.16.2 - 4.16.3. [1] https://git.sesse.net/?p=cubemap First OOPS (from 4.16.3) It also blows up with a similar trace on 4.17-rc2.
kTLS in combination with mlx4 is very unstable
Hello! kTLS looks fun, so I decided to play with it. It is quite spiffy - however with mlx4 I get kernel crashes I'm not seeing when testing on ixgbe. For testing I'm using a git build of the "stream reflector" cubemap[1] configured with kTLS and 8 worker threads running on 4 physical cores, loading it up with a ~13Mbps MPEG-TS stream pulled from satelite TV. The kernel seems to get increasingly unstable as I load it up with client connections. At about 9Gbps and 700 connections, it is okay at least for a while - it might run fine for say 45 minutes. Once it gets to 20 - 30Gbps, the kernel will usually start spewing OOPSes within minutes and the traffic drops. Some bad interaction between mlx4 and kTLS? Hardware is a quad core Xeon-D 1520 using a dual port Mellanox ConnectX-3 VPI with a single 40Gbps ethernet link configured. Mellanox mlx4 driver settings are kernel.org upstream defaults. Interface is configured with FQ qdisc and sockets are using BBR congestion control. Tested on kernel 4.14.34, 4.15.17, and 4.16.2 - 4.16.3. [1] https://git.sesse.net/?p=cubemap First OOPS (from 4.16.3): [ 660.467358] BUG: stack guard page was hit at b136e403 (stack is ded3f179..835ee6c5) [ 660.467422] kernel stack overflow (double-fault): [#1] SMP PTI [ 660.467457] Modules linked in: coretemp intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp iTCO_wdt gpio_ich iTCO_vendor_support kvm_intel mxm_wmi xfs libcrc32c kvm crc32c_generic irqbypass nls_iso8859_1 crct10dif_pclmul crc32_pclmul nls_cp437 ghash_clmulni_intel vfat fat aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_pch_thermal mei_me sg mei lpc_ich mfd_core evdev ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_pad tls ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 hid_generic usbhid hid mlx4_ib mlx4_en ib_core sd_mod ast i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect ehci_pci xhci_pci sysimgblt fb_sys_fops ahci libahci xhci_hcd ehci_hcd libata crc32c_intel nvme drm usbcore scsi_mod mlx4_core ixgbe i2c_core mdio usb_common devlink hwmon nvme_core rtc_cmos [ 660.467856] CPU: 4 PID: 660 Comm: cubemap Not tainted 4.16.0-1 #1 [ 660.467890] Hardware name: Supermicro Super Server/X10SDV-4C-TLN2F, BIOS 1.2c 09/19/2017 [ 660.467939] RIP: 0010:__kmalloc+0x7/0x1f0 [ 660.467962] RSP: 0018:abafc27b8000 EFLAGS: 00010206 [ 660.467992] RAX: 000d RBX: 0010 RCX: abafc27b8070 [ 660.468030] RDX: 98a0d0235490 RSI: 01080020 RDI: 001d [ 660.468069] RBP: 000d R08: 98a0d5be4860 R09: 98a0ec299180 [ 660.468106] R10: abafc27b80b8 R11: 0010 R12: 0010 [ 660.468145] R13: 98a0ec299180 R14: 98a0ec299180 R15: [ 660.468184] FS: 7f8a35ffb700() GS:98a17fd0() knlGS: [ 660.468227] CS: 0010 DS: ES: CR0: 80050033 [ 660.468258] CR2: abafc27b7ff8 CR3: 0004698ee001 CR4: 003606e0 [ 660.468297] DR0: DR1: DR2: [ 660.468334] DR3: DR6: fffe0ff0 DR7: 0400 [ 660.468373] Call Trace: [ 660.468401] gcmaes_encrypt.constprop.5+0x137/0x240 [aesni_intel] [ 660.468439] ? generic_gcmaes_encrypt+0x5f/0x80 [aesni_intel] [ 660.468476] ? gcmaes_wrapper_encrypt+0x36/0x80 [aesni_intel] [ 660.468511] ? tls_push_record+0x1d3/0x390 [tls] [ 660.468537] ? tls_push_record+0x1d3/0x390 [tls] [ 660.468565] ? tls_write_space+0x6a/0x80 [tls] [ 660.468593] ? do_tcp_sendpages+0x8d/0x580 [ 660.468618] ? tls_push_sg+0x74/0x130 [tls] [ 660.468643] ? tls_push_record+0x24a/0x390 [tls] [ 660.468671] ? tls_write_space+0x6a/0x80 [tls] [ 660.468697] ? do_tcp_sendpages+0x8d/0x580 [ 660.468722] ? tls_push_sg+0x74/0x130 [tls] [ 660.468748] ? tls_push_record+0x24a/0x390 [tls] [ 660.468776] ? tls_write_space+0x6a/0x80 [tls] [ 660.468802] ? do_tcp_sendpages+0x8d/0x580 [ 660.468826] ? tls_push_sg+0x74/0x130 [tls] [ 660.468852] ? tls_push_record+0x24a/0x390 [tls] [ 660.468880] ? tls_write_space+0x6a/0x80 [tls] [ 660.468906] ? do_tcp_sendpages+0x8d/0x580 [ 660.468931] ? tls_push_sg+0x74/0x130 [tls] [ 660.468957] ? tls_push_record+0x24a/0x390 [tls] [ 660.470165] ? tls_write_space+0x6a/0x80 [tls] [ 660.471363] ? do_tcp_sendpages+0x8d/0x580 [ 660.472555] ? tls_push_sg+0x74/0x130 [tls] [ 660.473713] ? tls_push_record+0x24a/0x390 [tls] [ 660.474838] ? tls_write_space+0x6a/0x80 [tls] [ 660.475927] ? do_tcp_sendpages+0x8d/0x580 [ 660.476977] ? tls_push_sg+0x74/0x130 [tls] [ 660.477999] ? tls_push_record+0x24a/0x390 [tls] [ 660.478968] ? tls_write_space+0x6a/0x80 [tls] [ 660.479902] ? do_tcp_sendpages+0x8d/0x580 [ 660.480790] ? tls_push_sg+0x74/0x130 [tls] [ 660.481644] ? tls_push_record+0x24a/0x390 [tls] [ 660.482483] ? tls_write_space+0x6a/0x80 [tls] [ 660.483301] ? do_tcp_sendpages+0x8d/0x580 [
