Re: svn commit: r343291 - in head/sys: dev/vmware/vmxnet3 net
On Sat, Feb 2, 2019 at 3:29 PM Patrick Kelsey wrote: > > > On Sat, Feb 2, 2019 at 9:28 AM Yuri Pankov wrote: > >> Patrick Kelsey wrote: >> > Author: pkelsey >> > Date: Tue Jan 22 01:11:17 2019 >> > New Revision: 343291 >> > URL: https://svnweb.freebsd.org/changeset/base/343291 >> > >> > Log: >> > onvert vmx(4) to being an iflib driver. >> > >> > Also, expose IFLIB_MAX_RX_SEGS to iflib drivers and add >> > iflib_dma_alloc_align() to the iflib API. >> > >> > Performance is generally better with the tunable/sysctl >> > dev.vmx..iflib.tx_abdicate=1. >> > >> > Reviewed by:shurd >> > MFC after: 1 week >> > Relnotes: yes >> > Sponsored by: RG Nets >> > Differential Revision: https://reviews.freebsd.org/D18761 >> >> This breaks vmx interfaces for me in ESXi 6.7 (output below). The >> review mentions setting hw.pci.honor_msi_blacklist="0" and it helps >> indeed -- worth mentioning in UPDATING? >> >> vmx0: port 0x3000-0x300f mem >> 0xfe903000-0xfe903fff,0xfe902000-0xfe902fff,0xfe90-0xfe901fff at >> device 0.0 on pci3 >> vmx0: Using 512 tx descriptors and 256 rx descriptors >> vmx0: msix_init qsets capped at 8 >> vmx0: intr CPUs: 20 queue msgs: 24 admincnt: 1 >> vmx0: Using 8 rx queues 8 tx queues >> vmx0: attempting to allocate 9 MSI-X vectors (25 supported) >> vmx0: failed to allocate 9 MSI-X vectors, err: 6 - using MSI >> vmx0: attempting to allocate 1 MSI vectors (1 supported) >> msi: routing MSI IRQ 25 to local APIC 6 vector 48 >> vmx0: using IRQ 25 for MSI >> vmx0: Using an MSI interrupt >> msi: Assigning MSI IRQ 25 to local APIC 25 vector 48 >> msi: Assigning MSI IRQ 25 to local APIC 24 vector 48 >> vmx0: bpf attached >> vmx0: Ethernet address: 00:00:00:00:00:33 >> vmx0: netmap queues/slots: TX 1/512, RX 1/512 >> vmx0: device enable command failed! >> vmx0: link state changed to UP >> vmx0: device enable command failed! >> >> > Setting hw.pci.honor_msi_blacklist="0" should only be necessary if you > want to operate with more than one queue. If > hw.pci.honor_msi_blacklist="0" is not set, then MSI-X will not be > available, and MSI will be used, which reduces the number of queues that > can be configured for use to 1. This case should work correctly. > > I am able to reproduce the behavior you described above on ESXi 6.7 using > the latest snapshot release (based on r343598). The error that appears in > the ESXi logs will be similar to: > > 2019-02-02T15:14:02.986Z| vcpu-1| I125: VMXNET3 user: failed to activate > 'Ethernet0', status: 0xbad0001 > > which vaguely means 'the device did not like something about the > configuration it was given'. I will see if I can determine the root > cause. Given that enabling MSI-X seems to work around the problem, and > based on other issues I encountered during development, I currently suspect > there is a problem with the interrupt index that is being configured for > the transmit queue in the device configuration structure when using MSI. > > Indeed, the interrupt index for the tx queue in MSI mode was the problem. This is now fixed in r343688 ( https://svnweb.freebsd.org/changeset/base/343688). Thanks for reporting the issue! -Patrick ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r343291 - in head/sys: dev/vmware/vmxnet3 net
On Sat, Feb 2, 2019 at 9:28 AM Yuri Pankov wrote: > Patrick Kelsey wrote: > > Author: pkelsey > > Date: Tue Jan 22 01:11:17 2019 > > New Revision: 343291 > > URL: https://svnweb.freebsd.org/changeset/base/343291 > > > > Log: > > onvert vmx(4) to being an iflib driver. > > > > Also, expose IFLIB_MAX_RX_SEGS to iflib drivers and add > > iflib_dma_alloc_align() to the iflib API. > > > > Performance is generally better with the tunable/sysctl > > dev.vmx..iflib.tx_abdicate=1. > > > > Reviewed by:shurd > > MFC after: 1 week > > Relnotes: yes > > Sponsored by: RG Nets > > Differential Revision: https://reviews.freebsd.org/D18761 > > This breaks vmx interfaces for me in ESXi 6.7 (output below). The > review mentions setting hw.pci.honor_msi_blacklist="0" and it helps > indeed -- worth mentioning in UPDATING? > > vmx0: port 0x3000-0x300f mem > 0xfe903000-0xfe903fff,0xfe902000-0xfe902fff,0xfe90-0xfe901fff at > device 0.0 on pci3 > vmx0: Using 512 tx descriptors and 256 rx descriptors > vmx0: msix_init qsets capped at 8 > vmx0: intr CPUs: 20 queue msgs: 24 admincnt: 1 > vmx0: Using 8 rx queues 8 tx queues > vmx0: attempting to allocate 9 MSI-X vectors (25 supported) > vmx0: failed to allocate 9 MSI-X vectors, err: 6 - using MSI > vmx0: attempting to allocate 1 MSI vectors (1 supported) > msi: routing MSI IRQ 25 to local APIC 6 vector 48 > vmx0: using IRQ 25 for MSI > vmx0: Using an MSI interrupt > msi: Assigning MSI IRQ 25 to local APIC 25 vector 48 > msi: Assigning MSI IRQ 25 to local APIC 24 vector 48 > vmx0: bpf attached > vmx0: Ethernet address: 00:00:00:00:00:33 > vmx0: netmap queues/slots: TX 1/512, RX 1/512 > vmx0: device enable command failed! > vmx0: link state changed to UP > vmx0: device enable command failed! > > Setting hw.pci.honor_msi_blacklist="0" should only be necessary if you want to operate with more than one queue. If hw.pci.honor_msi_blacklist="0" is not set, then MSI-X will not be available, and MSI will be used, which reduces the number of queues that can be configured for use to 1. This case should work correctly. I am able to reproduce the behavior you described above on ESXi 6.7 using the latest snapshot release (based on r343598). The error that appears in the ESXi logs will be similar to: 2019-02-02T15:14:02.986Z| vcpu-1| I125: VMXNET3 user: failed to activate 'Ethernet0', status: 0xbad0001 which vaguely means 'the device did not like something about the configuration it was given'. I will see if I can determine the root cause. Given that enabling MSI-X seems to work around the problem, and based on other issues I encountered during development, I currently suspect there is a problem with the interrupt index that is being configured for the transmit queue in the device configuration structure when using MSI. -Patrick ___ svn-src-all@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscr...@freebsd.org"
Re: svn commit: r343291 - in head/sys: dev/vmware/vmxnet3 net
Patrick Kelsey wrote: > Author: pkelsey > Date: Tue Jan 22 01:11:17 2019 > New Revision: 343291 > URL: https://svnweb.freebsd.org/changeset/base/343291 > > Log: > onvert vmx(4) to being an iflib driver. > > Also, expose IFLIB_MAX_RX_SEGS to iflib drivers and add > iflib_dma_alloc_align() to the iflib API. > > Performance is generally better with the tunable/sysctl > dev.vmx..iflib.tx_abdicate=1. > > Reviewed by:shurd > MFC after: 1 week > Relnotes: yes > Sponsored by: RG Nets > Differential Revision: https://reviews.freebsd.org/D18761 This breaks vmx interfaces for me in ESXi 6.7 (output below). The review mentions setting hw.pci.honor_msi_blacklist="0" and it helps indeed -- worth mentioning in UPDATING? vmx0: port 0x3000-0x300f mem 0xfe903000-0xfe903fff,0xfe902000-0xfe902fff,0xfe90-0xfe901fff at device 0.0 on pci3 vmx0: Using 512 tx descriptors and 256 rx descriptors vmx0: msix_init qsets capped at 8 vmx0: intr CPUs: 20 queue msgs: 24 admincnt: 1 vmx0: Using 8 rx queues 8 tx queues vmx0: attempting to allocate 9 MSI-X vectors (25 supported) vmx0: failed to allocate 9 MSI-X vectors, err: 6 - using MSI vmx0: attempting to allocate 1 MSI vectors (1 supported) msi: routing MSI IRQ 25 to local APIC 6 vector 48 vmx0: using IRQ 25 for MSI vmx0: Using an MSI interrupt msi: Assigning MSI IRQ 25 to local APIC 25 vector 48 msi: Assigning MSI IRQ 25 to local APIC 24 vector 48 vmx0: bpf attached vmx0: Ethernet address: 00:00:00:00:00:33 vmx0: netmap queues/slots: TX 1/512, RX 1/512 vmx0: device enable command failed! vmx0: link state changed to UP vmx0: device enable command failed! signature.asc Description: OpenPGP digital signature
svn commit: r343291 - in head/sys: dev/vmware/vmxnet3 net
Author: pkelsey
Date: Tue Jan 22 01:11:17 2019
New Revision: 343291
URL: https://svnweb.freebsd.org/changeset/base/343291
Log:
onvert vmx(4) to being an iflib driver.
Also, expose IFLIB_MAX_RX_SEGS to iflib drivers and add
iflib_dma_alloc_align() to the iflib API.
Performance is generally better with the tunable/sysctl
dev.vmx..iflib.tx_abdicate=1.
Reviewed by: shurd
MFC after:1 week
Relnotes: yes
Sponsored by: RG Nets
Differential Revision:https://reviews.freebsd.org/D18761
Modified:
head/sys/dev/vmware/vmxnet3/if_vmx.c
head/sys/dev/vmware/vmxnet3/if_vmxvar.h
head/sys/net/iflib.c
head/sys/net/iflib.h
Modified: head/sys/dev/vmware/vmxnet3/if_vmx.c
==
--- head/sys/dev/vmware/vmxnet3/if_vmx.cTue Jan 22 01:07:20 2019
(r343290)
+++ head/sys/dev/vmware/vmxnet3/if_vmx.cTue Jan 22 01:11:17 2019
(r343291)
@@ -1,6 +1,7 @@
/*-
* Copyright (c) 2013 Tsubai Masanari
* Copyright (c) 2013 Bryan Venteicher
+ * Copyright (c) 2018 Patrick Kelsey
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
@@ -24,7 +25,6 @@ __FBSDID("$FreeBSD$");
#include
#include
-#include
#include
#include
#include
@@ -34,7 +34,6 @@ __FBSDID("$FreeBSD$");
#include
#include
#include
-#include
#include
#include
@@ -46,9 +45,8 @@ __FBSDID("$FreeBSD$");
#include
#include
#include
+#include
-#include
-
#include
#include
#include
@@ -57,8 +55,6 @@ __FBSDID("$FreeBSD$");
#include
#include
-#include
-
#include
#include
#include
@@ -67,139 +63,95 @@ __FBSDID("$FreeBSD$");
#include
#include
+#include "ifdi_if.h"
+
#include "if_vmxreg.h"
#include "if_vmxvar.h"
#include "opt_inet.h"
#include "opt_inet6.h"
-#ifdef VMXNET3_FAILPOINTS
-#include
-static SYSCTL_NODE(DEBUG_FP, OID_AUTO, vmxnet3, CTLFLAG_RW, 0,
-"vmxnet3 fail points");
-#define VMXNET3_FP _debug_fail_point_vmxnet3
-#endif
-static int vmxnet3_probe(device_t);
-static int vmxnet3_attach(device_t);
-static int vmxnet3_detach(device_t);
-static int vmxnet3_shutdown(device_t);
+#define VMXNET3_VMWARE_VENDOR_ID 0x15AD
+#define VMXNET3_VMWARE_DEVICE_ID 0x07B0
+static pci_vendor_info_t vmxnet3_vendor_info_array[] =
+{
+ PVID(VMXNET3_VMWARE_VENDOR_ID, VMXNET3_VMWARE_DEVICE_ID, "VMware
VMXNET3 Ethernet Adapter"),
+ /* required last entry */
+ PVID_END
+};
+
+static void*vmxnet3_register(device_t);
+static int vmxnet3_attach_pre(if_ctx_t);
+static int vmxnet3_msix_intr_assign(if_ctx_t, int);
+static voidvmxnet3_free_irqs(struct vmxnet3_softc *);
+static int vmxnet3_attach_post(if_ctx_t);
+static int vmxnet3_detach(if_ctx_t);
+static int vmxnet3_shutdown(if_ctx_t);
+static int vmxnet3_suspend(if_ctx_t);
+static int vmxnet3_resume(if_ctx_t);
+
static int vmxnet3_alloc_resources(struct vmxnet3_softc *);
static voidvmxnet3_free_resources(struct vmxnet3_softc *);
static int vmxnet3_check_version(struct vmxnet3_softc *);
-static voidvmxnet3_initial_config(struct vmxnet3_softc *);
-static voidvmxnet3_check_multiqueue(struct vmxnet3_softc *);
+static voidvmxnet3_set_interrupt_idx(struct vmxnet3_softc *);
-static int vmxnet3_alloc_msix_interrupts(struct vmxnet3_softc *);
-static int vmxnet3_alloc_msi_interrupts(struct vmxnet3_softc *);
-static int vmxnet3_alloc_legacy_interrupts(struct vmxnet3_softc *);
-static int vmxnet3_alloc_interrupt(struct vmxnet3_softc *, int, int,
- struct vmxnet3_interrupt *);
-static int vmxnet3_alloc_intr_resources(struct vmxnet3_softc *);
-static int vmxnet3_setup_msix_interrupts(struct vmxnet3_softc *);
-static int vmxnet3_setup_legacy_interrupt(struct vmxnet3_softc *);
-static int vmxnet3_setup_interrupts(struct vmxnet3_softc *);
-static int vmxnet3_alloc_interrupts(struct vmxnet3_softc *);
+static int vmxnet3_queues_shared_alloc(struct vmxnet3_softc *);
+static voidvmxnet3_init_txq(struct vmxnet3_softc *, int);
+static int vmxnet3_tx_queues_alloc(if_ctx_t, caddr_t *, uint64_t *, int,
int);
+static voidvmxnet3_init_rxq(struct vmxnet3_softc *, int, int);
+static int vmxnet3_rx_queues_alloc(if_ctx_t, caddr_t *, uint64_t *, int,
int);
+static voidvmxnet3_queues_free(if_ctx_t);
-static voidvmxnet3_free_interrupt(struct vmxnet3_softc *,
- struct vmxnet3_interrupt *);
-static voidvmxnet3_free_interrupts(struct vmxnet3_softc *);
-
-#ifndef VMXNET3_LEGACY_TX
-static int vmxnet3_alloc_taskqueue(struct vmxnet3_softc *);
-static voidvmxnet3_start_taskqueue(struct vmxnet3_softc *);
-static voidvmxnet3_drain_taskqueue(struct vmxnet3_softc *);
-static voidvmxnet3_free_taskqueue(struct vmxnet3_softc *);
-#endif
-
-static int
