Hi Billy, Apologies for the delay. I was off-site for the last few days. *dpdk-devbind --status* does not yield version info at all :-( [*Maybe this is a bug too - to include dpdk version info*] I installed DPDK and OVS packages on Ubuntu 17.04 Beta2 (kernel 4.10.0-14-generic) from http://packages.wand.net.nz/
# *dpkg -s openvswitch-switch-dpdk* Package: openvswitch-switch-dpdk Status: install ok installed Priority: extra Section: net Installed-Size: 2355 Maintainer: Ubuntu Developers <[email protected]> Architecture: amd64 Source: openvswitch Version: 2.7.0-3 Depends: dpdk, openvswitch-switch (= 2.7.0-3), libc6 (>= 2.14), libcap-ng0, librte-eal3 (>= 16.04), librte-ethdev5 (>= 16.07~rc1), librte-mbuf2 (>= 16.04), librte-mempool2 (>= 16.07~rc1), librte-meter1 (>= 16.04), librte-pdump1 (>= 16.07~rc1), librte-pmd-ring2 (>= 16.04), librte-ring1 (>= 16.04), librte-vhost3 (>= 16.07~rc1), libssl1.0.0 (>= 1.0.0) Enhances: openvswitch-switch Description: DPDK enabled Open vSwitch switch implementation Open vSwitch is a production quality, multilayer, software-based, Ethernet virtual switch. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, IPFIX, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag). In addition, it is designed to support distribution across multiple physical servers similar to VMware's vNetwork distributed vswitch or Cisco's Nexus 1000V. . openvswitch-switch provides the userspace components and utilities for the Open vSwitch kernel-based switch. . DPDK is a set of libraries for fast packet processing. Applications run in user-space and communicate directly with dedicated network interfaces. . This package provides a DPDK enabled implementation of the ovs-vswitchd binary. Homepage: http://openvswitch.org/ Original-Maintainer: Open vSwitch developers <[email protected]> # *cpuid -1* .... .... Logical CPU cores (0x80000008/ecx): number of CPU cores - 1 = 0x0 (0) ApicIdCoreIdSize = 0x0 (0) (multi-processing synth): multi-core (c=18), hyper-threaded (t=2) (multi-processing method): Intel leaf 0xb (APIC widths synth): CORE_width=6 SMT_width=1 (APIC synth): PKG_ID=0 CORE_ID=41 SMT_ID=0 (synth) = Intel Core i7-5000 Extreme Edition (Haswell R2) / Xeon E5-x600 v3 (Haswell-EP C1/M1/R2), 22nm On Tue, Apr 4, 2017 at 9:14 AM, O Mahony, Billy <[email protected]> wrote: > Hi Shivram, > > Apologies if I missed this info in the email thread. > > For the bug report: > * What version of DPDK were you using? > * Also what is your hardware - NICs (out from dpdk_devbind.py --status) > and processor architecture (last few lines from cpuid -1 ) > > Thanks, > Billy. > > > > > -----Original Message----- > > From: O Mahony, Billy > > Sent: Tuesday, April 4, 2017 12:00 PM > > To: 'Darrell Ball' <[email protected]>; Brad Cowie <[email protected]>; > > Shivaram Mysore <[email protected]> > > Cc: [email protected] > > Subject: RE: [ovs-discuss] Error attaching device to DPDK > > > > Hi Darrell, > > > > /Billy > > > > > -----Original Message----- > > > From: Darrell Ball [mailto:[email protected]] > > > Sent: Thursday, March 30, 2017 8:51 PM > > > To: O Mahony, Billy <[email protected]>; Brad Cowie > > > <[email protected]>; Shivaram Mysore <[email protected]> > > > Cc: [email protected] > > > Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > > > > > > > > > On 3/29/17, 7:40 AM, "O Mahony, Billy" <[email protected]> > > wrote: > > > > > > Hi All, > > > > > > > > > > > > Just to add something to this conversation that has not been > > > explicitly mentioned below. > > > > > > > > > > > > Brad outlines how to set other_config:dpdk-socket-mem to reserve > > > hugepages from both NUMA nodes in order to be sure to avoid pci > > > topology issues. > > > > > > > > > > > > Strictly this doesn’t have to be done on both nodes – just on the > > > correct node :) – but sometimes it’s easier to just cover all the > bases. > > > > > > > > > > > > In addition you will need to ensure that: > > > > > > * That linux has hugepages available on both (or the right) nodes. > > > > > > * There is a pmd thread created on each node by setting > > > other_config:pmd-cpu-mask correctly – or again at least on the right > node. > > > > > > > > > > > > These config items are described in > > > > > > * ./Documentation/intro/install/dpdk.rst > > > > > > * ./Documentation/howto/dpdk.rst > > > > > > > > > > > > Currently if there is not a pmd thread on the NUMA node belonging > > > to the PCI device the device will not be polled (a warning message is > > > issued to this effect). > > > > > > > > > [Darrell] > > > From Linux POV, the “default” memory policy is to allocate the > > > hugepage pool across numa nodes, if possible; this assumes an > > > allocation was done, which we have no reports here otherwise. > > > iirc based on the logs in this case, which I cannot view anymore, > > > there was no pmd mask configuration specified. > > > The advertised behavior should be, in this case, that a pmd thread is > > > allocated on each numa node with at least one local dpdk interface > bound. > > > iirc, the dpdk socket memory was left at default, which only > > > pre-allocated from the hugepage pool on node 0 So, does it make sense > > > to also try to pre- allocate hugepages on a numa node, where a PMD > > thread is allocated ? > > > Pls correct me in any respects. > > > > > [[BO'M]] > > > > What you're looking for here is something that will work (even if not > > optimally). And leave the optimization of exact configurations of > > pmds/hugepage-allocations and so on until later? > > > > So even if pmd-cpu-mask and dpdk-socket-mem other_config items are > > omitted OVS will 'do the right thing' ? ie. place a pmd on each numa > node and > > take some hugepages on each NUMA node. > > > > It's hard to argue with that and I can put that on our internal backlog > of tasks. > > > > How about these rules? > > * If pmd-cpu-mask is not set OVS creates one pmd thread for each NUMA > > node. (current case) > > * if dpdk-socket-mem is not set then for each NUMA node dpdk will > reserve: > > ** one quarter of the available hugepage memory > > ** 512MB of 2MB hugepages > > If hugepage granularity requires more memory may be reserved. > > > > > > > > > > > > > > > > > > > > > > > There is currently a patch under consideration so that if a > > > NUMA-local pmd is not available for a device then device will be > > > assigned to some other pmd thread. While this is not as efficient due > > > to the need for the data to travel between NUMA nodes it is less > > frustrating to configure. > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https- > > > 3A__mail.openvswitch.org_pipermail_ovs-2Ddev_2017- > > > > > 2DFebruary_329175.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVh > > > FA09CGX7JQ5Ih- > > > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=QmpnH > > > wqCC8ZBxRGZgfCZuF-9SielPs89Yd-k7crooC8&e= > > > > > > > > > > > > If you install the hwloc package you can use “hwloc-calc > > > pci=0000:01:02.0 -- intersect Socket” to show the NUMA node that is > local > > to a pci device. > > > However I have seen some systems which report that pci device is local > > > to both nodes. The lstopo command can also be used for this. > > > > > > > > > > > > Hope some of that helps, > > > > > > > > > > > > Billy. > > > > > > > > > > > > > > > > > > From: [email protected] [mailto:ovs-discuss- > > > [email protected]] On Behalf Of Darrell Ball > > > > > > Sent: Wednesday, March 29, 2017 2:36 AM > > > > > > To: Brad Cowie <[email protected]>; Shivaram Mysore > > > <[email protected]> > > > > > > Cc: [email protected] > > > > > > Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > > > > > > > > > > > > > > > > > > > > > From: <[email protected]> on behalf of Brad > > > Cowie <[email protected]> > > > > > > Date: Tuesday, March 28, 2017 at 5:35 PM > > > > > > To: Shivaram Mysore <[email protected]> > > > > > > Cc: "[email protected]" <[email protected]> > > > > > > Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > > > > > > > > > Hi folks, > > > > > > I have been helping Shivaram off-list with this problem and just > > > wanted to update this thread with the solution. > > > > > > Basically the issue was with how hugepages were being preallocated > > > by OVS for use with DPDK. I believe by default OVS sets > > > other_config:dpdk- socket-mem to "1024,0". Which will give one 1GB > > > hugepage to CPU0 and none to CPU1. > > > > > > The ports Shivaram was trying to assign to DPDK were connected > > > physically via a PCI-E bus to CPU1 and since there are no hugepages > > > preallocated there for DPDK then we can't add it. > > > > > > The fix is simple, to assign a hugepage to CPU1: > > > > > > > > > > > > $ sudo ovs-vsctl --no-wait set Open_vSwitch . > > > other_config:dpdk-socket- mem="1024,1024" > > > > > > In OVS 2.7.0 unfortunately due to an issue (that is now fixed by > > > commit > > > ef6cca6fdc67f3cee21c6bb1c13c4ca7f8241314) you get a segfault rather > > > than an error message telling you the problem. > > > > > > In OVS 2.7.90 however the issue becomes quite clear as there is a > > > nice log message like such: > > > > > > > > > > > > 2017-03-29T00:19:57Z|00067|netdev_dpdk|ERR|Insufficient memory to > > > create memory pool for netdev dpdk-p0, with MTU 1500 on socket 1 > > > > > > There is documentation regarding these fields, but for case, the > > > documentation could simply state something like > > > > > > “you need to allocate hugepage memory on each numa node where > > dpdk > > > ports are bound and or those ports > > > > > > will not work with dpdk” > > > > > > The error log could also be a little more succinct like “no memory > > > pool allocated for netdev dpdk-p0,.., usage on socket 1” > > > > > > > > > > > > Possibly, going even further, when a PMD thread gets > > > auto-allocated to run on a given numa node, > > > > > > memory allocation is defaulted there as well. If it cannot be > > > allocated, an error log is generated. > > > > > > > > > > > > Unfortunately to those unfamiliar with DPDK and NUMA architectures > > > this won't be very obvious. Potentially we could add some additional > > > help to the DPDK documentation pages for OVS that explains the > > > other_config options you may need to reconfigure on multi-CPU NUMA > > machines? > > > > > > Brad > > > > > > > > > > > > On 29 March 2017 at 10:02, Shivaram Mysore > > > <[email protected]> wrote: > > > > > > Ok thanks. Will remember the same. Right now, I will not specify > > > anything as I have to get this to work first! > > > > > > > > > > > > On Tue, Mar 28, 2017 at 2:00 PM, Bodireddy, Bhanuprakash > > > <[email protected]> wrote: > > > > > > >Question: > > > > > > > > > > > > > >I don't have any specific cpu cores associated with DPDK. Will > > > this make any > > > > > > >difference? I also did not see this requirement as a MUST have > > > in the > > > > > > >documentation. > > > > > > > > > > > > I assume your question is on PMD threads. If you don’t specify any > > > thing explicitly > > > > > > PMD pmd thread shall be created and pinned to core 0. You can > > > explicitly select the no. of pmd threads > > > > > > and the cores by explicitly setting the affinity mask using > > > 'other_config:pmd-cpu-mask'. > > > > > > > > > > > > Based on the affinity mask, the PMD threads shall be spawned and > > > the rx queues will be uniformly > > > > > > distributed among the PMD threads. > > > > > > > > > > > > Refer https://urldefense.proofpoint.com/v2/url?u=http- > > > > > 3A__docs.openvswitch.org_en_latest_howto_dpdk_&d=DwIGaQ&c=uilaK9 > > > 0D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih- > > > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=JNQjCdB > > > cYJG8H1n4bT6k_zFhtkk6wcRE27AhEuQoqjg&e= . > > > > > > > > > > > > Also note that 'dpdk-lcore-mask' should be used to set the dpdk > > > lcore threads and vswitchd. > > > > > > > > > > > > - Bhanuprakash. > > > > > > > > > > > > > > > > > > > ># cat /proc/cmdline > > > > > > >BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic > > > root=/dev/mapper/intelsdn- > > > > > > >-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G > > > > > > >hugepagesz=1G hugepages=4 > > > > > > > > > > > > > > > > > > > > >Thanks > > > > > > > > > > > > > >/Shivaram > > > > > > > > > > > > > >On Tue, Mar 28, 2017 at 1:32 PM, Shivaram Mysore > > > > > > ><[email protected]> wrote: > > > > > > >I was advised on this mailing list that VFIO drivers are better > > > than IGB_UIO > > > + > > > > > > >more forgiving [1]. Hence switched to the same. > > > > > > > > > > > > > >[1] https://urldefense.proofpoint.com/v2/url?u=http-3A__dpdk- > > > 2Dguide.gitlab.io_dpdk- > > > > > 2Dguide_setup_binding.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r= > > > BVhFA09CGX7JQ5Ih- > > > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=KgBSWU > > > sNwqMWB_Pm2CHmGAVAO5enRJxRlvbS9MR-7Ww&e= > > > > > > > > > > > > > >/Shivaram > > > > > > > > > > > > > >On Tue, Mar 28, 2017 at 1:12 PM, Bodireddy, Bhanuprakash > > > > > > ><[email protected]> wrote: > > > > > > > > > > > > > >>any hints for me based on the info provided? > > > > > > > > > > > > > >Your BIOS options and bootcmd line options are appropriate and no > > > issues > > > > > > >there. I wanted to check if you tried igb_uio drivers > > > > > > >but saw your post here > > > https://urldefense.proofpoint.com/v2/url?u=https- > > > 3A__mail.openvswitch.org_pipermail_ovs- > > > 2D&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih- > > > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=qoh0Qu6 > > > HkVOjRIO_oNqdzBGTGjRdo5tppjkG2R3KpEw&e= > > > > > > >discuss/2017-March/043993.html, reporting a crash with UIO > drivers. > > > > > > > > > > > > > >Is that the reason why you moved to VFIO? I cross checked your > > > logs but > > > > > > >found nothing suspicious there. > > > > > > > > > > > > > >-Bhanuprakash. > > > > > > > > > > > > > >> > > > > > > >>Thanks > > > > > > >> > > > > > > >>On Tue, Mar 28, 2017 at 8:41 AM, Shivaram Mysore > > > > > > >><[email protected]> wrote: > > > > > > >>Hi, > > > > > > >> > > > > > > >># dmesg | grep -e DMAR -e IOMMU > > > > > > >>[ 0.000000] ACPI: DMAR 0x00000000798AF340 000158 (v01 SUPERM > > > SMCI-- > > > > > > >>MB 00000001 INTL 20091013) > > > > > > >>[ 0.000000] DMAR: IOMMU enabled > > > > > > >>[ 0.109987] DMAR: Host address width 46 > > > > > > >>[ 0.109988] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > > > > > > >>[ 0.109997] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap > > > > > > >>d2078c106f0466 ecap f020de > > > > > > >>[ 0.109998] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 > > > > > > >>[ 0.110002] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap > > > > > > >>d2078c106f0466 ecap f020de > > > > > > >>[ 0.110003] DMAR: RMRR base: 0x0000007ba75000 end: > > > 0x0000007ba84fff > > > > > > >>[ 0.110004] DMAR: ATSR flags: 0x0 > > > > > > >>[ 0.110005] DMAR: RHSA base: 0x000000c7ffc000 proximity > domain: > > 0x0 > > > > > > >>[ 0.110006] DMAR: RHSA base: 0x000000fbffc000 proximity > domain: > > 0x1 > > > > > > >>[ 0.110008] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 > > IOMMU > > > 0 > > > > > > >>[ 0.110008] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 > > IOMMU > > > 1 > > > > > > >>[ 0.110009] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 > > IOMMU > > > 1 > > > > > > >>[ 0.110010] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 > > > > > > >>[ 0.110011] DMAR-IR: x2apic is disabled because BIOS sets > x2apic opt > > > out > > > > > > >bit. > > > > > > >>[ 0.110012] DMAR-IR: Use 'intremap=no_x2apic_optout' to > override > > > the > > > > > > >>BIOS setting. > > > > > > >>[ 0.110902] DMAR-IR: Enabled IRQ remapping in xapic mode > > > > > > >>[ 1.687595] DMAR: dmar0: Using Queued invalidation > > > > > > >>[ 1.687608] DMAR: dmar1: Using Queued invalidation > > > > > > >>[ 1.688473] DMAR: Hardware identity mapping for device > 0000:ff:08.0 > > > > > > >>[ 1.688475] DMAR: Hardware identity mapping for device > 0000:ff:08.2 > > > > > > >>[ 1.688477] DMAR: Hardware identity mapping for device > 0000:ff:08.3 > > > > > > >>[ 1.688478] DMAR: Hardware identity mapping for device > 0000:ff:09.0 > > > > > > >>[ 1.688480] DMAR: Hardware identity mapping for device > 0000:ff:09.2 > > > > > > >>[ 1.688482] DMAR: Hardware identity mapping for device > 0000:ff:09.3 > > > > > > >>[ 1.688483] DMAR: Hardware identity mapping for device > 0000:ff:0b.0 > > > > > > >>[ 1.688485] DMAR: Hardware identity mapping for device > 0000:ff:0b.1 > > > > > > >>[ 1.688487] DMAR: Hardware identity mapping for device > 0000:ff:0b.2 > > > > > > >>[ 1.688488] DMAR: Hardware identity mapping for device > 0000:ff:0c.0 > > > > > > >>[ 1.688490] DMAR: Hardware identity mapping for device > 0000:ff:0c.1 > > > > > > >>[ 1.688491] DMAR: Hardware identity mapping for device > 0000:ff:0c.2 > > > > > > >>[ 1.688493] DMAR: Hardware identity mapping for device > 0000:ff:0c.3 > > > > > > >>[ 1.688494] DMAR: Hardware identity mapping for device > 0000:ff:0c.4 > > > > > > >>[ 1.688496] DMAR: Hardware identity mapping for device > 0000:ff:0c.5 > > > > > > >>[ 1.688498] DMAR: Hardware identity mapping for device > 0000:ff:0c.6 > > > > > > >>[ 1.688499] DMAR: Hardware identity mapping for device > 0000:ff:0c.7 > > > > > > >>[ 1.688500] DMAR: Hardware identity mapping for device > 0000:ff:0d.0 > > > > > > >>[ 1.688502] DMAR: Hardware identity mapping for device > 0000:ff:0d.1 > > > > > > >>[ 1.688504] DMAR: Hardware identity mapping for device > 0000:ff:0d.2 > > > > > > >>[ 1.688505] DMAR: Hardware identity mapping for device > 0000:ff:0d.3 > > > > > > >>[ 1.688506] DMAR: Hardware identity mapping for device > 0000:ff:0d.4 > > > > > > >>[ 1.688508] DMAR: Hardware identity mapping for device > 0000:ff:0d.5 > > > > > > >>[ 1.688509] DMAR: Hardware identity mapping for device > 0000:ff:0d.6 > > > > > > >>[ 1.688510] DMAR: Hardware identity mapping for device > 0000:ff:0d.7 > > > > > > >>[ 1.688511] DMAR: Hardware identity mapping for device > 0000:ff:0e.0 > > > > > > >>[ 1.688513] DMAR: Hardware identity mapping for device > 0000:ff:0e.1 > > > > > > >>[ 1.688514] DMAR: Hardware identity mapping for device > 0000:ff:0f.0 > > > > > > >>[ 1.688516] DMAR: Hardware identity mapping for device > 0000:ff:0f.1 > > > > > > >>[ 1.688517] DMAR: Hardware identity mapping for device > 0000:ff:0f.2 > > > > > > >>[ 1.688518] DMAR: Hardware identity mapping for device > 0000:ff:0f.3 > > > > > > >>[ 1.688520] DMAR: Hardware identity mapping for device > 0000:ff:0f.4 > > > > > > >>[ 1.688521] DMAR: Hardware identity mapping for device > 0000:ff:0f.5 > > > > > > >>[ 1.688523] DMAR: Hardware identity mapping for device > 0000:ff:0f.6 > > > > > > >>[ 1.688524] DMAR: Hardware identity mapping for device > 0000:ff:10.0 > > > > > > >>[ 1.688526] DMAR: Hardware identity mapping for device > 0000:ff:10.1 > > > > > > >>[ 1.688527] DMAR: Hardware identity mapping for device > 0000:ff:10.5 > > > > > > >>[ 1.688529] DMAR: Hardware identity mapping for device > 0000:ff:10.6 > > > > > > >>[ 1.688530] DMAR: Hardware identity mapping for device > 0000:ff:10.7 > > > > > > >>[ 1.688532] DMAR: Hardware identity mapping for device > 0000:ff:12.0 > > > > > > >>[ 1.688533] DMAR: Hardware identity mapping for device > 0000:ff:12.1 > > > > > > >>[ 1.688534] DMAR: Hardware identity mapping for device > 0000:ff:12.4 > > > > > > >>[ 1.688536] DMAR: Hardware identity mapping for device > 0000:ff:12.5 > > > > > > >>[ 1.688537] DMAR: Hardware identity mapping for device > 0000:ff:13.0 > > > > > > >>[ 1.688539] DMAR: Hardware identity mapping for device > 0000:ff:13.1 > > > > > > >>[ 1.688541] DMAR: Hardware identity mapping for device > 0000:ff:13.2 > > > > > > >>[ 1.688542] DMAR: Hardware identity mapping for device > 0000:ff:13.3 > > > > > > >>[ 1.688543] DMAR: Hardware identity mapping for device > 0000:ff:13.6 > > > > > > >>[ 1.688545] DMAR: Hardware identity mapping for device > 0000:ff:13.7 > > > > > > >>[ 1.688546] DMAR: Hardware identity mapping for device > 0000:ff:14.0 > > > > > > >>[ 1.688547] DMAR: Hardware identity mapping for device > 0000:ff:14.1 > > > > > > >>[ 1.688549] DMAR: Hardware identity mapping for device > 0000:ff:14.2 > > > > > > >>[ 1.688550] DMAR: Hardware identity mapping for device > 0000:ff:14.3 > > > > > > >>[ 1.688552] DMAR: Hardware identity mapping for device > 0000:ff:14.4 > > > > > > >>[ 1.688553] DMAR: Hardware identity mapping for device > 0000:ff:14.5 > > > > > > >>[ 1.688559] DMAR: Hardware identity mapping for device > 0000:ff:14.6 > > > > > > >>[ 1.688561] DMAR: Hardware identity mapping for device > 0000:ff:14.7 > > > > > > >>[ 1.688562] DMAR: Hardware identity mapping for device > 0000:ff:16.0 > > > > > > >>[ 1.688564] DMAR: Hardware identity mapping for device > 0000:ff:16.1 > > > > > > >>[ 1.688565] DMAR: Hardware identity mapping for device > 0000:ff:16.2 > > > > > > >>[ 1.688566] DMAR: Hardware identity mapping for device > 0000:ff:16.3 > > > > > > >>[ 1.688568] DMAR: Hardware identity mapping for device > 0000:ff:16.6 > > > > > > >>[ 1.688569] DMAR: Hardware identity mapping for device > 0000:ff:16.7 > > > > > > >>[ 1.688571] DMAR: Hardware identity mapping for device > 0000:ff:17.0 > > > > > > >>[ 1.688573] DMAR: Hardware identity mapping for device > 0000:ff:17.1 > > > > > > >>[ 1.688574] DMAR: Hardware identity mapping for device > 0000:ff:17.2 > > > > > > >>[ 1.688576] DMAR: Hardware identity mapping for device > 0000:ff:17.3 > > > > > > >>[ 1.688577] DMAR: Hardware identity mapping for device > 0000:ff:17.4 > > > > > > >>[ 1.688579] DMAR: Hardware identity mapping for device > 0000:ff:17.5 > > > > > > >>[ 1.688581] DMAR: Hardware identity mapping for device > 0000:ff:17.6 > > > > > > >>[ 1.688582] DMAR: Hardware identity mapping for device > 0000:ff:17.7 > > > > > > >>[ 1.688584] DMAR: Hardware identity mapping for device > 0000:ff:1e.0 > > > > > > >>[ 1.688586] DMAR: Hardware identity mapping for device > 0000:ff:1e.1 > > > > > > >>[ 1.688588] DMAR: Hardware identity mapping for device > 0000:ff:1e.2 > > > > > > >>[ 1.688590] DMAR: Hardware identity mapping for device > 0000:ff:1e.3 > > > > > > >>[ 1.688592] DMAR: Hardware identity mapping for device > 0000:ff:1e.4 > > > > > > >>[ 1.688594] DMAR: Hardware identity mapping for device > 0000:ff:1f.0 > > > > > > >>[ 1.688596] DMAR: Hardware identity mapping for device > 0000:ff:1f.2 > > > > > > >>[ 1.688610] DMAR: Hardware identity mapping for device > 0000:7f:08.0 > > > > > > >>[ 1.688612] DMAR: Hardware identity mapping for device > 0000:7f:08.2 > > > > > > >>[ 1.688614] DMAR: Hardware identity mapping for device > 0000:7f:08.3 > > > > > > >>[ 1.688616] DMAR: Hardware identity mapping for device > 0000:7f:09.0 > > > > > > >>[ 1.688617] DMAR: Hardware identity mapping for device > 0000:7f:09.2 > > > > > > >>[ 1.688619] DMAR: Hardware identity mapping for device > 0000:7f:09.3 > > > > > > >>[ 1.688621] DMAR: Hardware identity mapping for device > 0000:7f:0b.0 > > > > > > >>[ 1.688623] DMAR: Hardware identity mapping for device > 0000:7f:0b.1 > > > > > > >>[ 1.688625] DMAR: Hardware identity mapping for device > 0000:7f:0b.2 > > > > > > >>[ 1.688627] DMAR: Hardware identity mapping for device > 0000:7f:0c.0 > > > > > > >>[ 1.688629] DMAR: Hardware identity mapping for device > 0000:7f:0c.1 > > > > > > >>[ 1.688631] DMAR: Hardware identity mapping for device > 0000:7f:0c.2 > > > > > > >>[ 1.688632] DMAR: Hardware identity mapping for device > 0000:7f:0c.3 > > > > > > >>[ 1.688634] DMAR: Hardware identity mapping for device > 0000:7f:0c.4 > > > > > > >>[ 1.688636] DMAR: Hardware identity mapping for device > 0000:7f:0c.5 > > > > > > >>[ 1.688637] DMAR: Hardware identity mapping for device > 0000:7f:0c.6 > > > > > > >>[ 1.688639] DMAR: Hardware identity mapping for device > 0000:7f:0c.7 > > > > > > >>[ 1.688641] DMAR: Hardware identity mapping for device > 0000:7f:0d.0 > > > > > > >>[ 1.688642] DMAR: Hardware identity mapping for device > 0000:7f:0d.1 > > > > > > >>[ 1.688644] DMAR: Hardware identity mapping for device > 0000:7f:0d.2 > > > > > > >>[ 1.688646] DMAR: Hardware identity mapping for device > 0000:7f:0d.3 > > > > > > >>[ 1.688647] DMAR: Hardware identity mapping for device > 0000:7f:0d.4 > > > > > > >>[ 1.688649] DMAR: Hardware identity mapping for device > 0000:7f:0d.5 > > > > > > >>[ 1.688650] DMAR: Hardware identity mapping for device > 0000:7f:0d.6 > > > > > > >>[ 1.688652] DMAR: Hardware identity mapping for device > 0000:7f:0d.7 > > > > > > >>[ 1.688653] DMAR: Hardware identity mapping for device > 0000:7f:0e.0 > > > > > > >>[ 1.688655] DMAR: Hardware identity mapping for device > 0000:7f:0e.1 > > > > > > >>[ 1.688657] DMAR: Hardware identity mapping for device > 0000:7f:0f.0 > > > > > > >>[ 1.688658] DMAR: Hardware identity mapping for device > 0000:7f:0f.1 > > > > > > >>[ 1.688660] DMAR: Hardware identity mapping for device > 0000:7f:0f.2 > > > > > > >>[ 1.688661] DMAR: Hardware identity mapping for device > 0000:7f:0f.3 > > > > > > >>[ 1.688663] DMAR: Hardware identity mapping for device > 0000:7f:0f.4 > > > > > > >>[ 1.688664] DMAR: Hardware identity mapping for device > 0000:7f:0f.5 > > > > > > >>[ 1.688666] DMAR: Hardware identity mapping for device > 0000:7f:0f.6 > > > > > > >>[ 1.688668] DMAR: Hardware identity mapping for device > 0000:7f:10.0 > > > > > > >>[ 1.688669] DMAR: Hardware identity mapping for device > 0000:7f:10.1 > > > > > > >>[ 1.688671] DMAR: Hardware identity mapping for device > 0000:7f:10.5 > > > > > > >>[ 1.688673] DMAR: Hardware identity mapping for device > 0000:7f:10.6 > > > > > > >>[ 1.688674] DMAR: Hardware identity mapping for device > 0000:7f:10.7 > > > > > > >>[ 1.688676] DMAR: Hardware identity mapping for device > 0000:7f:12.0 > > > > > > >>[ 1.688678] DMAR: Hardware identity mapping for device > 0000:7f:12.1 > > > > > > >>[ 1.688684] DMAR: Hardware identity mapping for device > 0000:7f:12.4 > > > > > > >>[ 1.688686] DMAR: Hardware identity mapping for device > 0000:7f:12.5 > > > > > > >>[ 1.688688] DMAR: Hardware identity mapping for device > 0000:7f:13.0 > > > > > > >>[ 1.688689] DMAR: Hardware identity mapping for device > 0000:7f:13.1 > > > > > > >>[ 1.688691] DMAR: Hardware identity mapping for device > 0000:7f:13.2 > > > > > > >>[ 1.688692] DMAR: Hardware identity mapping for device > 0000:7f:13.3 > > > > > > >>[ 1.688694] DMAR: Hardware identity mapping for device > 0000:7f:13.6 > > > > > > >>[ 1.688696] DMAR: Hardware identity mapping for device > 0000:7f:13.7 > > > > > > >>[ 1.688697] DMAR: Hardware identity mapping for device > 0000:7f:14.0 > > > > > > >>[ 1.688699] DMAR: Hardware identity mapping for device > 0000:7f:14.1 > > > > > > >>[ 1.688701] DMAR: Hardware identity mapping for device > 0000:7f:14.2 > > > > > > >>[ 1.688703] DMAR: Hardware identity mapping for device > 0000:7f:14.3 > > > > > > >>[ 1.688704] DMAR: Hardware identity mapping for device > 0000:7f:14.4 > > > > > > >>[ 1.688706] DMAR: Hardware identity mapping for device > 0000:7f:14.5 > > > > > > >>[ 1.688708] DMAR: Hardware identity mapping for device > 0000:7f:14.6 > > > > > > >>[ 1.688710] DMAR: Hardware identity mapping for device > 0000:7f:14.7 > > > > > > >>[ 1.688712] DMAR: Hardware identity mapping for device > 0000:7f:16.0 > > > > > > >>[ 1.688714] DMAR: Hardware identity mapping for device > 0000:7f:16.1 > > > > > > >>[ 1.688716] DMAR: Hardware identity mapping for device > 0000:7f:16.2 > > > > > > >>[ 1.688718] DMAR: Hardware identity mapping for device > 0000:7f:16.3 > > > > > > >>[ 1.688720] DMAR: Hardware identity mapping for device > 0000:7f:16.6 > > > > > > >>[ 1.688722] DMAR: Hardware identity mapping for device > 0000:7f:16.7 > > > > > > >>[ 1.688724] DMAR: Hardware identity mapping for device > 0000:7f:17.0 > > > > > > >>[ 1.688726] DMAR: Hardware identity mapping for device > 0000:7f:17.1 > > > > > > >>[ 1.688729] DMAR: Hardware identity mapping for device > 0000:7f:17.2 > > > > > > >>[ 1.688731] DMAR: Hardware identity mapping for device > 0000:7f:17.3 > > > > > > >>[ 1.688733] DMAR: Hardware identity mapping for device > 0000:7f:17.4 > > > > > > >>[ 1.688735] DMAR: Hardware identity mapping for device > 0000:7f:17.5 > > > > > > >>[ 1.688737] DMAR: Hardware identity mapping for device > 0000:7f:17.6 > > > > > > >>[ 1.688739] DMAR: Hardware identity mapping for device > 0000:7f:17.7 > > > > > > >>[ 1.688741] DMAR: Hardware identity mapping for device > 0000:7f:1e.0 > > > > > > >>[ 1.688742] DMAR: Hardware identity mapping for device > 0000:7f:1e.1 > > > > > > >>[ 1.688744] DMAR: Hardware identity mapping for device > 0000:7f:1e.2 > > > > > > >>[ 1.688746] DMAR: Hardware identity mapping for device > 0000:7f:1e.3 > > > > > > >>[ 1.688748] DMAR: Hardware identity mapping for device > 0000:7f:1e.4 > > > > > > >>[ 1.688750] DMAR: Hardware identity mapping for device > 0000:7f:1f.0 > > > > > > >>[ 1.688752] DMAR: Hardware identity mapping for device > 0000:7f:1f.2 > > > > > > >>[ 1.688769] DMAR: Hardware identity mapping for device > > 0000:00:00.0 > > > > > > >>[ 1.688774] DMAR: Hardware identity mapping for device > > 0000:00:01.0 > > > > > > >>[ 1.688778] DMAR: Hardware identity mapping for device > > 0000:00:01.1 > > > > > > >>[ 1.688783] DMAR: Hardware identity mapping for device > > 0000:00:03.0 > > > > > > >>[ 1.688785] DMAR: Hardware identity mapping for device > > 0000:00:04.0 > > > > > > >>[ 1.688787] DMAR: Hardware identity mapping for device > > 0000:00:04.1 > > > > > > >>[ 1.688790] DMAR: Hardware identity mapping for device > > 0000:00:04.2 > > > > > > >>[ 1.688792] DMAR: Hardware identity mapping for device > > 0000:00:04.3 > > > > > > >>[ 1.688794] DMAR: Hardware identity mapping for device > > 0000:00:04.4 > > > > > > >>[ 1.688797] DMAR: Hardware identity mapping for device > > 0000:00:04.5 > > > > > > >>[ 1.688799] DMAR: Hardware identity mapping for device > > 0000:00:04.6 > > > > > > >>[ 1.688801] DMAR: Hardware identity mapping for device > > 0000:00:04.7 > > > > > > >>[ 1.688803] DMAR: Hardware identity mapping for device > > 0000:00:05.0 > > > > > > >>[ 1.688807] DMAR: Hardware identity mapping for device > > 0000:00:05.1 > > > > > > >>[ 1.688809] DMAR: Hardware identity mapping for device > > 0000:00:05.2 > > > > > > >>[ 1.688811] DMAR: Hardware identity mapping for device > > 0000:00:05.4 > > > > > > >>[ 1.688814] DMAR: Hardware identity mapping for device > > 0000:00:11.0 > > > > > > >>[ 1.688816] DMAR: Hardware identity mapping for device > > 0000:00:11.4 > > > > > > >>[ 1.688818] DMAR: Hardware identity mapping for device > > 0000:00:14.0 > > > > > > >>[ 1.688821] DMAR: Hardware identity mapping for device > > 0000:00:16.0 > > > > > > >>[ 1.688823] DMAR: Hardware identity mapping for device > > 0000:00:16.1 > > > > > > >>[ 1.688825] DMAR: Hardware identity mapping for device > > 0000:00:1a.0 > > > > > > >>[ 1.688828] DMAR: Hardware identity mapping for device > 0000:00:1c.0 > > > > > > >>[ 1.688831] DMAR: Hardware identity mapping for device > 0000:00:1c.2 > > > > > > >>[ 1.688833] DMAR: Hardware identity mapping for device > > 0000:00:1d.0 > > > > > > >>[ 1.688835] DMAR: Hardware identity mapping for device > 0000:00:1f.0 > > > > > > >>[ 1.688837] DMAR: Hardware identity mapping for device > 0000:00:1f.2 > > > > > > >>[ 1.688844] DMAR: Hardware identity mapping for device > 0000:00:1f.3 > > > > > > >>[ 1.688846] DMAR: Hardware identity mapping for device > 0000:00:1f.6 > > > > > > >>[ 1.688864] DMAR: Hardware identity mapping for device > > 0000:01:00.0 > > > > > > >>[ 1.688870] DMAR: Hardware identity mapping for device > > 0000:01:00.1 > > > > > > >>[ 1.688876] DMAR: Hardware identity mapping for device > > 0000:01:00.2 > > > > > > >>[ 1.688883] DMAR: Hardware identity mapping for device > > 0000:01:00.3 > > > > > > >>[ 1.688895] DMAR: Hardware identity mapping for device > > 0000:80:01.0 > > > > > > >>[ 1.688899] DMAR: Hardware identity mapping for device > > 0000:80:02.0 > > > > > > >>[ 1.688903] DMAR: Hardware identity mapping for device > > 0000:80:03.0 > > > > > > >>[ 1.688906] DMAR: Hardware identity mapping for device > > 0000:80:04.0 > > > > > > >>[ 1.688908] DMAR: Hardware identity mapping for device > > 0000:80:04.1 > > > > > > >>[ 1.688911] DMAR: Hardware identity mapping for device > > 0000:80:04.2 > > > > > > >>[ 1.688913] DMAR: Hardware identity mapping for device > > 0000:80:04.3 > > > > > > >>[ 1.688915] DMAR: Hardware identity mapping for device > > 0000:80:04.4 > > > > > > >>[ 1.688918] DMAR: Hardware identity mapping for device > > 0000:80:04.5 > > > > > > >>[ 1.688920] DMAR: Hardware identity mapping for device > > 0000:80:04.6 > > > > > > >>[ 1.688923] DMAR: Hardware identity mapping for device > > 0000:80:04.7 > > > > > > >>[ 1.688936] DMAR: Hardware identity mapping for device > > 0000:80:05.0 > > > > > > >>[ 1.688940] DMAR: Hardware identity mapping for device > > 0000:80:05.1 > > > > > > >>[ 1.688943] DMAR: Hardware identity mapping for device > > 0000:80:05.2 > > > > > > >>[ 1.688945] DMAR: Hardware identity mapping for device > > 0000:80:05.4 > > > > > > >>[ 1.688956] DMAR: Hardware identity mapping for device > > 0000:81:00.0 > > > > > > >>[ 1.688961] DMAR: Hardware identity mapping for device > > 0000:81:00.1 > > > > > > >>[ 1.688974] DMAR: Hardware identity mapping for device > > 0000:82:00.0 > > > > > > >>[ 1.688980] DMAR: Hardware identity mapping for device > > 0000:82:00.1 > > > > > > >>[ 1.688986] DMAR: Hardware identity mapping for device > > 0000:82:00.2 > > > > > > >>[ 1.688992] DMAR: Hardware identity mapping for device > > 0000:82:00.3 > > > > > > >>[ 1.689006] DMAR: Hardware identity mapping for device > > 0000:83:00.0 > > > > > > >>[ 1.689012] DMAR: Hardware identity mapping for device > > 0000:83:00.1 > > > > > > >>[ 1.689018] DMAR: Hardware identity mapping for device > > 0000:83:00.2 > > > > > > >>[ 1.689024] DMAR: Hardware identity mapping for device > > 0000:83:00.3 > > > > > > >>[ 1.689025] DMAR: Setting RMRR: > > > > > > >>[ 1.689026] DMAR: Ignoring identity map for HW passthrough > device > > > > > > >>0000:00:14.0 [0x7ba75000 - 0x7ba84fff] > > > > > > >>[ 1.689027] DMAR: Ignoring identity map for HW passthrough > device > > > > > > >>0000:00:1a.0 [0x7ba75000 - 0x7ba84fff] > > > > > > >>[ 1.689028] DMAR: Ignoring identity map for HW passthrough > device > > > > > > >>0000:00:1d.0 [0x7ba75000 - 0x7ba84fff] > > > > > > >>[ 1.689036] DMAR: Prepare 0-16MiB unity mapping for LPC > > > > > > >>[ 1.689036] DMAR: Ignoring identity map for HW passthrough > device > > > > > > >>0000:00:1f.0 [0x0 - 0xffffff] > > > > > > >>[ 1.689040] DMAR: Intel(R) Virtualization Technology for > Directed I/O > > > > > > >>[ 2.299766] DMAR: 32bit 0000:00:11.4 uses non-identity mapping > > > > > > >>[ 2.301941] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping > > > > > > >>[ 2.302679] DMAR: Setting identity map for device 0000:00:1a.0 > > > [0x7ba75000 > > > > > > >- > > > > > > >>0x7ba84fff] > > > > > > >>[ 2.320910] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping > > > > > > >>[ 2.321685] DMAR: Setting identity map for device 0000:00:1d.0 > > > > > > >[0x7ba75000 > > > > > > >>- 0x7ba84fff] > > > > > > >># > > > > > > >> > > > > > > >> > > > > > > >>On Tue, Mar 28, 2017 at 1:46 AM, Bodireddy, Bhanuprakash > > > > > > >><[email protected]> wrote: > > > > > > >>Hello shivaram, > > > > > > >> > > > > > > >>Darrell has already asked if VT-d is enabled? > > > > > > >>Your cmdline options are appropriate as you have iommu > > > > > > >>enabled(intel_iommu=on iommu=pt). > > > > > > >>can you also check the output of below mentioned command to see > > > if VT-d > > > > > > >is > > > > > > >>enabled in BIOS? > > > > > > >> > > > > > > >>$ dmesg | grep -e DMAR -e IOMMU > > > > > > >> > > > > > > >>Regards, > > > > > > >>Bhanuprakash. > > > > > > >> > > > > > > >>From: [email protected] [mailto:ovs-discuss- > > > > > > >>[email protected]] On Behalf Of Shivaram Mysore > > > > > > >>Sent: Tuesday, March 28, 2017 6:25 AM > > > > > > >>To: Darrell Ball <[email protected]> > > > > > > >>Cc: [email protected] > > > > > > >>Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > > > >> > > > > > > >># uname -r > > > > > > >>4.10.0-14-generic > > > > > > >> > > > > > > >># cat /proc/cmdline > > > > > > >>BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic > > > > > > >root=/dev/mapper/intelsdn- > > > > > > >>-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G > > > > > > >>hugepagesz=1G hugepages=4 > > > > > > >> > > > > > > >># uname -a > > > > > > >>Linux intelsdn 4.10.0-14-generic #16-Ubuntu SMP Fri Mar 17 > > > 15:19:26 UTC > > > > > > >2017 > > > > > > >>x86_64 x86_64 x86_64 GNU/Linux > > > > > > >> > > > > > > >># cat /etc/*release > > > > > > >>DISTRIB_ID=Ubuntu > > > > > > >>DISTRIB_RELEASE=17.04 > > > > > > >>DISTRIB_CODENAME=zesty > > > > > > >>DISTRIB_DESCRIPTION="Ubuntu Zesty Zapus (development branch)" > > > > > > >>NAME="Ubuntu" > > > > > > >>VERSION="17.04 (Zesty Zapus)" > > > > > > >>ID=ubuntu > > > > > > >>ID_LIKE=debian > > > > > > >>PRETTY_NAME="Ubuntu Zesty Zapus (development branch)" > > > > > > >>VERSION_ID="17.04" > > > > > > >> > > > > > > >> > > > > > > >>Nothing in the syslog related > > > > > > >> > > > > > > >>On Mon, Mar 27, 2017 at 10:02 PM, Darrell Ball > > > <[email protected]> > > > wrote: > > > > > > >>I see the pastebins now. > > > > > > >> > > > > > > >>What kernel version are you using ? > > > > > > >>VT-d enabled ? > > > > > > >>Anything related in syslog ? > > > > > > >> > > > > > > >>From: Shivaram Mysore <[email protected]> > > > > > > >>Date: Monday, March 27, 2017 at 9:34 PM > > > > > > >>To: Darrell Ball <[email protected]> > > > > > > >>Cc: "[email protected]" <[email protected]> > > > > > > >>Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > > > >> > > > > > > >>One more datapoint, please check the paste bin log in URL sent > > > > > > >earlier. Dpdk- > > > > > > >>devbind works and you can see results in the paste bin log > > > > > > >> > > > > > > >>/Shivaram > > > > > > >>::Sent from my mobile device:: > > > > > > >> > > > > > > >>On Mar 27, 2017, at 8:34 PM, Darrell Ball <[email protected]> > wrote: > > > > > > >>see inline below > > > > > > >> > > > > > > >>But a few questions: > > > > > > >>1) Are you using the recommended version of dpdk with OVS > 2.7 – > > > 16.11 ? > > > > > > >>2) Are you getting a crash still or is this post crash ? > > > > > > >>3) Was the previous crash still happening with 16.11 or > only 17.02 ? > > > > > > >> > > > > > > >>Thanks > > > > > > >>Darrell > > > > > > >> > > > > > > >>From: <[email protected]> on behalf of > > > Shivaram > > > > > > >>Mysore <[email protected]> > > > > > > >>Date: Monday, March 27, 2017 at 7:45 PM > > > > > > >>To: "[email protected]" <[email protected]> > > > > > > >>Subject: [ovs-discuss] Error attaching device to DPDK > > > > > > >> > > > > > > >>Hello, > > > > > > >> > > > > > > >>I have installed OVS 2.7 running. When I do: > > > > > > >>1. # ovs-vsctl add-port ovs-ip64-br0 dpdk-p0 -- set > Interface dpdk-p0 > > > > > > >>type=dpdk options:dpdk-devargs=0000:82:00.0 > > > > > > >>2. ovs-vsctl: Error detected while setting up 'dpdk-p0': > Error > > attaching > > > > > > >>device '0000:82:00.0' to DPDK. See ovs-vswitchd log for details. > > > > > > >> > > > > > > >>Possibly, the port is not bound. > > > > > > >>Check using > > > > > > >>$DPDK_DIR/tools/dpdk-devbind.py –status > > > > > > >>and see the documentation for more detail > > > > > > >> > > > > > > >>and the log file shows: > > > > > > >>1. 2017-03-28T02:04:40.004Z|00064|netdev_dpdk|WARN|Error > > > attaching > > > > > > >>device '0000:82:00.0' to DPDK > > > > > > >>2. 2017-03-28T02:04:40.004Z|00065|netdev|WARN|dpdk-p0: > could > > > > > > >>not set configuration (Invalid argument) > > > > > > >> > > > > > > >>What would cause such a situation? > > > > > > >> > > > > > > >>Commands run: https://urldefense.proofpoint.com/v2/url?u=https- > > > > > 3A__pastebin.com_0P754x0t&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r= > > > BVhFA09CGX7JQ5Ih- > > > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=XEhP0isS > > > M6FAfGw-HXbOfMLfomYnhSo1G2UxvC1qjp8&e= > > > > > > >>Detailed OVS Log: > > > https://urldefense.proofpoint.com/v2/url?u=https- > > > > > 3A__pastebin.com_ctBhLuRG&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r= > > > BVhFA09CGX7JQ5Ih- > > > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=90tudSR > > > NE_HlxrCclX83wTmqlUL5xMPrRha7_-fMAKs&e= > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > discuss mailing list > > > > > > [email protected] > > > > > > https://urldefense.proofpoint.com/v2/url?u=https- > > > 3A__mail.openvswitch.org_mailman_listinfo_ovs- > > > > > 2Ddiscuss&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih > > > -uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=9- > > > oI5HFUKs-b_kDAFcnn2vFwTcuE2xw2SFvd-zE6vWY&e= > > > > > > > > > > > > > > > > > > > >
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
