Also, make sure to use the latest OSv with this commit - https://github.com/cloudius-systems/osv/commit/b4b4829e33400d9c22285f16c42a0119d79851cf.
On Monday, September 13, 2021 at 7:20:41 PM UTC-4 Sergio Inzunza wrote: > Thank you so much Waldek. I will look over these changes. If I have > success with the modern DPDK implementation I will update the group. > > On Monday, September 13, 2021 at 4:08:53 PM UTC-7 [email protected] > wrote: > >> >> https://github.com/cloudius-systems/osv-apps/commit/63a4abde3ad71d3ef160831b9ad0a5efda99a5ec >> >> On Mon, Sep 13, 2021 at 6:57 PM Waldek Kozaczuk <[email protected]> >> wrote: >> >>> I have pushed a commit to a dpdk-example that changes pci_scan_one() to >>> correctly identify PCI devices. It also relies on some changes I made to >>> OSv itself to allow for these changes. >>> >>> Waldek >>> >>> On Mon, Sep 13, 2021 at 18:24 Sergio Inzunza <[email protected]> wrote: >>> >>>> Hi Waldek, >>>> >>>> wow this some interesting stuff! >>>> >>>> I did try setting up the two network bridges as you described a while >>>> back, however I still was not able to run l2fwd. >>>> >>>> I have been working on a modern DPDK implementation for OSv, >>>> specifically for DPDK v20.05. However, I have not been able to test if it >>>> fully works yet. I was testing it on the l2fwd application, but without it >>>> being able to run I had no results. >>>> >>>> The PCI driver was where I probably had the majority of trouble. As the >>>> modern DPDK versions have restructured the driver code rather >>>> significantly. But the functionality for pci_scan_one() and pci_scan() >>>> remained relatively similar. I still however, was casting each dev to >>>> pci::device. I am going to try messing with your new findings over the >>>> next >>>> couple of days. >>>> >>>> Thank you so much! >>>> >>>> Sincerely, >>>> Sergio Inzunza >>>> >>>> On Thursday, September 9, 2021 at 7:38:57 PM UTC-7 [email protected] >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have just found some time to look at it and play with this DPDK app. >>>>> And I have some interesting news to share. >>>>> >>>>> On Sunday, August 29, 2021 at 8:29:45 PM UTC-4 Sergio Inzunza wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I hope everyone is doing great. I was wondering if anyone had any >>>>>> experience with running the dpdk-example image >>>>>> https://github.com/cloudius-systems/osv-apps/tree/184beda56c0bc01f8991ca2abcabacc6181ff57c/dpdk-example >>>>>> >>>>>> I have tried running the image by exporting it to Lib-Virt as stated >>>>>> in the example and then running the l2fwd application. However, I get >>>>>> the >>>>>> following error: >>>>>> >>>>>> EAL: Error - exiting with code: 1 >>>>>> Cause: No Ethernet ports - bye >>>>>> >>>>>> >>>>> First of all, I assume you have managed to setup two bridges - default >>>>> and net2 - as l2fwd requires two NICs. Setting up the net2 two is not >>>>> really described in the app readme and I am planning to update it. >>>>> >>>>> For now, you can follow method 2 in >>>>> https://computingforgeeks.com/how-to-create-and-configure-bridge-networking-for-kvm-in-linux/. >>>>> >>>>> In essence, you can export the default network using virsh, change what >>>>> is >>>>> needed and use virsh to create a new bridge using new xml file. Roughly >>>>> like so: >>>>> >>>>> *virsh* net-dumpxml default >>>>> >>>>> cat net2.xml >>>>> <network> >>>>> <name>net2</name> >>>>> <forward mode='nat'> >>>>> <nat> >>>>> <port start='1024' end='65535'/> >>>>> </nat> >>>>> </forward> >>>>> <bridge name='virbr1' stp='on' delay='0'/> >>>>> <ip address='192.168.123.1' netmask='255.255.255.0'> >>>>> <dhcp> >>>>> <range start='192.168.123.2' end='192.168.123.254'/> >>>>> </dhcp> >>>>> </ip> >>>>> </network> >>>>> >>>>> virsh net-create --file ./net2.xml >>>>> >>>>> I had trouble seeing any output from OSv guest when using virsh to >>>>> actually run OSv as it is documented in README (it would be nice to get >>>>> to >>>>> the bottom of this). But instead, you can use "./script/run.py -n >>>>> --dry-output" to create a shell script and manually add a second device >>>>> for >>>>> virbr1 bridge like so: >>>>> >>>>> ./scripts/run.py -n --dry-run >>>>> /home/wkozaczuk/projects/osv-master/scripts/../scripts/imgedit.py >>>>> setargs /home/wkozaczuk/projects/osv-master/build/last/usr.img >>>>> "--rootfs=zfs --console=serial --verbose --maxnic=0 /l2fwd --no-shconf -c >>>>> f >>>>> -n 2 --log-level 8 -m 768 -- -p 3" >>>>> qemu-system-x86_64 \ >>>>> -m 2G \ >>>>> -smp 4 \ >>>>> -vnc :1 \ >>>>> -gdb tcp::1234,server,nowait \ >>>>> -device virtio-blk-pci,id=blk0,drive=hd0,scsi=off,bootindex=0 \ >>>>> -drive >>>>> file=/home/wkozaczuk/projects/osv-master/build/last/usr.img,if=none,id=hd0,cache=none,aio=native >>>>> >>>>> \ >>>>> -netdev >>>>> bridge,id=hn0,br=virbr0,helper=/usr/lib/qemu/qemu-bridge-helper \ >>>>> -device virtio-net-pci,netdev=hn0,id=nic0 \ >>>>> -device virtio-rng-pci \ >>>>> -enable-kvm \ >>>>> -cpu host,+x2apic \ >>>>> -chardev stdio,mux=on,id=stdio,signal=off \ >>>>> -mon chardev=stdio,mode=readline \ >>>>> -device isa-serial,chardev=stdio >>>>> >>>>> add 2 lines: >>>>> -netdev >>>>> bridge,id=hn1,br=virbr1,helper=/usr/lib/qemu/qemu-bridge-helper \ >>>>> -device virtio-net-pci,netdev=hn1,id=nic1 \ >>>>> >>>>> I was wondering if maybe there is some additional steps to be done >>>>>> that weren't included. Possibly on binding the unikernel nic to the dpdk >>>>>> driver. >>>>>> >>>>> >>>>> But even if you did all this, l2fwd would still not detect any NICs >>>>> just like in our example. >>>>> >>>>> After some digging, I have discovered the problem lies in >>>>> lib/librte_eal/osvapp/eal/eal_pci.cc, where we have logic to discover PCI >>>>> devices that are not attached. In short, the logic there does not work >>>>> due >>>>> to some changes I made when refactoring virtio devices and drivers code >>>>> layer in OSv (see the commit >>>>> https://github.com/cloudius-systems/osv/commit/43777010faf8e3fe3abbfa65c915ce05b2715581) >>>>> >>>>> 2 years ago when doing work to support Firecracker. For more insight, >>>>> read >>>>> the section *Virtio* of the blog article - >>>>> http://blog.osv.io/blog/2019/04/19/making-OSv-run-on-firecraker/. In >>>>> essence, the device objects registered in DeviceManager are not >>>>> necessarily >>>>> instances of pci::device class (they may be instances of >>>>> virtio::virtio_device instead). So this code in eal_pci.cc becomes >>>>> problematic: >>>>> >>>>> >>>>> /* Scan one pci entry, and fill the devices list from it. */ >>>>> static int >>>>> pci_scan_one(hw::hw_device* dev) >>>>> { >>>>> u8 bus, device, func; >>>>> auto pci_dev = static_cast<pci::device*>(dev); >>>>> auto rte_dev = new rte_pci_device(); >>>>> >>>>> /* get bus id, device id, function no */ >>>>> pci_dev->get_bdf(bus, device, func); >>>>> **** >>>>> } >>>>> >>>>> /* >>>>> * Scan the content of the PCI bus, and add the devices in the devices >>>>> * list. Call pci_scan_one() for each pci entry found. >>>>> */ >>>>> static int >>>>> pci_scan(void) >>>>> { >>>>> unsigned dev_count = 0; >>>>> int err = 0; >>>>> >>>>> auto dm = hw::device_manager::instance(); >>>>> dm->for_each_device([&dev_count, &err] (hw::hw_device* dev) { >>>>> if (dev->is_attached()) >>>>> return; >>>>> int ret = pci_scan_one(dev); >>>>> if (ret < 0) { >>>>> err++; >>>>> } else { >>>>> dev_count += ret; >>>>> } >>>>> }); >>>>> >>>>> if (err) >>>>> return -1; >>>>> >>>>> RTE_LOG(ERR, EAL, "PCI scan found %u devices\n", dev_count); >>>>> return 0; >>>>> } >>>>> >>>>> As you can see we call for_each_device() on device_manager and then in >>>>> pci_scan_one() cast each dev to pci::device* which does NOT always work >>>>> (it >>>>> actually never works for devices we are interested - virtio ones). >>>>> The virtio ones registered in device_manager are instances of >>>>> virtio::virtio_device. >>>>> >>>>> I do not have a correct fix for that (most likely it will require some >>>>> changes to OSv interfaces and the DPDK OSv app as well) but be on the >>>>> lookout for some patches. If somebody has good suggestions for some clean >>>>> design solution here, I would appreciate it (how to expose PCI devices to >>>>> DPDK app). >>>>> >>>>> Meanwhile, I hacked things and the app starts correctly: >>>>> >>>>> sudo ./run_with_2_nics.sh >>>>> OSv v0.56.0-3-g985d4d76 >>>>> 4 CPUs detected >>>>> Firmware vendor: SeaBIOS >>>>> console=serial >>>>> bsd: initializing - done >>>>> VFS: mounting ramfs at / >>>>> VFS: mounting devfs at /dev >>>>> net: initializing - done >>>>> -> Device Manager: registered device with ID:305627270 >>>>> -> Device Manager: registered device with ID:1879081094 >>>>> -> Device Manager: registered device with ID:1880129670 >>>>> -> Device Manager: registered device with ID:1897103494 >>>>> -> Device Manager: registered device with ID:286331444 >>>>> -> Device Manager: registered device with ID:137972 >>>>> -> Device Manager: registered device with ID:72436 >>>>> -> Device Manager: registered device with ID:72436 >>>>> -> Device Manager: registered device with ID:269044 >>>>> virtio-blk: Add blk device instances 0 as vblk0, devsize=268435456 >>>>> random: virtio-rng registered as a source. >>>>> random: intel drng, rdrand registered as a source. >>>>> random: <Software, Yarrow> initialized >>>>> Config space of: 00:00.0 >>>>> Config space of: 00:01.0 >>>>> Config space of: 00:01.1 >>>>> Config space of: 00:01.3 >>>>> Config space of: 00:02.0 >>>>> Config space of: 00:03.0 >>>>> Config space of: 00:04.0 >>>>> Config space of: 00:05.0 >>>>> Config space of: 00:06.0 >>>>> random: device unblocked. >>>>> VFS: unmounting /dev >>>>> VFS: mounting zfs at /zfs >>>>> zfs: mounting osv/zfs from device /dev/vblk0.1 >>>>> VFS: mounting devfs at /dev >>>>> VFS: mounting procfs at /proc >>>>> VFS: mounting sysfs at /sys >>>>> BSD shrinker: event handler list found: 0xffffa00000d9d080 >>>>> BSD shrinker found: 1 >>>>> BSD shrinker: unlocked, running >>>>> Booted up in 607.68 ms >>>>> Cmdline: /l2fwd --no-shconf -c f -n 2 --log-level 8 -m 768 -- -p 3 >>>>> >>>>> EAL: Detected lcore 0 as core 0 on socket 0 >>>>> EAL: Detected lcore 1 as core 1 on socket 0 >>>>> EAL: Detected lcore 2 as core 2 on socket 0 >>>>> EAL: Detected lcore 3 as core 3 on socket 0 >>>>> EAL: Support maximum 128 logical core(s) by configuration. >>>>> EAL: Detected 4 lcore(s) >>>>> EAL: pci_scan_one() read BDF: [0:2.0]! >>>>> EAL: bar0 mmio at 0xffff8000fd000000 >>>>> EAL: bar1 not available >>>>> pci_dev:0xffffa00000d8d780 >>>>> EAL: pci_scan_one() read BDF: [0:4.0]! >>>>> EAL: bar0 NON-mmio at 0xc080 >>>>> EAL: bar1 mmio at 0xffff8000feb92000 >>>>> EAL: bar2 not available >>>>> pci_dev:0xffffa00000d8d880 >>>>> EAL: pci_scan_one() read BDF: [0:5.0]! >>>>> EAL: bar0 NON-mmio at 0xc0a0 >>>>> EAL: bar1 mmio at 0xffff8000feb93000 >>>>> EAL: bar2 not available >>>>> pci_dev:0x60002 >>>>> EAL: pci_scan_one() read BDF: [0:1.0]! >>>>> EAL: bar0 not available >>>>> EAL: pci_scan_one() inserted BEFORE >>>>> EAL: pci_scan_one() read BDF: [0:1.1]! >>>>> EAL: bar0 not available >>>>> EAL: pci_scan_one() inserted BEFORE >>>>> EAL: pci_scan_one() read BDF: [0:1.3]! >>>>> EAL: bar0 not available >>>>> EAL: pci_scan_one() inserted BEFORE >>>>> EAL: PCI scan found 6 devices >>>>> EAL: Setting up memory... >>>>> EAL: Mapped memory segment 0 @ 0xffff80013d800000: >>>>> physaddr:0x13d800000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff80013b600000: >>>>> physaddr:0x13b600000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800139400000: >>>>> physaddr:0x139400000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800137200000: >>>>> physaddr:0x137200000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800135000000: >>>>> physaddr:0x135000000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800132e00000: >>>>> physaddr:0x132e00000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800130c00000: >>>>> physaddr:0x130c00000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff80012ea00000: >>>>> physaddr:0x12ea00000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff80012c800000: >>>>> physaddr:0x12c800000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff80012a600000: >>>>> physaddr:0x12a600000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800128400000: >>>>> physaddr:0x128400000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800126200000: >>>>> physaddr:0x126200000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800124000000: >>>>> physaddr:0x124000000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800121e00000: >>>>> physaddr:0x121e00000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff80011fc00000: >>>>> physaddr:0x11fc00000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff80011da00000: >>>>> physaddr:0x11da00000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff80011b800000: >>>>> physaddr:0x11b800000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800119600000: >>>>> physaddr:0x119600000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800117400000: >>>>> physaddr:0x117400000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800115200000: >>>>> physaddr:0x115200000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800113000000: >>>>> physaddr:0x113000000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff800110e00000: >>>>> physaddr:0x110e00000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff80010ec00000: >>>>> physaddr:0x10ec00000, len 33554432 >>>>> EAL: Mapped memory segment 0 @ 0xffff8000bde00000: >>>>> physaddr:0xbde00000, len 33554432 >>>>> EAL: TSC frequency is ~0 KHz >>>>> EAL: Master lcore 0 is ready (tid=d8e400;cpuset=[0]) >>>>> PMD: ENICPMD trace: rte_enic_pmd_init >>>>> EAL: lcore 1 is ready (tid=d8e430;cpuset=[1]) >>>>> EAL: lcore 2 is ready (tid=d8e440;cpuset=[2]) >>>>> EAL: lcore 3 is ready (tid=d8e420;cpuset=[3]) >>>>> EAL: PCI device 0000:00:04.0 on NUMA socket -1 >>>>> EAL: probe driver: 1af4:1000 rte_virtio_pmd >>>>> EAL: PCI device 0000:00:05.0 on NUMA socket -1 >>>>> EAL: probe driver: 1af4:1000 rte_virtio_pmd >>>>> Lcore 0: RX port 0 >>>>> Lcore 1: RX port 1 >>>>> Initializing port 0... done: >>>>> Port 0, MAC address: 52:54:00:12:34:56 >>>>> >>>>> Initializing port 1... done: >>>>> Port 1, MAC address: 52:54:00:12:34:57 >>>>> >>>>> >>>>> Checking link statusdone >>>>> Port 0 Link Up - speed 10000 Mbps - full-duplex >>>>> Port 1 Link Up - speed 10000 Mbps - full-duplex >>>>> L2FWD: entering main loop on lcore 1 >>>>> L2FWD: entering main loop on lcore 0 >>>>> L2FWD: lcore 3 has nothing to do >>>>> L2FWD: lcore 2 has nothing to do >>>>> L2FWD: -- lcoreid=1 portid=1 >>>>> L2FWD: -- lcoreid=0 portid=0 >>>>> >>>>> >>>>> Port statistics ==================================== >>>>> Statistics for port 0 ------------------------------ >>>>> Packets sent: 22 >>>>> Packets received: 14 >>>>> Packets dropped: 0 >>>>> Statistics for port 1 ------------------------------ >>>>> Packets sent: 14 >>>>> Packets received: 22 >>>>> Packets dropped: 0 >>>>> Statistics for port 2 ------------------------------ >>>>> ... >>>>> >>>>> Waldek >>>>> >>>>> PS. Have you had a chance to update the app >>>>> https://github.com/syuu1228/osv-dpdk/tree/osv-head to newest DPDK? >>>>> >>>>> >>>>> >>>>>> Thanks in advance, >>>>>> Sergio Inzunza >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "OSv Development" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/osv-dev/07ed352e-315a-4379-ae20-3072e5a1cc47n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/osv-dev/07ed352e-315a-4379-ae20-3072e5a1cc47n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/d31a010a-b87a-4e10-80ef-5452e50439d5n%40googlegroups.com.
