Hi, I have pushed some commits to address this issue.
Waldek On Thursday, September 9, 2021 at 10:38:57 PM UTC-4 Waldek Kozaczuk wrote: > Hi, > > I have just found some time to look at it and play with this DPDK app. And > I have some interesting news to share. > > On Sunday, August 29, 2021 at 8:29:45 PM UTC-4 Sergio Inzunza wrote: > >> Hello, >> >> I hope everyone is doing great. I was wondering if anyone had any >> experience with running the dpdk-example image >> https://github.com/cloudius-systems/osv-apps/tree/184beda56c0bc01f8991ca2abcabacc6181ff57c/dpdk-example >> >> I have tried running the image by exporting it to Lib-Virt as stated in >> the example and then running the l2fwd application. However, I get the >> following error: >> >> EAL: Error - exiting with code: 1 >> Cause: No Ethernet ports - bye >> >> > First of all, I assume you have managed to setup two bridges - default and > net2 - as l2fwd requires two NICs. Setting up the net2 two is not really > described in the app readme and I am planning to update it. > > For now, you can follow method 2 in > https://computingforgeeks.com/how-to-create-and-configure-bridge-networking-for-kvm-in-linux/. > > In essence, you can export the default network using virsh, change what is > needed and use virsh to create a new bridge using new xml file. Roughly > like so: > > *virsh* net-dumpxml default > > cat net2.xml > <network> > <name>net2</name> > <forward mode='nat'> > <nat> > <port start='1024' end='65535'/> > </nat> > </forward> > <bridge name='virbr1' stp='on' delay='0'/> > <ip address='192.168.123.1' netmask='255.255.255.0'> > <dhcp> > <range start='192.168.123.2' end='192.168.123.254'/> > </dhcp> > </ip> > </network> > > virsh net-create --file ./net2.xml > > I had trouble seeing any output from OSv guest when using virsh to > actually run OSv as it is documented in README (it would be nice to get to > the bottom of this). But instead, you can use "./script/run.py -n > --dry-output" to create a shell script and manually add a second device for > virbr1 bridge like so: > > ./scripts/run.py -n --dry-run > /home/wkozaczuk/projects/osv-master/scripts/../scripts/imgedit.py setargs > /home/wkozaczuk/projects/osv-master/build/last/usr.img "--rootfs=zfs > --console=serial --verbose --maxnic=0 /l2fwd --no-shconf -c f -n 2 > --log-level 8 -m 768 -- -p 3" > qemu-system-x86_64 \ > -m 2G \ > -smp 4 \ > -vnc :1 \ > -gdb tcp::1234,server,nowait \ > -device virtio-blk-pci,id=blk0,drive=hd0,scsi=off,bootindex=0 \ > -drive > file=/home/wkozaczuk/projects/osv-master/build/last/usr.img,if=none,id=hd0,cache=none,aio=native > > \ > -netdev bridge,id=hn0,br=virbr0,helper=/usr/lib/qemu/qemu-bridge-helper \ > -device virtio-net-pci,netdev=hn0,id=nic0 \ > -device virtio-rng-pci \ > -enable-kvm \ > -cpu host,+x2apic \ > -chardev stdio,mux=on,id=stdio,signal=off \ > -mon chardev=stdio,mode=readline \ > -device isa-serial,chardev=stdio > > add 2 lines: > -netdev bridge,id=hn1,br=virbr1,helper=/usr/lib/qemu/qemu-bridge-helper \ > -device virtio-net-pci,netdev=hn1,id=nic1 \ > > I was wondering if maybe there is some additional steps to be done that >> weren't included. Possibly on binding the unikernel nic to the dpdk driver. >> > > But even if you did all this, l2fwd would still not detect any NICs just > like in our example. > > After some digging, I have discovered the problem lies in > lib/librte_eal/osvapp/eal/eal_pci.cc, where we have logic to discover PCI > devices that are not attached. In short, the logic there does not work due > to some changes I made when refactoring virtio devices and drivers code > layer in OSv (see the commit > https://github.com/cloudius-systems/osv/commit/43777010faf8e3fe3abbfa65c915ce05b2715581) > > 2 years ago when doing work to support Firecracker. For more insight, read > the section *Virtio* of the blog article - > http://blog.osv.io/blog/2019/04/19/making-OSv-run-on-firecraker/. In > essence, the device objects registered in DeviceManager are not necessarily > instances of pci::device class (they may be instances of > virtio::virtio_device instead). So this code in eal_pci.cc becomes > problematic: > > > /* Scan one pci entry, and fill the devices list from it. */ > static int > pci_scan_one(hw::hw_device* dev) > { > u8 bus, device, func; > auto pci_dev = static_cast<pci::device*>(dev); > auto rte_dev = new rte_pci_device(); > > /* get bus id, device id, function no */ > pci_dev->get_bdf(bus, device, func); > **** > } > > /* > * Scan the content of the PCI bus, and add the devices in the devices > * list. Call pci_scan_one() for each pci entry found. > */ > static int > pci_scan(void) > { > unsigned dev_count = 0; > int err = 0; > > auto dm = hw::device_manager::instance(); > dm->for_each_device([&dev_count, &err] (hw::hw_device* dev) { > if (dev->is_attached()) > return; > int ret = pci_scan_one(dev); > if (ret < 0) { > err++; > } else { > dev_count += ret; > } > }); > > if (err) > return -1; > > RTE_LOG(ERR, EAL, "PCI scan found %u devices\n", dev_count); > return 0; > } > > As you can see we call for_each_device() on device_manager and then in > pci_scan_one() cast each dev to pci::device* which does NOT always work (it > actually never works for devices we are interested - virtio ones). > The virtio ones registered in device_manager are instances of > virtio::virtio_device. > > I do not have a correct fix for that (most likely it will require some > changes to OSv interfaces and the DPDK OSv app as well) but be on the > lookout for some patches. If somebody has good suggestions for some clean > design solution here, I would appreciate it (how to expose PCI devices to > DPDK app). > > Meanwhile, I hacked things and the app starts correctly: > > sudo ./run_with_2_nics.sh > OSv v0.56.0-3-g985d4d76 > 4 CPUs detected > Firmware vendor: SeaBIOS > console=serial > bsd: initializing - done > VFS: mounting ramfs at / > VFS: mounting devfs at /dev > net: initializing - done > -> Device Manager: registered device with ID:305627270 > -> Device Manager: registered device with ID:1879081094 > -> Device Manager: registered device with ID:1880129670 > -> Device Manager: registered device with ID:1897103494 > -> Device Manager: registered device with ID:286331444 > -> Device Manager: registered device with ID:137972 > -> Device Manager: registered device with ID:72436 > -> Device Manager: registered device with ID:72436 > -> Device Manager: registered device with ID:269044 > virtio-blk: Add blk device instances 0 as vblk0, devsize=268435456 > random: virtio-rng registered as a source. > random: intel drng, rdrand registered as a source. > random: <Software, Yarrow> initialized > Config space of: 00:00.0 > Config space of: 00:01.0 > Config space of: 00:01.1 > Config space of: 00:01.3 > Config space of: 00:02.0 > Config space of: 00:03.0 > Config space of: 00:04.0 > Config space of: 00:05.0 > Config space of: 00:06.0 > random: device unblocked. > VFS: unmounting /dev > VFS: mounting zfs at /zfs > zfs: mounting osv/zfs from device /dev/vblk0.1 > VFS: mounting devfs at /dev > VFS: mounting procfs at /proc > VFS: mounting sysfs at /sys > BSD shrinker: event handler list found: 0xffffa00000d9d080 > BSD shrinker found: 1 > BSD shrinker: unlocked, running > Booted up in 607.68 ms > Cmdline: /l2fwd --no-shconf -c f -n 2 --log-level 8 -m 768 -- -p 3 > > EAL: Detected lcore 0 as core 0 on socket 0 > EAL: Detected lcore 1 as core 1 on socket 0 > EAL: Detected lcore 2 as core 2 on socket 0 > EAL: Detected lcore 3 as core 3 on socket 0 > EAL: Support maximum 128 logical core(s) by configuration. > EAL: Detected 4 lcore(s) > EAL: pci_scan_one() read BDF: [0:2.0]! > EAL: bar0 mmio at 0xffff8000fd000000 > EAL: bar1 not available > pci_dev:0xffffa00000d8d780 > EAL: pci_scan_one() read BDF: [0:4.0]! > EAL: bar0 NON-mmio at 0xc080 > EAL: bar1 mmio at 0xffff8000feb92000 > EAL: bar2 not available > pci_dev:0xffffa00000d8d880 > EAL: pci_scan_one() read BDF: [0:5.0]! > EAL: bar0 NON-mmio at 0xc0a0 > EAL: bar1 mmio at 0xffff8000feb93000 > EAL: bar2 not available > pci_dev:0x60002 > EAL: pci_scan_one() read BDF: [0:1.0]! > EAL: bar0 not available > EAL: pci_scan_one() inserted BEFORE > EAL: pci_scan_one() read BDF: [0:1.1]! > EAL: bar0 not available > EAL: pci_scan_one() inserted BEFORE > EAL: pci_scan_one() read BDF: [0:1.3]! > EAL: bar0 not available > EAL: pci_scan_one() inserted BEFORE > EAL: PCI scan found 6 devices > EAL: Setting up memory... > EAL: Mapped memory segment 0 @ 0xffff80013d800000: physaddr:0x13d800000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff80013b600000: physaddr:0x13b600000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800139400000: physaddr:0x139400000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800137200000: physaddr:0x137200000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800135000000: physaddr:0x135000000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800132e00000: physaddr:0x132e00000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800130c00000: physaddr:0x130c00000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff80012ea00000: physaddr:0x12ea00000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff80012c800000: physaddr:0x12c800000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff80012a600000: physaddr:0x12a600000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800128400000: physaddr:0x128400000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800126200000: physaddr:0x126200000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800124000000: physaddr:0x124000000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800121e00000: physaddr:0x121e00000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff80011fc00000: physaddr:0x11fc00000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff80011da00000: physaddr:0x11da00000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff80011b800000: physaddr:0x11b800000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800119600000: physaddr:0x119600000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800117400000: physaddr:0x117400000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800115200000: physaddr:0x115200000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800113000000: physaddr:0x113000000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff800110e00000: physaddr:0x110e00000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff80010ec00000: physaddr:0x10ec00000, > len 33554432 > EAL: Mapped memory segment 0 @ 0xffff8000bde00000: physaddr:0xbde00000, > len 33554432 > EAL: TSC frequency is ~0 KHz > EAL: Master lcore 0 is ready (tid=d8e400;cpuset=[0]) > PMD: ENICPMD trace: rte_enic_pmd_init > EAL: lcore 1 is ready (tid=d8e430;cpuset=[1]) > EAL: lcore 2 is ready (tid=d8e440;cpuset=[2]) > EAL: lcore 3 is ready (tid=d8e420;cpuset=[3]) > EAL: PCI device 0000:00:04.0 on NUMA socket -1 > EAL: probe driver: 1af4:1000 rte_virtio_pmd > EAL: PCI device 0000:00:05.0 on NUMA socket -1 > EAL: probe driver: 1af4:1000 rte_virtio_pmd > Lcore 0: RX port 0 > Lcore 1: RX port 1 > Initializing port 0... done: > Port 0, MAC address: 52:54:00:12:34:56 > > Initializing port 1... done: > Port 1, MAC address: 52:54:00:12:34:57 > > > Checking link statusdone > Port 0 Link Up - speed 10000 Mbps - full-duplex > Port 1 Link Up - speed 10000 Mbps - full-duplex > L2FWD: entering main loop on lcore 1 > L2FWD: entering main loop on lcore 0 > L2FWD: lcore 3 has nothing to do > L2FWD: lcore 2 has nothing to do > L2FWD: -- lcoreid=1 portid=1 > L2FWD: -- lcoreid=0 portid=0 > > > Port statistics ==================================== > Statistics for port 0 ------------------------------ > Packets sent: 22 > Packets received: 14 > Packets dropped: 0 > Statistics for port 1 ------------------------------ > Packets sent: 14 > Packets received: 22 > Packets dropped: 0 > Statistics for port 2 ------------------------------ > ... > > Waldek > > PS. Have you had a chance to update the app > https://github.com/syuu1228/osv-dpdk/tree/osv-head to newest DPDK? > > > >> Thanks in advance, >> Sergio Inzunza >> > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/6d0c048f-057d-405d-8264-2b290385a6b2n%40googlegroups.com.
