Iscsi booting off a virtual function using ixgbevf works correctly during the first boot, but after booting into a guest os running linux and hard rebooting, the guest will no longer iscsi boot. I tracked the issue down to an issue with the reset of the virtual function. Reset does not actually reset the memory addresses for the vf queues. Since the linux driver sets up multiple send/receive queues, and the ipxe code only sets up one, the packets get copied to random memory addresses that ipxe does not know about.
You can see in the code that handles a vf reset[1] in the ixgbe driver that it doesn't actually reset the send and receive queues. Additionally there isn't a reliable way to reset these settings from userspace on the host side. The best I could do was force the adapter to reset completely using an ethtool command[2], which clearly is not ideal since it resets other vfs as well. I briefly considered modifying the ixgbe driver in linux to expose a command through syfs to completely reset one of its vfs but I decided against it. (incidentally it is also impossible to "unset" the mac address of a vf from linux. It starts out as 00:00:00:00:00:00, but once you set it to a value using iproute2, you cannot return it to unset). Ultimately I settled on making the vf completely reset its registers to default values from the ipxe ixgbevf code, which is what the patch does. I found all of the registers that were getting modified via ethregs[3], and I reset them. The testing showed that in addition to resetting rx and tx queues, I also needed to reset packet split receive type register as well as reset some flags in the SRRCTL register to get things back to the default. It is possible that some future version of linux will set other values that I missed that will also need to be reset, but this does the trick for now. Vish [1] http://lxr.free-electrons.com/source/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c#L791 [2] http://lxr.free-electrons.com/source/drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c#L986 [3] https://sourceforge.net/projects/e1000/files/Ethregs%20-%20Register%20Dump%20Tool/1.20.1/ Vishvananda Ishaya (1): [intelxvf] Reset all vf settings before config src/drivers/net/intel.c | 25 +++++++++++++++++++++++++ src/drivers/net/intel.h | 2 ++ src/drivers/net/intelxvf.c | 17 ++++++++++++++++- src/drivers/net/intelxvf.h | 9 ++++++++- 4 files changed, 51 insertions(+), 2 deletions(-) -- 2.5.0 _______________________________________________ ipxe-devel mailing list ipxe-devel@lists.ipxe.org https://lists.ipxe.org/mailman/listinfo.cgi/ipxe-devel