Hey, I had NFS over RDMA working but hit this bug with o_direct. I cannot remember the exact circumstances of this cropping up.
https://bugzilla.linux-nfs.org/show_bug.cgi?id=228 You can work around this with KVM by explicitly specifying the block size in the XML. I do not know how you would implement this in openstack. With difficulty I imagine :) <qemu:commandline><qemu:arg value='-set'/><qemu:arg value='-set'/><qemu:arg value='-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,logical_block_size=4096'/></qemu:commandline> In general RDMA is best avoided for production setups. NFSoverRDMA support has been dumped from Mellanox OFED and Redhat support is really choppy. I saw weird kernel panics here and there and other general unhappiness. ipoib is generally fast enough as long as you have it in connected mode and set the frame size appropriately try setting it a wee bit bigger than the block size of your filesystem. ta for now Andrew Holway On Jan 10, 2013, at 10:14 PM, Mark Lehrer wrote: > > Has anyone here been able to make Openstack + KVM work with Infiniband & > NFS-RDMA? > > I have spent a couple of days here trying to make it work, but with no luck. > At first I thought the problem was NFS3 and lockd, but I tried NFSv4 and I > have the same problem. I also disabled AppArmor just as a test but that > didn't help. > > Things work fine with Infinband if I don't use RDMA, and I have many > non-libvirt QEMU-KVM vm's working fine using RDMA. > > When this problem occurs, the qemu process is CPU locked, strace shows calls > to FUTEX, select, and lots of EAGAIN messages. If nobody else here is using > this setup, I'll keep digging to find out exactly which option is causing > this problem. > > Thanks, > Mark > > > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : [email protected] > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
