Also, You might be better off asking on the KVM list for this low level stuff.
http://www.linux-kvm.org/page/Lists,_IRC#Mailing_Lists On Jan 14, 2013, at 11:15 AM, Andrew Holway wrote: > Hey, > > I had NFS over RDMA working but hit this bug with o_direct. I cannot remember > the exact circumstances of this cropping up. > > https://bugzilla.linux-nfs.org/show_bug.cgi?id=228 > > You can work around this with KVM by explicitly specifying the block size in > the XML. I do not know how you would implement this in openstack. With > difficulty I imagine :) > > <qemu:commandline><qemu:arg value='-set'/><qemu:arg value='-set'/><qemu:arg > value='-device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,logical_block_size=4096'/></qemu:commandline> > > In general RDMA is best avoided for production setups. NFSoverRDMA support > has been dumped from Mellanox OFED and Redhat support is really choppy. I saw > weird kernel panics here and there and other general unhappiness. > > ipoib is generally fast enough as long as you have it in connected mode and > set the frame size appropriately try setting it a wee bit bigger than the > block size of your filesystem. > > ta for now > > Andrew Holway > > > > On Jan 10, 2013, at 10:14 PM, Mark Lehrer wrote: > >> >> Has anyone here been able to make Openstack + KVM work with Infiniband & >> NFS-RDMA? >> >> I have spent a couple of days here trying to make it work, but with no luck. >> At first I thought the problem was NFS3 and lockd, but I tried NFSv4 and I >> have the same problem. I also disabled AppArmor just as a test but that >> didn't help. >> >> Things work fine with Infinband if I don't use RDMA, and I have many >> non-libvirt QEMU-KVM vm's working fine using RDMA. >> >> When this problem occurs, the qemu process is CPU locked, strace shows calls >> to FUTEX, select, and lots of EAGAIN messages. If nobody else here is using >> this setup, I'll keep digging to find out exactly which option is causing >> this problem. >> >> Thanks, >> Mark >> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~openstack >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp > > > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : [email protected] > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
