Re: [openstack-dev] [nova] SR-IOV and IOMMU check
On Tue, Apr 01, 2014 at 04:59:34AM +, Luohao (brian) wrote: Now, VFIO hasn't been made generally supported by most enterprise linux distributions, and as I know, the current pci passthrough /SR-IOV implementation is still based on a historical approach. Probably we can consider the switch to VFIO framework in later releases. Actually, libvirt will automatically use VFIO if it is available on the host OS by default. This is the case with any recent Fedora and will thus be the case with forthcoming RHEL-7. So nova wouldn't have todo anything to enable use of VFIO - it is supposed to just work when available. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] SR-IOV and IOMMU check
According the recent PCI passthru and SR-IOV design wiki as follows, my understanding is the pci filter resides on a compute node needs to be aware of VFIO to determine which devices can satisfy user requests. Feel free to correct me if it is wrong. https://wiki.openstack.org/wiki/PCI_passthrough_SRIOV_support - Hao -Original Message- From: Daniel P. Berrange [mailto:berra...@redhat.com] Sent: Tuesday, April 01, 2014 1:58 PM To: Luohao (brian) Cc: OpenStack Development Mailing List (not for usage questions); Jinbo (Justin) Subject: Re: [openstack-dev] [nova] SR-IOV and IOMMU check On Tue, Apr 01, 2014 at 04:59:34AM +, Luohao (brian) wrote: Now, VFIO hasn't been made generally supported by most enterprise linux distributions, and as I know, the current pci passthrough /SR-IOV implementation is still based on a historical approach. Probably we can consider the switch to VFIO framework in later releases. Actually, libvirt will automatically use VFIO if it is available on the host OS by default. This is the case with any recent Fedora and will thus be the case with forthcoming RHEL-7. So nova wouldn't have todo anything to enable use of VFIO - it is supposed to just work when available. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] SR-IOV and IOMMU check
On Fri, Mar 28, 2014 at 11:14:49PM -0400, Steve Gordon wrote: - Original Message - This is the approach mentioned by linux-kvm.org http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM 3. reboot and verify that your system has IOMMU support AMD Machine dmesg | grep AMD-Vi ... AMD-Vi: Enabling IOMMU at :00:00.2 cap 0x40 AMD-Vi: Lazy IO/TLB flushing enabled AMD-Vi: Initialized for Passthrough Mode ... Intel Machine dmesg | grep -e DMAR -e IOMMU ... DMAR:DRHD base: 0x00feb03000 flags: 0x0 IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000 ... Right, but the question is whether grepping dmesg is an acceptable/stable API to be relying on from the Nova level. Basically what I'm saying is the reason there isn't a robust way to check this from OpenStack is that there doesn't appear to be a robust way to check this from the kernel? Historically there was no good way to determine this from the kernel. Dmesg logs were the best there is. With new style VFIO, however, we can now reliably determine the level of support and even more importantly what PCI devices must be handled together as a group. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] SR-IOV and IOMMU check
Now, VFIO hasn't been made generally supported by most enterprise linux distributions, and as I know, the current pci passthrough /SR-IOV implementation is still based on a historical approach. Probably we can consider the switch to VFIO framework in later releases. -Original Message- From: Daniel P. Berrange [mailto:berra...@redhat.com] Sent: Monday, March 31, 2014 5:05 PM To: OpenStack Development Mailing List (not for usage questions) Cc: Jinbo (Justin) Subject: Re: [openstack-dev] [nova] SR-IOV and IOMMU check On Fri, Mar 28, 2014 at 11:14:49PM -0400, Steve Gordon wrote: - Original Message - This is the approach mentioned by linux-kvm.org http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM 3. reboot and verify that your system has IOMMU support AMD Machine dmesg | grep AMD-Vi ... AMD-Vi: Enabling IOMMU at :00:00.2 cap 0x40 AMD-Vi: Lazy IO/TLB flushing enabled AMD-Vi: Initialized for Passthrough Mode ... Intel Machine dmesg | grep -e DMAR -e IOMMU ... DMAR:DRHD base: 0x00feb03000 flags: 0x0 IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000 ... Right, but the question is whether grepping dmesg is an acceptable/stable API to be relying on from the Nova level. Basically what I'm saying is the reason there isn't a robust way to check this from OpenStack is that there doesn't appear to be a robust way to check this from the kernel? Historically there was no good way to determine this from the kernel. Dmesg logs were the best there is. With new style VFIO, however, we can now reliably determine the level of support and even more importantly what PCI devices must be handled together as a group. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] SR-IOV and IOMMU check
- Original Message - Hi, all Currently openstack can support SR-IOV device pass-through (at least there are some patches for this), but the prerequisite to this is both IOMMU and SR-IOV must be enabled correctly, it seems there is not a robust way to check this in openstack, I have implemented a way to do this and hope it can be committed into upstream, this can help find the issue beforehand, instead of letting kvm report the issue no IOMMU found until the VM is started. I didn't find an appropriate place to put into this, do you think this is necessary? Where can it be put into? Welcome your advice and thank you in advance. What's the mechanism you are using on the host side to determine that IOMMU is supported/enabled? Thanks, Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] SR-IOV and IOMMU check
This is the approach mentioned by linux-kvm.org http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM 3. reboot and verify that your system has IOMMU support AMD Machine dmesg | grep AMD-Vi ... AMD-Vi: Enabling IOMMU at :00:00.2 cap 0x40 AMD-Vi: Lazy IO/TLB flushing enabled AMD-Vi: Initialized for Passthrough Mode ... Intel Machine dmesg | grep -e DMAR -e IOMMU ... DMAR:DRHD base: 0x00feb03000 flags: 0x0 IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000 ... -Original Message- From: Steve Gordon [mailto:sgor...@redhat.com] Sent: Saturday, March 29, 2014 3:37 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] SR-IOV and IOMMU check - Original Message - Hi, all Currently openstack can support SR-IOV device pass-through (at least there are some patches for this), but the prerequisite to this is both IOMMU and SR-IOV must be enabled correctly, it seems there is not a robust way to check this in openstack, I have implemented a way to do this and hope it can be committed into upstream, this can help find the issue beforehand, instead of letting kvm report the issue no IOMMU found until the VM is started. I didn't find an appropriate place to put into this, do you think this is necessary? Where can it be put into? Welcome your advice and thank you in advance. What's the mechanism you are using on the host side to determine that IOMMU is supported/enabled? Thanks, Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] SR-IOV and IOMMU check
- Original Message - This is the approach mentioned by linux-kvm.org http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM 3. reboot and verify that your system has IOMMU support AMD Machine dmesg | grep AMD-Vi ... AMD-Vi: Enabling IOMMU at :00:00.2 cap 0x40 AMD-Vi: Lazy IO/TLB flushing enabled AMD-Vi: Initialized for Passthrough Mode ... Intel Machine dmesg | grep -e DMAR -e IOMMU ... DMAR:DRHD base: 0x00feb03000 flags: 0x0 IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000 ... Right, but the question is whether grepping dmesg is an acceptable/stable API to be relying on from the Nova level. Basically what I'm saying is the reason there isn't a robust way to check this from OpenStack is that there doesn't appear to be a robust way to check this from the kernel? Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
