On Fri, Feb 23, 2018 at 1:10 AM Peter Xu <pet...@redhat.com> wrote: > On Fri, Feb 23, 2018 at 12:32:13AM -0500, Jintack Lim wrote: > > Hi Peter, > > > > Hope you had great holidays! > > > > On Thu, Feb 22, 2018 at 10:55 PM, Peter Xu <pet...@redhat.com> wrote: > > > On Tue, Feb 20, 2018 at 11:03:46PM -0500, Jintack Lim wrote: > > >> Hi, > > >> > > >> I'm using vhost with the virtual intel-iommu, and this page[1] shows > > >> the QEMU command line example. > > >> > > >> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \ > > >> -device intel-iommu,intremap=on,device-iotlb=on \ > > >> -device ioh3420,id=pcie.1,chassis=1 \ > > >> -device > > >> > virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on > > >> \ > > >> -netdev tap,id=net0,vhostforce \ > > >> $IMAGE_PATH > > >> > > >> I wonder what's the impact of using device-iotlb and ats options as > > >> they are described necessary. > > >> > > >> In my understanding, vhost in the kernel only looks at > > >> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a > > >> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg > > >> basically to cache mappings correctly in the vhost, so I wonder what's > > >> the role of ats in this case. > > > > > > The "ats" as virtio device parameter will add ATS capability to the > > > PCI device. > > > > > > The "device-iotlb" as intel-iommu parameter will enable ATS in the > > > IOMMU device (and also report that in ACPI field). > > > > > > If both parameters are provided IIUC it means guest will know virtio > > > device has device-iotlb and it'll treat the device specially (e.g., > > > guest will need to send device-iotlb invalidations). > > > > Oh, I see. I was focusing on how QEMU and vhost work in the host, but > > I think I missed the guest part! Thanks. I see that the Intel IOMMU > > driver has has_iotlb_device flag for that purpose. > > > > > > > > We'd better keep these parameters when running virtio devices with > > > vIOMMU. For the rest of vhost/arm specific questions, I'll leave to > > > others. > > > > It seems like SMMU is not checking ATS capability - at least > > ats_enabled flag - but I may miss something here as well :) > > > > > > > > PS: Though IIUC the whole ATS thing may not really be necessary for > > > current VT-d emulation, since even with ATS vhost is registering UNMAP > > > IOMMU notifiers (see vhost_iommu_region_add()), and IIUC that means > > > vhost will receive IOTLB invalidations even without ATS support, and > > > it _might_ still work. > > > > Right. That's what I thought. > > > > Come to think of it, I'm not sure why we need to flush mappings in > > IOMMU and devices separately in the first place... Any thoughts? > > I don't know ATS much, neither. > > You can have a look at chap 4 of vt-d spec: > > One approach to scaling IOTLBs is to enable I/O devices to > participate in the DMA remapping with IOTLBs implemented at > the devices. The Device-IOTLBs alleviate pressure for IOTLB > resources in the core logic, and provide opportunities for > devices to improve performance by pre-fetching address > translations before issuing DMA requests. This may be useful > for devices with strict DMA latency requirements (such as > isochronous devices), and for devices that have large DMA > working set or multiple active DMA streams. > > So I think it's for performance's sake. For example, the DMA operation > won't need to be translated at all if it's pre-translated, so it can > have less latency. And also, that'll offload some of the translation > process so that workload can be more distributed. > > When with that (caches located both on IOMMU's and device's side), we > need to invalidate all the cache when needed. >
Right. I think my question was not clear. My question was that why don’t IOMMU invalidate device-iotlb along with its mappings in one go. Then IOMMU device driver doesn’t need to flush device-iotlb explicitly. Maybe the reason is that ATS and IOMMU are not always coupled.. but I guess it’s time for me to get some more background :) > > > > Your reply was really helpful to me. I appreciate it. > > My pleasure. Thanks, > > > > > Thanks, > > Jintack > > > > > But there can be other differences, like > > > performance, etc. > > > > > >> > > >> A related question is that if we use SMMU emulation[2] on ARM without > > >> those options, does vhost cache mappings as if it has a device-iotlb? > > >> (I guess this is the case.) > > >> > > >> I'm pretty new to QEMU code, so I might be missing something. Can > > >> somebody shed some light on it? > > >> > > >> [1] https://wiki.qemu.org/Features/VT-d > > >> [2] > http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html > > >> > > >> Thanks, > > >> Jintack > > >> > > > > > > -- > > > Peter Xu > > > > > > > -- > Peter Xu > >