On Fri, 2012-09-28 at 20:15 +0100, David Woodhouse wrote: > On Fri, 2012-09-28 at 12:36 -0400, Linda Knippers wrote: > > I can only speak to the HP servers. We have been shipping devices > > 'for a while' that provide sensor-type data to the platform. The > > device does DMA writes to a range of memory (the RMRR) and > > iLO does DMA reads of that data. > > > > This works in general but not when the 'iommu=pt' boot option is > > used. This patch associates the RMRR with the devices when > > they are moved out of the "si" domain. > > That much makes sense, I think, because they're moved out of the > hardware SI domain *early*, when we realise they're actually only > capable of 32-bit DMA and we have >4GiB of RAM. Right?
Correct. By default all devices are added to SI domain assuming that these devices are 64-bit devices. When we detect the device is a 32-bit device based on the requested dma-mask, it gets removed from SI domain, hence looses its RMRR association. In the meantime dma continues causing DMA errors. This patch is re-processing RMRRs for the device in question and doing re-assignment. > > It sounds like this isn't a case of the device being used by a native > driver or given to KVM, and subsequently released. This is just booting > up and losing the RMRR regions on a device which the OS *hasn't* really > touched. So that should be fixed. > > > Based on Alex's comments about moving RMRR devices between domains, > > it sounds like we also have a problem without the 'iommu=pt' boot > > option if someone assigns one of those devices to a guest. > > Yeah... but why *would* they? What possible reason would we have to > assign either the sensor device, or the iLO, to a KVM guest. Or to have > a native driver that attempts to do DMA from them? > > Obviously, in an ideal world we'd have proper native drivers for the > sensor device. But I'm guessing that's not the case here; it's used by > the firmware and we're not supposed to be touching it? > > And yes, obviously a better hardware design (from the OS/IOMMU point of > view) would be to have a path for the sensor data that *doesn't* go via > host RAM and thus via the IOMMU twice. But while that's a lesson that's > hopefully been learned and will be implemented in future, we have to > deal with the existing hardware and its (ab)use of RMRRs. > Right. We do have hardware that is relying on being able to do dma from devices to a system RAM via RMRR. -- Shuah _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
