As additional data points:
1) Driver developers should *never* have been writing device drivers
that relied on S/G DMA. If they did, then they are in violation of the
DDI, and their drivers are *buggy*. The fact that an IOMMU is involved,
or not, is largely an implementation detail.
2) FORCE_PHYSICAL *is* documented in the man page! (see
ddi_dma_attr(9s)) -- it says nothing about MBus addresses etc. Anyone
relying on that exclusive semantic is probably using an ancient sparcv8
driver, and probably not even running on modern Solaris.
3) The Mbus semantic is *not* mutually exclusive.
4) Most of our drivers work on both SPARC and x86 -- and don't need to
know about IOMMU or not. There are no #ifdef's for most of this code.
There are many many examples I could point to -- in fact, I've yet to
write any code that cared, and I've yet to see any code for x86 Solaris
that would care (unless the code was not using the DDI at all, and
that's an entirely different story.) Much more common is breakage
resulting from SPARC code assuming that they can get a single contiguous
DMA region -- but that code usually gets fixed to deal with either
multiple cookies or DMA windows when it gets ported to x86. (So in my
experience the portability problems are generally the *reverse* of what
I think your concern is.)
5) The "undocumented" property as an escape hatch will still exist -- if
we need to expose it in the future to workaround some buggy driver, then
so be it. But we should *not* be advising developers to use this. We
*should* be advising them to a) ideally not care about IOMMU or not
(i.e. make your code portable!), or b) use FORCE_PHYSICAL if you have
some specific requirement.
-- Garrett
Vikram Hegde wrote:
> Hi,
>
> I think specifiying a FORCE_IOMMU flag will turn this into an
> experimental feature that will never become mainstream in the
> forseeable feature. Driver developers will simply not take interest
> until it becomes mainstream and it will never become mainstream
> without developer interest. We will also be at a competitive
> disadvantage compared to Windows and Linux both of which will have
> IOMMU support. I agree with Garrett and Wes that using FORCE_PHYSICAL
> is probably the right balance of caution (there is an out for driver
> developers) and evolving the DDI - we can't afford a glacial pace.
>
> Vikram
>
> Matthew Jacob wrote:
>> Vikram Hegde wrote:
>>> Hi Matthew,
>>>> No, sorry.
>>> Any reason why FORCE_PHYSICAL is not acceptable ? It has been around
>>> for the SPARC side for some time now and seems to meet the needs of
>>> everyone on that side. It provides exactly the same functionality as
>>> you want.
>>>
>> Because it changes the effective semantics of what it was previously.
>> Previously it was a performance hack to get 36 bit MBus addresses and
>> get around the IOMMU limitations (at least IIRC) and as such it
>> encapsulated a semantic that was specific to SPARC. So much so that
>> the man page even mentions it.
>>
>> S/G DMA on the other hand, has been specific more to non-SPARC and
>> has been the default there. Turning around and now making it /not/
>> the default on the /hope/ that those who don't want that change will
>> get around to recompiling and changing their drivers ignores the
>> intent of what the DDI is about.
>>
>> In short, similarity in names of objects does not mean similarity in
>> effective and assumed semantics.
>>
>> In general I think it's bad policy for Sun to create interfaces that
>> consumers of those interfaces cannot disable if there are workable
>> alternates. It's even more problematic when the default underlying
>> implementation just changes. Furthermore, even if you classified such
>> interfaces as evolving, things that have been long standing have at
>> least some expectation of remaining the same from (minor) release to
>> release.
>>
>> The /whole /point of the DDI is to create a framework that is
>> friendly and usable and reliable for third parties to use. It does
>> /not/ exist to have private side interfaces for Sun internal use
>> only, although nearly all of the DDI has ended up that way (which was
>> /not/ the intent of some of us when we ent through the effort of
>> starting this).
>>
>> You have in some senses two goals here now, though. The first is the
>> DDI idea. The second is to improve the underlying framework in a way
>> that is believable and measurable and helps support broadest
>> framework within Sun itself. As such, the IOMMU addition is probably
>> a good idea- certainly the people proposing it believe so or they
>> wouldn't be doing it.
>>
>> However, if you're not making this a major release type of change (in
>> which case, IMHO, such as it is, lots of things should be thrown out
>> and redone from scratch), then the addition of the IOMMU as the
>> underlying transport instead of S/G DMA should be done in a less
>> invasive way so that previous binaries don't actually do anything
>> different (as a default). If it turns out that they are /provably/
>> correct in environments where the IOMMU is used instead, then the
>> P-teams and the third parties could then either recompile or
>> otherwise turn on that usage. That is prudent.
>>
>> So if you're going to make such a big underlying change, I would
>> suggest instead that until it's a major release, you could add a
>> FORCE_IOMMU flag. This would have the effect of taking driver
>> binaries to work exactly as they have done before and allow drivers
>> written by both Sun and third parties to /try out/ the IOMMU
>> environment as needed or desired.
>>
>> The above isn't probably reasoned out as well as it should be, but it
>> does seem to me that you should either dump the DDI and it's
>> overcautious "let's not change things (for decades)" approach (and
>> take an approach more like (the good parts) of Linux), or if you're
>> going to treat the DDI in its multiple roles, err very much on the
>> side of caution.
>>
>> -matt
>>
>>
>
>
