Re: Access to DMA memory while DMA in progress?

[David Holland]

On Fri, Oct 27, 2017 at 11:15:34AM -0400, Mouse wrote:
 > Heh.  Well, reading the x86 bus_dma implementation (amd64 doesn't seem
 > to have a separate bus_dma implementation of its own) leads me to think
 > it has no such issues; all POSTREAD does there is copy from the bounce
 > buffer (if the transfer is bounced) and issue an lfence memory barrier.
 > (Hmm, I wonder if the memory barrier needs to be before the memcpy)

This was discussed at length on irc over the weekend, and the
conclusion seems to be "yes". With a side order of "why can't
architecture manuals ever be clear about these things?"

-- 
David A. Holland
dholl...@netbsd.org

Re: Access to DMA memory while DMA in progress?

[Eduardo Horvath]

On Fri, 27 Oct 2017, Mouse wrote:

> >> But I'm not sure what sort of sync calls I need to make.  [...]
> > You want to do a bus_dmamap_sync(BUS_DMASYNC_POSTREAD) [...]
> > In the NIC example above, you map the ring buffer with
> > BUS_DMA_COHERENT, fill it up and do a
> > bus_dmamap_sync(BUS_DMASYNC_PREREAD).  When you want to read it
> > (usually after getting an interrupt) you do
> > bus_dmamap_sync(BUS_DMASYNC_POSTREAD) before doing the read.
> 
> Don't you need to PREWRITE after filling it?  Based on the mental
> models I've formed, that feels necessary.

You'd want to do a PREWRITE and a POSTWRITE, but since writing wasn't part 
of your usage model I skipped that part.

Eduardo

Re: Access to DMA memory while DMA in progress?

[Mouse]

>>> [...access to DMA buffer while DMA in progress...ordering...]
>> Hm!  On such hardware, then, you can't count on any particular
>> portion of a DMA transfer being visible until the whole transfer is
>> finished?
> Yes.  I'm assuming here that the driver would do a data cache
> invalidate (for the address range, if possible) at DMA end.

That sounds to me like something bus_dmamap_sync(BUS_DMASYNC_POSTREAD)
would do.  If so, it's not an issue for me; I'm perfectly fine with
some such operation being part of any CPU access to the buffer during
the transfer.

> I don't know much about x86 style platforms.  An example of the sort
> of platform I mentioned would be the MIPS R5000.  I still have some
> scars from building a fast router on top of its incoherent DMA...

Heh.  Well, reading the x86 bus_dma implementation (amd64 doesn't seem
to have a separate bus_dma implementation of its own) leads me to think
it has no such issues; all POSTREAD does there is copy from the bounce
buffer (if the transfer is bounced) and issue an lfence memory barrier.
(Hmm, I wonder if the memory barrier needs to be before the memcpy)

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: Access to DMA memory while DMA in progress?

[Paul.Koning]

> On Oct 27, 2017, at 10:36 AM, Mouse  wrote:
> 
>>> I would like to read the DMA buffer while DMA is still going on.
>>> [...]  I'm fine if the CPU's view lags the hardware's view slightly,
>>> but I do care about the CPU's view of the DMA write order matching
>>> the hardware's: that is, if the CPU sees the value written by a
>>> given DMA cycle, then the CPU must also see the values written by
>>> all previous DMA cycles.
>> I'm not sure if that requirement is necessarily supported by hardware.  For $
> 
> Hm!  On such hardware, then, you can't count on any particular portion
> of a DMA transfer being visible until the whole transfer is finished?

Yes.  I'm assuming here that the driver would do a data cache invalidate
(for the address range, if possible) at DMA end.  Given that, during the
transfer you would see pieces that weren't in the cache before, and would
not see pieces for which there are cache hits.

> For my purposes, unelss amd64 is such a platform, I'm willing to write
> off portability to such machines.  Is there any way to detect them from
> within the driver?  I could just ignore the issue, but I'd prefer to
> give an error at attach time.

I don't know much about x86 style platforms.  An example of the sort of
platform I mentioned would be the MIPS R5000.  I still have some scars
from building a fast router on top of its incoherent DMA...

paul

Re: Access to DMA memory while DMA in progress?

[Mouse]

>> I would like to read the DMA buffer while DMA is still going on.
>> [...]  I'm fine if the CPU's view lags the hardware's view slightly,
>> but I do care about the CPU's view of the DMA write order matching
>> the hardware's: that is, if the CPU sees the value written by a
>> given DMA cycle, then the CPU must also see the values written by
>> all previous DMA cycles.
> I'm not sure if that requirement is necessarily supported by hardware.  For $

Hm!  On such hardware, then, you can't count on any particular portion
of a DMA transfer being visible until the whole transfer is finished?

For my purposes, unelss amd64 is such a platform, I'm willing to write
off portability to such machines.  Is there any way to detect them from
within the driver?  I could just ignore the issue, but I'd prefer to
give an error at attach time.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: Access to DMA memory while DMA in progress?

[Paul.Koning]

> On Oct 27, 2017, at 9:38 AM, Mouse  wrote:
> 
> ...
> I would like to read the DMA buffer while DMA is still going on.  That
> is, I have a buffer of (say) 64K and the hardware is busily writing
> into it; I want to read the buffer and see what the hardware has
> written in the memory it has written and what used to be there in the
> memory it hasn't.  I'm fine if the CPU's view lags the hardware's view
> slightly, but I do care about the CPU's view of the DMA write order
> matching the hardware's: that is, if the CPU sees the value written by
> a given DMA cycle, then the CPU must also see the values written by all
> previous DMA cycles. 

I'm not sure if that requirement is necessarily supported by hardware.  For 
example, in machines that have incoherent DMA, I would think it isn't.

paul