Re: usb_pc_cpu_flush
Hi Ian, Any news on this issue? I've put the following patch into -current for your convenience: https://svnweb.freebsd.org/changeset/base/278074 Kohji: Maybe you also want to test and review this patch? --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
Hi Ian,
Please find attached a patch which disable the double mapping of
allocated DMA capable buffers in the USB stack for all drivers.
Can you test it and check out the total memory usage before and after
the patch? Also verify it if makes any difference on any platforms.
Thank you!
--HPS
Index: sys/dev/usb/usb_transfer.c
===
--- sys/dev/usb/usb_transfer.c (revision 277240)
+++ sys/dev/usb/usb_transfer.c (working copy)
@@ -237,7 +237,7 @@
n_obj = 1;
} else {
/* compute number of objects per page */
- n_obj = (USB_PAGE_SIZE / size);
+ n_obj = 1; /* (USB_PAGE_SIZE / size); */
/*
* Compute number of DMA chunks, rounded up
* to nearest one:
@@ -273,8 +273,11 @@
&parm->curr_xfer->xroot->dma_parent_tag;
}
- if (ppc) {
- *ppc = parm->xfer_page_cache_ptr;
+ if (ppc != NULL) {
+ if (n_obj != 1)
+ *ppc = parm->xfer_page_cache_ptr;
+ else
+ *ppc = parm->dma_page_cache_ptr;
}
r = count; /* set remainder count */
z = n_obj * size; /* set allocation size */
@@ -281,7 +284,18 @@
pc = parm->xfer_page_cache_ptr;
pg = parm->dma_page_ptr;
- for (x = 0; x != n_dma_pc; x++) {
+ if (n_obj == 1) {
+ for (x = 0; x != n_dma_pc; x++) {
+ if (usb_pc_alloc_mem(parm->dma_page_cache_ptr,
+ pg, z, align)) {
+ return (1); /* failure */
+ }
+ /* Make room for one DMA page cache and "n_dma_pg" pages */
+ parm->dma_page_cache_ptr++;
+ pg += n_dma_pg;
+ }
+ } else {
+ for (x = 0; x != n_dma_pc; x++) {
if (r < n_obj) {
/* compute last remainder */
@@ -294,7 +308,7 @@
}
/* Set beginning of current buffer */
buf = parm->dma_page_cache_ptr->buffer;
- /* Make room for one DMA page cache and one page */
+ /* Make room for one DMA page cache and "n_dma_pg" pages */
parm->dma_page_cache_ptr++;
pg += n_dma_pg;
@@ -314,6 +328,7 @@
}
mtx_unlock(pc->tag_parent->mtx);
}
+ }
}
parm->xfer_page_cache_ptr = pc;
___
freebsd-usb@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-usb
To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 01/14/15 16:03, Ian Lepore wrote: If you are allocating the memory using bus_dmamem_alloc() then no, absolutely not. It's working for you by accident because of the USB_HOST_ALIGN hacks which I think date back to freebsd 8 or so. When I talked to you a couple years ago about doing it the right way, you were outraged (perhaps rightly so) about the inefficiency involved, because of things like allocating a 16 byte buffer causing the allocation of a full page just so that the caching attributes could be changed. So I fixed that with the busdma zone allocator stuff that makes it very efficient to allocate many tiny dma buffers (coherent or not), and when you use buffers allocated that way, the sync operations code can make some safe assumptions and avoid doing bounces due to cacheline alignments. If buffers were allocated right-sized now that it's efficient to do so, the USB_HOST_ALIGN hack could go away. Hmm, actually it might be necessary to start using the busdma zone allocator in mips as well before undoing the hack. Hi Ian, This sounds good. I'll make a quick patch you can try this week which splits the USB busdma allocations again. It is a quite trivial thing to do. Can you or Kott then test this patch? --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On Wed, 2015-01-14 at 06:13 +0100, Hans Petter Selasky wrote:
> On 01/13/15 21:25, Ian Lepore wrote:
> > On Tue, 2015-01-13 at 17:52 +0100, Hans Petter Selasky wrote:
> >> On 01/13/15 17:29, Ian Lepore wrote:
> >>> On Tue, 2015-01-13 at 16:57 +0100, Hans Petter Selasky wrote:
[...]
> >>
> >> Hi,
> >>
> >> Can you give an example of a NIC driver which you consider a good
> >> example which use DMA both for data (not only mbufs) and the control
> >> path and use busdma as you consider to be correct?
> >>
> >> --HPS
> >
> > dev/ffec/if_ffec.c. I'm not happy with the fact that it takes two calls
> > (a PRE and a POST) to accomplish a single action, but that's the right
> > way to do things in the current busdma world, PRE and POST operations
> > need to be paired.
> >
> > I think we need new busdma support for shared concurrent access
> > descriptors, because it's a type of dma that just isn't supported well
> > by the existing API. There should be a new flag for bus_dmamem_alloc()
> > that indicates the memory is going to be used for such shared access
> > (because some platforms may be able to provide memory that's mapped
> > optimally for such situations), and there should be new sync ops that
> > don't require a pair of calls to accomplish a single action.
> >
> > All of this is in the context of shared descriptor memory. Regular IO
> > buffers should just use the proper sequence of PRE and POST syncs (and
> > most especially should *never* do POSTREAD before PREREAD like the
> > current usb_pc_cpu_invalidate() does, because with bounce buffers that
> > will just not work right).
> >
>
> Hi,
>
> I understand and that can be done for IO-buffers like in the FFEC
> driver. For other buffers it is a bigger task, however:
>
> I see that some memory is allocated using "BUS_DMA_COHERENT" in
> if_ffec.c and in those cases no busdma sync operations are performed! Is
> that correct? For example "rxdesc_ring" is allocated coherently and no
> cache sync ops are done before and after access.
>
No, it does do sync operations on the descriptors rings, because one of
the documented rules of coherent memory is that you still have to do all
the normal sync ops. Traditionally coherent has meant "completely
disable caching" on arm and you could get away with not doing the sync
ops, but that's not true anymore. Arm memory subsystems are more
complex these days and sync ops are needed in some cases even for
uncached memory.
In the current ffec code, sync ops on descriptor memory are done on line
655/657 (these could really be back to back without the TDAR write
between them). At this point new descriptors have been written to the
tx ring and the PREWRITE makes the changes visible to the hardware, the
write into the TDAR register informs the hardware that descriptors were
changed and it should rescan the list. The POSTWRITE is a no-op but
included for technical correctness.
Another desc-ring related set of sync ops is on lines 683-684. In this
case the hardware has updated some of the descriptors and the
PREREAD/POSTREAD sync ops ensure that the cpu will see the changes. The
same is true on lines 850-851. This is the place where having two calls
is really a performance hit, because both PRE and POSTREAD sync will do
an invalidate, and it really only needs to be done once, but the busdma
conventions are that PRE and POST ops need to come in properly-sequenced
pairs.
And finally on 899-900 is the rx code similar to the tx code on 655/657,
where the driver has updated descriptors in the ring and done a PREWRITE
to make the changes visible to the hardware.
> The buffer that Mr. Kott hit a crash on should also be allocated
> coherently. Looking at the ARM busdma code in -current I see a possible bug:
>
> > int
> > _bus_dmamap_load_buffer(bus_dma_tag_t dmat, bus_dmamap_t map, void *buf,
> > bus_size_t buflen, struct pmap *pmap, int flags, bus_dma_segment_t
> > *segs,
> > int *segp)
> > {
> > bus_size_t sgsize;
> > bus_addr_t curaddr;
> > struct sync_list *sl;
> > vm_offset_t vaddr = (vm_offset_t)buf;
> > int error = 0;
> >
> > if (segs == NULL)
> > segs = dmat->segments;
> > if ((flags & BUS_DMA_LOAD_MBUF) != 0)
> > map->flags |= DMAMAP_CACHE_ALIGNED;
> >
> > if ((dmat->flags & BUS_DMA_COULD_BOUNCE) != 0) {
> > _bus_dmamap_count_pages(dmat, map, pmap, buf, buflen,
> > flags);
> > if (map->pagesneeded != 0) {
> > error = _bus_dmamap_reserve_pages(dmat, map, flags);
> > if (error)
> > return (error);
> > }
> > }
> > CTR3(KTR_BUSDMA, "lowaddr= %d boundary= %d, "
> > "alignment= %d", dmat->lowaddr, dmat->boundary,
> > dmat->alignment);
> >
> > while (buflen > 0) {
> > /*
> > * Get the physical address for this segme
Re: usb_pc_cpu_flush
On 01/13/15 21:25, Ian Lepore wrote:
On Tue, 2015-01-13 at 17:52 +0100, Hans Petter Selasky wrote:
On 01/13/15 17:29, Ian Lepore wrote:
On Tue, 2015-01-13 at 16:57 +0100, Hans Petter Selasky wrote:
On 01/13/15 16:40, Ian Lepore wrote:
On Tue, 2015-01-13 at 16:27 +0100, Hans Petter Selasky wrote:
On 01/13/15 15:49, Ian Lepore wrote:
On Tue, 2015-01-13 at 00:14 -0700, kott wrote:
Yes with cache disabled, this problem is not seen. Seems to be with a issue
with l2 cache.
Thanks kott
Except that there are no known problems with l2 cache on armv7 right
now. There are known problems with the USB driver using the busdma
routines incorrectly, which accidentally works okay on x86 platforms but
likely not so well on others.
Hi,
If there is a problem it is in "usb_pc_cpu_flush()" or
"usb_pc_cpu_invalidate()":
void
usb_pc_cpu_flush(struct usb_page_cache *pc)
{
if (pc->page_offset_end == pc->page_offset_buf) {
/* nothing has been loaded into this page cache! */
return;
}
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE);
}
USB has a very simple DMA sync language, either flush or invalidate.
These are used correctly from what I can see with regard to the FreeBSD
USB specification.
(unless a simple restart somehow fixes the problem)
If the "usb_pc_cpu_flush()" function does not cause the CPU cache to be
written to RAM before the function returns, please let me know.
--HPS
You have an incomplete concept of how busdma sync operations work. It
isn't a simple "cpu cache written to ram" operation, there are bounce
buffers and other complexities involved that require that the sync
operations be done at the correct time in the correct order, and the
current usb driver doesn't do that. Instead it does things like
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTREAD);
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREREAD);
And that's just nonsense that will lead to problems like delivering
random buffer garbage to/from a device.
To the degree that USB works at all on non-x86 platforms it works by
accident. Usually. Except when it doesn't.
-- Ian
Hi,
Bounce buffers are perfectly fine with USB as long as the busdma does
what it is told. If there is no easy way to do a simple "cache flush" or
"cache invalide" or bounce buffer "flush" or bounce buffer "invalidate"
multiple times in a row, then busdma cannot co-exist with USB. It is not
because I'm stubborn, but because of the way USB DMA controllers are
designed.
With USB chipsets we sometimes need to read the RAM area for a single
buffer multiple times to poll for updates. From what I've been told in
the past BUSDMA does.
--HPS
--HPS
--HPS
And so we reach the same old impasse, where you declare that USB is
special and doesn't have to behave like other drivers, even though it is
in no way unique in terms of having things like concurrent shared access
to descriptor memory, something that virtually every modern NIC has.
Hi,
Can you give an example of a NIC driver which you consider a good
example which use DMA both for data (not only mbufs) and the control
path and use busdma as you consider to be correct?
--HPS
dev/ffec/if_ffec.c. I'm not happy with the fact that it takes two calls
(a PRE and a POST) to accomplish a single action, but that's the right
way to do things in the current busdma world, PRE and POST operations
need to be paired.
I think we need new busdma support for shared concurrent access
descriptors, because it's a type of dma that just isn't supported well
by the existing API. There should be a new flag for bus_dmamem_alloc()
that indicates the memory is going to be used for such shared access
(because some platforms may be able to provide memory that's mapped
optimally for such situations), and there should be new sync ops that
don't require a pair of calls to accomplish a single action.
All of this is in the context of shared descriptor memory. Regular IO
buffers should just use the proper sequence of PRE and POST syncs (and
most especially should *never* do POSTREAD before PREREAD like the
current usb_pc_cpu_invalidate() does, because with bounce buffers that
will just not work right).
Hi,
I understand and that can be done for IO-buffers like in the FFEC
driver. For other buffers it is a bigger task, however:
I see that some memory is allocated using "BUS_DMA_COHERENT" in
if_ffec.c and in those cases no busdma sync operations are performed! Is
that correct? For example "rxdesc_ring" is allocated coherently and no
cache sync ops are done before and after access.
The buffer that Mr. Kott hit a crash on should also be allocated
coherently. Looking at the ARM busdma code in -current I see a possible bug:
int
_bus
Re: usb_pc_cpu_flush
On Tue, 2015-01-13 at 17:52 +0100, Hans Petter Selasky wrote:
> On 01/13/15 17:29, Ian Lepore wrote:
> > On Tue, 2015-01-13 at 16:57 +0100, Hans Petter Selasky wrote:
> >> On 01/13/15 16:40, Ian Lepore wrote:
> >>> On Tue, 2015-01-13 at 16:27 +0100, Hans Petter Selasky wrote:
> >>>> On 01/13/15 15:49, Ian Lepore wrote:
> >>>>> On Tue, 2015-01-13 at 00:14 -0700, kott wrote:
> >>>>>> Yes with cache disabled, this problem is not seen. Seems to be with a
> >>>>>> issue
> >>>>>> with l2 cache.
> >>>>>> Thanks kott
> >>>>>
> >>>>> Except that there are no known problems with l2 cache on armv7 right
> >>>>> now. There are known problems with the USB driver using the busdma
> >>>>> routines incorrectly, which accidentally works okay on x86 platforms but
> >>>>> likely not so well on others.
> >>>>>
> >>>>
> >>>> Hi,
> >>>>
> >>>> If there is a problem it is in "usb_pc_cpu_flush()" or
> >>>> "usb_pc_cpu_invalidate()":
> >>>>
> >>>> void
> >>>> usb_pc_cpu_flush(struct usb_page_cache *pc)
> >>>> {
> >>>>if (pc->page_offset_end == pc->page_offset_buf) {
> >>>>/* nothing has been loaded into this page cache! */
> >>>>return;
> >>>>}
> >>>>bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE);
> >>>> }
> >>>>
> >>>> USB has a very simple DMA sync language, either flush or invalidate.
> >>>> These are used correctly from what I can see with regard to the FreeBSD
> >>>> USB specification.
> >>>> (unless a simple restart somehow fixes the problem)
> >>>> If the "usb_pc_cpu_flush()" function does not cause the CPU cache to be
> >>>> written to RAM before the function returns, please let me know.
> >>>>
> >>>> --HPS
> >>>
> >>> You have an incomplete concept of how busdma sync operations work. It
> >>> isn't a simple "cpu cache written to ram" operation, there are bounce
> >>> buffers and other complexities involved that require that the sync
> >>> operations be done at the correct time in the correct order, and the
> >>> current usb driver doesn't do that. Instead it does things like
> >>>
> >>> bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTREAD);
> >>> bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREREAD);
> >>>
> >>> And that's just nonsense that will lead to problems like delivering
> >>> random buffer garbage to/from a device.
> >>>
> >>> To the degree that USB works at all on non-x86 platforms it works by
> >>> accident. Usually. Except when it doesn't.
> >>>
> >>> -- Ian
> >>>
> >>
> >> Hi,
> >>
> >> Bounce buffers are perfectly fine with USB as long as the busdma does
> >> what it is told. If there is no easy way to do a simple "cache flush" or
> >> "cache invalide" or bounce buffer "flush" or bounce buffer "invalidate"
> >> multiple times in a row, then busdma cannot co-exist with USB. It is not
> >> because I'm stubborn, but because of the way USB DMA controllers are
> >> designed.
> >>
> >> With USB chipsets we sometimes need to read the RAM area for a single
> >> buffer multiple times to poll for updates. From what I've been told in
> >> the past BUSDMA does.
> >>
> >> --HPS
> >>
> >> --HPS
> >>
> >> --HPS
> >>
> >
> > And so we reach the same old impasse, where you declare that USB is
> > special and doesn't have to behave like other drivers, even though it is
> > in no way unique in terms of having things like concurrent shared access
> > to descriptor memory, something that virtually every modern NIC has.
> >
>
> Hi,
>
> Can you give an example of a NIC driver which you consider a good
> example which use DMA both for data (not only mbufs) and the control
> path and use busdma as you consider to be correct?
>
> --HPS
dev/ffec/if_ffec.c. I'm not happy with the fact that it takes two calls
(a PRE and a POST) to accomplish a single action, but that's the right
way to do things in the current busdma world, PRE and POST operations
need to be paired.
I think we need new busdma support for shared concurrent access
descriptors, because it's a type of dma that just isn't supported well
by the existing API. There should be a new flag for bus_dmamem_alloc()
that indicates the memory is going to be used for such shared access
(because some platforms may be able to provide memory that's mapped
optimally for such situations), and there should be new sync ops that
don't require a pair of calls to accomplish a single action.
All of this is in the context of shared descriptor memory. Regular IO
buffers should just use the proper sequence of PRE and POST syncs (and
most especially should *never* do POSTREAD before PREREAD like the
current usb_pc_cpu_invalidate() does, because with bounce buffers that
will just not work right).
-- Ian
___
freebsd-usb@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-usb
To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
Hi Kott, Can you tell us which ARM kernel configuration file you are using for FreeBSD? Reading through this thread I cannot see this piece of information yet. Have you tested other ARM platforms? Thank you, --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 01/13/15 17:52, Hans Petter Selasky wrote: Hi, Can you give an example of a NIC driver which you consider a good example which use DMA both for data (not only mbufs) and the control path and use busdma as you consider to be correct? --HPS So that I can compare this agains what USB is doing. --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 01/13/15 17:29, Ian Lepore wrote:
On Tue, 2015-01-13 at 16:57 +0100, Hans Petter Selasky wrote:
On 01/13/15 16:40, Ian Lepore wrote:
On Tue, 2015-01-13 at 16:27 +0100, Hans Petter Selasky wrote:
On 01/13/15 15:49, Ian Lepore wrote:
On Tue, 2015-01-13 at 00:14 -0700, kott wrote:
Yes with cache disabled, this problem is not seen. Seems to be with a issue
with l2 cache.
Thanks kott
Except that there are no known problems with l2 cache on armv7 right
now. There are known problems with the USB driver using the busdma
routines incorrectly, which accidentally works okay on x86 platforms but
likely not so well on others.
Hi,
If there is a problem it is in "usb_pc_cpu_flush()" or
"usb_pc_cpu_invalidate()":
void
usb_pc_cpu_flush(struct usb_page_cache *pc)
{
if (pc->page_offset_end == pc->page_offset_buf) {
/* nothing has been loaded into this page cache! */
return;
}
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE);
}
USB has a very simple DMA sync language, either flush or invalidate.
These are used correctly from what I can see with regard to the FreeBSD
USB specification.
If the "usb_pc_cpu_flush()" function does not cause the CPU cache to be
written to RAM before the function returns, please let me know.
--HPS
You have an incomplete concept of how busdma sync operations work. It
isn't a simple "cpu cache written to ram" operation, there are bounce
buffers and other complexities involved that require that the sync
operations be done at the correct time in the correct order, and the
current usb driver doesn't do that. Instead it does things like
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTREAD);
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREREAD);
And that's just nonsense that will lead to problems like delivering
random buffer garbage to/from a device.
To the degree that USB works at all on non-x86 platforms it works by
accident. Usually. Except when it doesn't.
-- Ian
Hi,
Bounce buffers are perfectly fine with USB as long as the busdma does
what it is told. If there is no easy way to do a simple "cache flush" or
"cache invalide" or bounce buffer "flush" or bounce buffer "invalidate"
multiple times in a row, then busdma cannot co-exist with USB. It is not
because I'm stubborn, but because of the way USB DMA controllers are
designed.
With USB chipsets we sometimes need to read the RAM area for a single
buffer multiple times to poll for updates. From what I've been told in
the past BUSDMA does.
--HPS
--HPS
--HPS
And so we reach the same old impasse, where you declare that USB is
special and doesn't have to behave like other drivers, even though it is
in no way unique in terms of having things like concurrent shared access
to descriptor memory, something that virtually every modern NIC has.
Hi,
Can you give an example of a NIC driver which you consider a good
example which use DMA both for data (not only mbufs) and the control
path and use busdma as you consider to be correct?
--HPS
___
freebsd-usb@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-usb
To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On Tue, 2015-01-13 at 16:57 +0100, Hans Petter Selasky wrote:
> On 01/13/15 16:40, Ian Lepore wrote:
> > On Tue, 2015-01-13 at 16:27 +0100, Hans Petter Selasky wrote:
> >> On 01/13/15 15:49, Ian Lepore wrote:
> >>> On Tue, 2015-01-13 at 00:14 -0700, kott wrote:
> >>>> Yes with cache disabled, this problem is not seen. Seems to be with a
> >>>> issue
> >>>> with l2 cache.
> >>>> Thanks kott
> >>>
> >>> Except that there are no known problems with l2 cache on armv7 right
> >>> now. There are known problems with the USB driver using the busdma
> >>> routines incorrectly, which accidentally works okay on x86 platforms but
> >>> likely not so well on others.
> >>>
> >>
> >> Hi,
> >>
> >> If there is a problem it is in "usb_pc_cpu_flush()" or
> >> "usb_pc_cpu_invalidate()":
> >>
> >> void
> >> usb_pc_cpu_flush(struct usb_page_cache *pc)
> >> {
> >> if (pc->page_offset_end == pc->page_offset_buf) {
> >> /* nothing has been loaded into this page cache! */
> >> return;
> >> }
> >> bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE);
> >> }
> >>
> >> USB has a very simple DMA sync language, either flush or invalidate.
> >> These are used correctly from what I can see with regard to the FreeBSD
> >> USB specification.
> >>
> >> If the "usb_pc_cpu_flush()" function does not cause the CPU cache to be
> >> written to RAM before the function returns, please let me know.
> >>
> >> --HPS
> >
> > You have an incomplete concept of how busdma sync operations work. It
> > isn't a simple "cpu cache written to ram" operation, there are bounce
> > buffers and other complexities involved that require that the sync
> > operations be done at the correct time in the correct order, and the
> > current usb driver doesn't do that. Instead it does things like
> >
> > bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTREAD);
> > bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREREAD);
> >
> > And that's just nonsense that will lead to problems like delivering
> > random buffer garbage to/from a device.
> >
> > To the degree that USB works at all on non-x86 platforms it works by
> > accident. Usually. Except when it doesn't.
> >
> > -- Ian
> >
>
> Hi,
>
> Bounce buffers are perfectly fine with USB as long as the busdma does
> what it is told. If there is no easy way to do a simple "cache flush" or
> "cache invalide" or bounce buffer "flush" or bounce buffer "invalidate"
> multiple times in a row, then busdma cannot co-exist with USB. It is not
> because I'm stubborn, but because of the way USB DMA controllers are
> designed.
>
> With USB chipsets we sometimes need to read the RAM area for a single
> buffer multiple times to poll for updates. From what I've been told in
> the past BUSDMA does.
>
> --HPS
>
> --HPS
>
> --HPS
>
And so we reach the same old impasse, where you declare that USB is
special and doesn't have to behave like other drivers, even though it is
in no way unique in terms of having things like concurrent shared access
to descriptor memory, something that virtually every modern NIC has.
-- Ian
___
freebsd-usb@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-usb
To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 01/13/15 16:57, Hans Petter Selasky wrote: BUSDMA does. --HPS ... allow this. --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 01/13/15 16:40, Ian Lepore wrote:
On Tue, 2015-01-13 at 16:27 +0100, Hans Petter Selasky wrote:
On 01/13/15 15:49, Ian Lepore wrote:
On Tue, 2015-01-13 at 00:14 -0700, kott wrote:
Yes with cache disabled, this problem is not seen. Seems to be with a issue
with l2 cache.
Thanks kott
Except that there are no known problems with l2 cache on armv7 right
now. There are known problems with the USB driver using the busdma
routines incorrectly, which accidentally works okay on x86 platforms but
likely not so well on others.
Hi,
If there is a problem it is in "usb_pc_cpu_flush()" or
"usb_pc_cpu_invalidate()":
void
usb_pc_cpu_flush(struct usb_page_cache *pc)
{
if (pc->page_offset_end == pc->page_offset_buf) {
/* nothing has been loaded into this page cache! */
return;
}
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE);
}
USB has a very simple DMA sync language, either flush or invalidate.
These are used correctly from what I can see with regard to the FreeBSD
USB specification.
If the "usb_pc_cpu_flush()" function does not cause the CPU cache to be
written to RAM before the function returns, please let me know.
--HPS
You have an incomplete concept of how busdma sync operations work. It
isn't a simple "cpu cache written to ram" operation, there are bounce
buffers and other complexities involved that require that the sync
operations be done at the correct time in the correct order, and the
current usb driver doesn't do that. Instead it does things like
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTREAD);
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREREAD);
And that's just nonsense that will lead to problems like delivering
random buffer garbage to/from a device.
To the degree that USB works at all on non-x86 platforms it works by
accident. Usually. Except when it doesn't.
-- Ian
Hi,
Bounce buffers are perfectly fine with USB as long as the busdma does
what it is told. If there is no easy way to do a simple "cache flush" or
"cache invalide" or bounce buffer "flush" or bounce buffer "invalidate"
multiple times in a row, then busdma cannot co-exist with USB. It is not
because I'm stubborn, but because of the way USB DMA controllers are
designed.
With USB chipsets we sometimes need to read the RAM area for a single
buffer multiple times to poll for updates. From what I've been told in
the past BUSDMA does.
--HPS
--HPS
--HPS
___
freebsd-usb@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-usb
To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On Tue, 2015-01-13 at 16:27 +0100, Hans Petter Selasky wrote:
> On 01/13/15 15:49, Ian Lepore wrote:
> > On Tue, 2015-01-13 at 00:14 -0700, kott wrote:
> >> Yes with cache disabled, this problem is not seen. Seems to be with a issue
> >> with l2 cache.
> >> Thanks kott
> >
> > Except that there are no known problems with l2 cache on armv7 right
> > now. There are known problems with the USB driver using the busdma
> > routines incorrectly, which accidentally works okay on x86 platforms but
> > likely not so well on others.
> >
>
> Hi,
>
> If there is a problem it is in "usb_pc_cpu_flush()" or
> "usb_pc_cpu_invalidate()":
>
> void
> usb_pc_cpu_flush(struct usb_page_cache *pc)
> {
> if (pc->page_offset_end == pc->page_offset_buf) {
> /* nothing has been loaded into this page cache! */
> return;
> }
> bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE);
> }
>
> USB has a very simple DMA sync language, either flush or invalidate.
> These are used correctly from what I can see with regard to the FreeBSD
> USB specification.
>
> If the "usb_pc_cpu_flush()" function does not cause the CPU cache to be
> written to RAM before the function returns, please let me know.
>
> --HPS
You have an incomplete concept of how busdma sync operations work. It
isn't a simple "cpu cache written to ram" operation, there are bounce
buffers and other complexities involved that require that the sync
operations be done at the correct time in the correct order, and the
current usb driver doesn't do that. Instead it does things like
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_POSTREAD);
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREREAD);
And that's just nonsense that will lead to problems like delivering
random buffer garbage to/from a device.
To the degree that USB works at all on non-x86 platforms it works by
accident. Usually. Except when it doesn't.
-- Ian
___
freebsd-usb@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-usb
To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 01/13/15 15:49, Ian Lepore wrote:
On Tue, 2015-01-13 at 00:14 -0700, kott wrote:
Yes with cache disabled, this problem is not seen. Seems to be with a issue
with l2 cache.
Thanks kott
Except that there are no known problems with l2 cache on armv7 right
now. There are known problems with the USB driver using the busdma
routines incorrectly, which accidentally works okay on x86 platforms but
likely not so well on others.
Hi,
If there is a problem it is in "usb_pc_cpu_flush()" or
"usb_pc_cpu_invalidate()":
void
usb_pc_cpu_flush(struct usb_page_cache *pc)
{
if (pc->page_offset_end == pc->page_offset_buf) {
/* nothing has been loaded into this page cache! */
return;
}
bus_dmamap_sync(pc->tag, pc->map, BUS_DMASYNC_PREWRITE);
}
USB has a very simple DMA sync language, either flush or invalidate.
These are used correctly from what I can see with regard to the FreeBSD
USB specification.
If the "usb_pc_cpu_flush()" function does not cause the CPU cache to be
written to RAM before the function returns, please let me know.
--HPS
___
freebsd-usb@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-usb
To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On Tue, 2015-01-13 at 00:14 -0700, kott wrote: > Yes with cache disabled, this problem is not seen. Seems to be with a issue > with l2 cache. > Thanks kott Except that there are no known problems with l2 cache on armv7 right now. There are known problems with the USB driver using the busdma routines incorrectly, which accidentally works okay on x86 platforms but likely not so well on others. -- Ian ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
Yes with cache disabled, this problem is not seen. Seems to be with a issue with l2 cache. Thanks kott -- View this message in context: http://freebsd.1045724.n5.nabble.com/usb-pc-cpu-flush-tp5975583p5980199.html Sent from the freebsd-usb mailing list archive at Nabble.com. ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
Will check with cache disabled. this may be the problem -- View this message in context: http://freebsd.1045724.n5.nabble.com/usb-pc-cpu-flush-tp5975583p5976832.html Sent from the freebsd-usb mailing list archive at Nabble.com. ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
Hi, There might be an option somewhere you can set in the boot enviroment, which will disable the CPU cache ... I guess something is failing in that area. Else we would see the same with regular PCs too. --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 12/24/14 15:35, kott via freebsd-usb wrote: did try with freebsd11 also and it fails in the same point at usb_pc_cpu_flush. Is this a known issue ? any suggestions would help This is not a known issue. Which ARM kernel are you building? --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
did try with freebsd11 also and it fails in the same point at usb_pc_cpu_flush. Is this a known issue ? any suggestions would help -- View this message in context: http://freebsd.1045724.n5.nabble.com/usb-pc-cpu-flush-tp5975583p5975773.html Sent from the freebsd-usb mailing list archive at Nabble.com. ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 12/24/14 13:22, kott via freebsd-usb wrote: This platform is armv7 freebsd10. it does fail in the second usb_pc_cpu_flush Did you try booting a freebsd11 kernel? --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
This platform is armv7 freebsd10. it does fail in the second usb_pc_cpu_flush -- View this message in context: http://freebsd.1045724.n5.nabble.com/usb-pc-cpu-flush-tp5975583p5975763.html Sent from the freebsd-usb mailing list archive at Nabble.com. ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
Thanks platform is freebsd10 , armv7 based. -- View this message in context: http://freebsd.1045724.n5.nabble.com/usb-pc-cpu-flush-tp5975583p5975752.html Sent from the freebsd-usb mailing list archive at Nabble.com. ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 12/24/14 04:41, kott via freebsd-usb wrote: the value of "r0" is 0. Hi, In the function "_ehci_append_qh" do you know if it is the second or first call to "usb_pc_cpu_flush" which fails. The pointer which is passed as an argument to this function resides in the memory which is flushed, so most likely the cache sync operations are not working like expected! What platform is this? Non-freebsd? Did it work before? --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
the value of "r0" is 0. >>>>>> _ehci_append_qh: 0xf3dd2300 to 0xf3dc3c00 Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex ehci0 (usb_def_mtx) r = 0 (0xc73ece74) locked sys/dev/usb/usb_transfer.c exclusive sleep mutex USB device mutex (USB device mutex) r = 0 (0xc73da850) locked sys/de/usb/usb_transfer.c Fatal kernel mode data abort: 'Translation Fault (S)' trapframe: 0xf3de6be0 FSR=0005, FAR=0018, spsr=6113 r0 =, r1 =6113, r2 =00c0, r3 =00fd2302 r4 =f3dd2300, r5 =f3dc3c00, r6 =c12b244c, r7 =c76420c0 r8 =c73ed2a8, r9 =, r10=c7642000, r11=f3de6c3c r12=f3de6c40, ssp=f3de6c30, slr=c105ec48, pc =c1069e34 Stopped at usb_pc_cpu_flush+0xc: ldr r2, [r0, #0x018] >>>>>>> -- View this message in context: http://freebsd.1045724.n5.nabble.com/usb-pc-cpu-flush-tp5975583p5975691.html Sent from the freebsd-usb mailing list archive at Nabble.com. ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
Re: usb_pc_cpu_flush
On 12/23/14 19:26, kott wrote: Hello - I am seeing this fault when i plugin a external usb device, and was wondering if this indicates DMA/Cache coherency problem - _ehci_append_qh: 0xf3dd2300 to 0xf3dc3c00 Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex ehci0 (usb_def_mtx) r = 0 (0xc73ece74) locked @ sys/dev/usb/usb_transfer.c exclusive sleep mutex USB device mutex (USB device mutex) r = 0 (0xc73da850) locked @ /sys/dev/usb/usb_transfer.c Stopped at usb_pc_cpu_flush+0xc: ldr r2, [r0, #0x018] Thanks kott Hi, The flush is supposed to call some other functions which write the memory cache to RAM. It should not panic like that. Can you get the complete backtrace. I guess that some memory content might have been lost due to missing or wrong cache sync operations. What is the value of "r0" ? --HPS ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
usb_pc_cpu_flush
Hello - I am seeing this fault when i plugin a external usb device, and was wondering if this indicates DMA/Cache coherency problem - _ehci_append_qh: 0xf3dd2300 to 0xf3dc3c00 Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex ehci0 (usb_def_mtx) r = 0 (0xc73ece74) locked @ sys/dev/usb/usb_transfer.c exclusive sleep mutex USB device mutex (USB device mutex) r = 0 (0xc73da850) locked @ /sys/dev/usb/usb_transfer.c Stopped at usb_pc_cpu_flush+0xc: ldr r2, [r0, #0x018] Thanks kott -- View this message in context: http://freebsd.1045724.n5.nabble.com/usb-pc-cpu-flush-tp5975583.html Sent from the freebsd-usb mailing list archive at Nabble.com. ___ freebsd-usb@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-usb To unsubscribe, send any mail to "freebsd-usb-unsubscr...@freebsd.org"
