Hi everyone,
Short version:
I'm having an issue with direct DMA transfer from a device to host memory.
It seems some of the data is not transferring to the appropriate page.
Some more details:
I'm debugging a home made PCI driver for our board (Kalray), attached to a
x86_64 host running centos7 (3.10.0-327.el7.x86_64)
In the current case, a userland application transfers back and forth data
through read/write operations on a file.
On the kernel side, it triggers DMA transfers through the PCI to/from our board
memory.
We followed what pretty much all docs said about direct I/O to user buffers:
1) get_user_pages() (in the current case, it's at most 16 pages at once)
2) convert to a scatterlist
3) pci_map_sg
4) eventually coalesce sg (Intel IOMMU is enabled, so it's usually possible)
4) A lot of DMA engine handling code, using the dmaengine layer and virt-dma
5) wait for transfer complete, in the mean time, go back to (1) to schedule
more work, if any
6) pci_unmap_sg
7) for read (card2host) transfer, set_page_dirty_lock
8) page_cache_release
In 99,9999% it works perfectly.
However, I have one userland application where a few pages are not written by a
read (card2host) transfer.
The buffer is memset them to a different value so I can check that nothing has
overwritten them.
I know (PCI protocol analyser) that the data left our board for the "right"
address (the one set in the sg by pci_map_sg).
I tried reading the data between the pci_unmap_sg and the set_page_dirty, using
uint32_t *addr = page_address(trans->pages[0]);
dev_warn(&pdata->pdev->dev, "val = %x\n", *addr);
and it has the expected value.
But if I try to copy_from_user (using the address coming from userland, the one
passed to get_user_pages), the data has not been written and I see the memset
value.
I manage to build a test case that fails all the time, but never at the same
offset within the buffer.
It's always in the middle (never at the start nor end), for a few pages long
(varies between runs).
Am I missing something? Could it be possible that I'm not writing to the right
page?
If you need more information, feel free to ask
Thanks in advance
Nicolas
