Issue on unmap_grant_pages before releasing dmabuf
Greetings. I need your advise about the problem I'm facing right now: I'm working on the new type of dmabuf export implementation. This is going to be new ioctl to the gntdev and it's purpose is to use external buffer, provided by file descriptor as the backing storage during export to grant refs. Few words about the functionality I'm working on right now: My setup is based on IMX8QM (please see PS below if you need configuration details) When using dma-buf exporter to create dma-buf with backing storage and map it to the grant refs, provided from the domain, we've met a problem, that several HW (i.MX8 gpu in our case) do not support external buffer and requires backing storage to be created using it's native tools (eglCreateImageKHR returns EGL_NO_IMAGE_KHR for buffers, which were not created using gbm_bo_create). That's why new ioctls were added to be able to pass existing dma-buffer fd as input parameter and use it as backing storage to export to refs. Kernel version on IMX8QM board is 5.10.72 and itworks fine on this kernel version. New ioctls source code can be found here: https://github.com/oleksiimoisieiev/linux/tree/gntdev_map_buf_upstr_for-linus-6.1_2 Now regarding the problem I've met when rebased those code on master: On my test stand I use DRM_IOCTL_MODE_CREATE_DUMB/DRM_IOCTL_MODE_DESTROY_DUMB ioctls to allocate buffer and I'm observing the following backtrace on DRM_IOCTL_MODE_DESTROY_DUMB: Unable to handle kernel paging request at virtual address 00038798 Mem abort info: ESR = 0x9605 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x05: level 1 translation fault Data abort info: ISV = 0, ISS = 0x0005 CM = 0, WnR = 0 user pgtable: 4k pages, 48-bit VAs, pgdp=6df98000 [00038798] pgd=080064f4f003, p4d=080064f4f003, pud= Internal error: Oops: 9605 [#1] PREEMPT SMP Modules linked in: xen_pciback overlay crct10dif_ce ip_tables x_tables ipv6 PU: 0 PID: 34 Comm: kworker/0:1 Not tainted 6.0.0 #85 Hardware name: linux,dummy-virt (DT) Workqueue: events virtio_gpu_dequeue_ctrl_func pstate: 00c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : check_move_unevictable_folios+0xb8/0x4d0 lr : check_move_unevictable_folios+0xb4/0x4d0 sp : 881a3ad0 x29: 881a3ad0 x28: 03856ac98800 x27: x26: de7b168ee9d8 x25: 03856ae26008 x24: x23: de7b1758d6c0 x22: 0001 x21: 881a3b68 x20: 0001 x19: fc0e15935040 x18: x17: 250a68e3d000 x16: 0012 x15: 8000881a38d7 x14: x13: de7b175a3150 x12: 2c55 x11: 0ec7 x10: de7b176113f8 x9 : de7b175a3150 x8 : 00014ec7 x7 : de7b175fb150 x6 : 881a3b70 x5 : 0001 x4 : x3 : 03856ac98850 x2 : x1 : x0 : 00038700 Call trace: check_move_unevictable_folios+0xb8/0x4d0 check_move_unevictable_pages+0x8c/0x110 drm_gem_put_pages+0x118/0x198 drm_gem_shmem_put_pages_locked+0x4c/0x70 drm_gem_shmem_unpin+0x30/0x50 virtio_gpu_cleanup_object+0x84/0x130 virtio_gpu_cmd_unref_cb+0x18/0x2c virtio_gpu_dequeue_ctrl_func+0x124/0x290 process_one_work+0x1d0/0x320 worker_thread+0x14c/0x444 kthread+0x10c/0x110 ret_from_fork+0x10/0x20 Code: 97fc3fe1 aa1303e0 94003ac7 b480 (f9404c00) ---[ end trace ]--- After some investigation I think I've found the cause of the problem: This is the functionality, added in commit 3f9f1c67572f5e5e6dc84216d48d1480f3c4fcf6 which introduces safe mechanism to unmap grant pages which is waiting until page_count(page) = 1 before doing unmap. The problem is that DRM_IOCTL_MODE_CREATE_DUMB creates buffer where page_count(page) = 2. On my QEMU test stand I'm using Xen 4.16 (aarch64) with debian based Dom0 + DomU on the latest kernels. I've created some apps for testing: The first one is to allocate grant refs on DomU: https://github.com/oleksiimoisieiev/linux/tree/gntdev_map_buf_upstr_for-linus-6.1_2 The name is test.ko and it can be built using command: cd ./test; make NOTE: makefile expects kernel build to be present on ../../test-build directory. It should be run on DomU using command: insmod test.ko; echo "create" > /sys/kernel/etx_sysfs/etx_value Result will be the following: [ 126.104903] test: loading out-of-tree module taints kernel. [ 126.150586] Sysfs - Write!!! [ 126.150773] create [ 126.150773] [ 126.150888] Hello, World! [ 126.151203] Hello, World! [ 126.151324] gref 301 [ 126.151376] addr 0883d000 [ 126.151431] gref 302 [ 126.151454] addr 0883e000 [ 126.151478] gref 303 [ 126.151497] addr 0883f000 [ 126.151525] gref 304 [ 126.151546] addr 0884 [ 126.151573] gref 305 [ 126.151593] addr 08841000 The second is for dom0 and can be found here: https://github.com/oleksiimois
Re: Issue on unmap_grant_pages before releasing dmabuf
On Wed, Oct 19, 2022 at 02:30:13PM +0200, Juergen Gross wrote: > On 19.10.22 12:43, Oleksii Moisieiev wrote: > > Greetings. > > > > I need your advise about the problem I'm facing right now: > > I'm working on the new type of dmabuf export implementation. This > > is going to be new ioctl to the gntdev and it's purpose is to use > > external buffer, provided by file descriptor as the backing storage > > during export to grant refs. > > Few words about the functionality I'm working on right now: > > My setup is based on IMX8QM (please see PS below if you need > > configuration details) > > > > When using dma-buf exporter to create dma-buf with backing storage and > > map it to the grant refs, provided from the domain, we've met a problem, > > that several HW (i.MX8 gpu in our case) do not support external buffer > > and requires backing storage to be created using it's native tools > > (eglCreateImageKHR returns EGL_NO_IMAGE_KHR for buffers, which were not > > created using gbm_bo_create). > > That's why new ioctls were added to be able to pass existing dma-buffer > > fd as input parameter and use it as backing storage to export to refs. > > Kernel version on IMX8QM board is 5.10.72 and itworks fine on this kernel > > version. > > > > New ioctls source code can be found here: > > > > https://github.com/oleksiimoisieiev/linux/tree/gntdev_map_buf_upstr_for-linus-6.1_2 > > Now regarding the problem I've met when rebased those code on master: > > On my test stand I use > > DRM_IOCTL_MODE_CREATE_DUMB/DRM_IOCTL_MODE_DESTROY_DUMB ioctls > > to allocate buffer and I'm observing the following backtrace on > > DRM_IOCTL_MODE_DESTROY_DUMB: > > > > Unable to handle kernel paging request at virtual address 00038798 > > Mem abort info: > >ESR = 0x9605 > >EC = 0x25: DABT (current EL), IL = 32 bits > >SET = 0, FnV = 0 > >EA = 0, S1PTW = 0 > >FSC = 0x05: level 1 translation fault > > Data abort info: > >ISV = 0, ISS = 0x0005 > >CM = 0, WnR = 0 > > user pgtable: 4k pages, 48-bit VAs, pgdp=6df98000 > > [00038798] pgd=080064f4f003, p4d=080064f4f003, > > pud= > > Internal error: Oops: 9605 [#1] PREEMPT SMP > > Modules linked in: xen_pciback overlay crct10dif_ce ip_tables x_tables ipv6 > > PU: 0 PID: 34 Comm: kworker/0:1 Not tainted 6.0.0 #85 > > Hardware name: linux,dummy-virt (DT) > > Workqueue: events virtio_gpu_dequeue_ctrl_func > > pstate: 00c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) > > pc : check_move_unevictable_folios+0xb8/0x4d0 > > lr : check_move_unevictable_folios+0xb4/0x4d0 > > sp : 881a3ad0 > > x29: 881a3ad0 x28: 03856ac98800 x27: > > x26: de7b168ee9d8 x25: 03856ae26008 x24: > > x23: de7b1758d6c0 x22: 0001 x21: 881a3b68 > > x20: 0001 x19: fc0e15935040 x18: > > x17: 250a68e3d000 x16: 0012 x15: 8000881a38d7 > > x14: x13: de7b175a3150 x12: 2c55 > > x11: 0ec7 x10: de7b176113f8 x9 : de7b175a3150 > > x8 : 00014ec7 x7 : de7b175fb150 x6 : 881a3b70 > > x5 : 0001 x4 : x3 : 03856ac98850 > > x2 : x1 : x0 : 00038700 > > Call trace: > > check_move_unevictable_folios+0xb8/0x4d0 > > check_move_unevictable_pages+0x8c/0x110 > > drm_gem_put_pages+0x118/0x198 > > drm_gem_shmem_put_pages_locked+0x4c/0x70 > > drm_gem_shmem_unpin+0x30/0x50 > > virtio_gpu_cleanup_object+0x84/0x130 > > virtio_gpu_cmd_unref_cb+0x18/0x2c > > virtio_gpu_dequeue_ctrl_func+0x124/0x290 > > process_one_work+0x1d0/0x320 > > worker_thread+0x14c/0x444 > > kthread+0x10c/0x110 > > ret_from_fork+0x10/0x20 > > Code: 97fc3fe1 aa1303e0 94003ac7 b480 (f9404c00) > > ---[ end trace ]--- > > > > After some investigation I think I've found the cause of the problem: > > This is the functionality, added in commit > > 3f9f1c67572f5e5e6dc84216d48d1480f3c4fcf6 which > > introduces safe mechanism to unmap grant pages which is waiting until > > page_count(page) = 1 > > before doing unmap. > > The problem is that DRM_IOCTL_MODE_CREATE_DUMB creates buffer where > > page_count(page) = 2. > > > > On my QEMU test stand I'm using Xen 4.16 (aarch64) with debian based Dom0 + > > DomU on the latest > > kernels. > > I've created some apps for testing: > > The first one is to allocate grant refs on DomU: > > https://github.com/oleksiimoisieiev/linux/tree/gntdev_map_buf_upstr_for-linus-6.1_2 > > The name is test.ko and it can be built using command: > > cd ./test; make > > NOTE: makefile expects kernel build to be present on ../../test-build > > directory. > > > > It should be run on DomU using command: > > insmod test.ko; echo "create" > /sys/kernel/etx_sysfs/etx_value > > > > Result will be the following: > > [ 126.104903] test: loading out-of
Re: Issue on unmap_grant_pages before releasing dmabuf
On 19.10.22 12:43, Oleksii Moisieiev wrote: Greetings. I need your advise about the problem I'm facing right now: I'm working on the new type of dmabuf export implementation. This is going to be new ioctl to the gntdev and it's purpose is to use external buffer, provided by file descriptor as the backing storage during export to grant refs. Few words about the functionality I'm working on right now: My setup is based on IMX8QM (please see PS below if you need configuration details) When using dma-buf exporter to create dma-buf with backing storage and map it to the grant refs, provided from the domain, we've met a problem, that several HW (i.MX8 gpu in our case) do not support external buffer and requires backing storage to be created using it's native tools (eglCreateImageKHR returns EGL_NO_IMAGE_KHR for buffers, which were not created using gbm_bo_create). That's why new ioctls were added to be able to pass existing dma-buffer fd as input parameter and use it as backing storage to export to refs. Kernel version on IMX8QM board is 5.10.72 and itworks fine on this kernel version. New ioctls source code can be found here: https://github.com/oleksiimoisieiev/linux/tree/gntdev_map_buf_upstr_for-linus-6.1_2 Now regarding the problem I've met when rebased those code on master: On my test stand I use DRM_IOCTL_MODE_CREATE_DUMB/DRM_IOCTL_MODE_DESTROY_DUMB ioctls to allocate buffer and I'm observing the following backtrace on DRM_IOCTL_MODE_DESTROY_DUMB: Unable to handle kernel paging request at virtual address 00038798 Mem abort info: ESR = 0x9605 EC = 0x25: DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 FSC = 0x05: level 1 translation fault Data abort info: ISV = 0, ISS = 0x0005 CM = 0, WnR = 0 user pgtable: 4k pages, 48-bit VAs, pgdp=6df98000 [00038798] pgd=080064f4f003, p4d=080064f4f003, pud= Internal error: Oops: 9605 [#1] PREEMPT SMP Modules linked in: xen_pciback overlay crct10dif_ce ip_tables x_tables ipv6 PU: 0 PID: 34 Comm: kworker/0:1 Not tainted 6.0.0 #85 Hardware name: linux,dummy-virt (DT) Workqueue: events virtio_gpu_dequeue_ctrl_func pstate: 00c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : check_move_unevictable_folios+0xb8/0x4d0 lr : check_move_unevictable_folios+0xb4/0x4d0 sp : 881a3ad0 x29: 881a3ad0 x28: 03856ac98800 x27: x26: de7b168ee9d8 x25: 03856ae26008 x24: x23: de7b1758d6c0 x22: 0001 x21: 881a3b68 x20: 0001 x19: fc0e15935040 x18: x17: 250a68e3d000 x16: 0012 x15: 8000881a38d7 x14: x13: de7b175a3150 x12: 2c55 x11: 0ec7 x10: de7b176113f8 x9 : de7b175a3150 x8 : 00014ec7 x7 : de7b175fb150 x6 : 881a3b70 x5 : 0001 x4 : x3 : 03856ac98850 x2 : x1 : x0 : 00038700 Call trace: check_move_unevictable_folios+0xb8/0x4d0 check_move_unevictable_pages+0x8c/0x110 drm_gem_put_pages+0x118/0x198 drm_gem_shmem_put_pages_locked+0x4c/0x70 drm_gem_shmem_unpin+0x30/0x50 virtio_gpu_cleanup_object+0x84/0x130 virtio_gpu_cmd_unref_cb+0x18/0x2c virtio_gpu_dequeue_ctrl_func+0x124/0x290 process_one_work+0x1d0/0x320 worker_thread+0x14c/0x444 kthread+0x10c/0x110 ret_from_fork+0x10/0x20 Code: 97fc3fe1 aa1303e0 94003ac7 b480 (f9404c00) ---[ end trace ]--- After some investigation I think I've found the cause of the problem: This is the functionality, added in commit 3f9f1c67572f5e5e6dc84216d48d1480f3c4fcf6 which introduces safe mechanism to unmap grant pages which is waiting until page_count(page) = 1 before doing unmap. The problem is that DRM_IOCTL_MODE_CREATE_DUMB creates buffer where page_count(page) = 2. On my QEMU test stand I'm using Xen 4.16 (aarch64) with debian based Dom0 + DomU on the latest kernels. I've created some apps for testing: The first one is to allocate grant refs on DomU: https://github.com/oleksiimoisieiev/linux/tree/gntdev_map_buf_upstr_for-linus-6.1_2 The name is test.ko and it can be built using command: cd ./test; make NOTE: makefile expects kernel build to be present on ../../test-build directory. It should be run on DomU using command: insmod test.ko; echo "create" > /sys/kernel/etx_sysfs/etx_value Result will be the following: [ 126.104903] test: loading out-of-tree module taints kernel. [ 126.150586] Sysfs - Write!!! [ 126.150773] create [ 126.150773] [ 126.150888] Hello, World! [ 126.151203] Hello, World! [ 126.151324] gref 301 [ 126.151376] addr 0883d000 [ 126.151431] gref 302 [ 126.151454] addr 0883e000 [ 126.151478] gref 303 [ 126.151497] addr 0883f000 [ 126.151525] gref 304 [ 126.151546] addr 0884 [ 126.151573] gref 305 [ 126.151593] addr 08841000 The second i
