Hi Guennadi, Thank you for the patch.
As discussed over jabber, we're suffering from an AB-BA deadlock that this patch could make worse. The mmap code path takes mm->mmap_sem in the kernel' mmap handler before calling the driver's mmap handler. The driver will then lock the queue mutex before calling vb2_mmap (or let the core lock the vdev mutex). In the VIDIOC_QBUF code path, the driver will first lock the queue mutex (or let the core lock the vdev mutex) and call vb2_qbuf. This will then take mm- >mmap_sem to call get_user_pages for USERPTR buffers. The two locks are taken in different orders depending on the code paths, resulting in a possible AB-BA deadlock. mmap'ing USERPTR buffers doesn't make sense, so application will likely not experience any deadlock. However, if VIDIOC_CREATE_BUFS allows mixing MMAP and USERPTR buffers, we could suddenly see a rise of deadlock issues resulting from valid use cases. This problem needs to be fixed anyway, as a rogue application can currently produce a kernel deadlock. The fix can probably be pretty simple if we decide not to support mixing MMAP and USERPTR buffers, but it might become more complex if we need to support that. I'm interested in hearing what others think about this. -- Regards, Laurent Pinchart -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
