Hi, Jean
On 2022/4/22 下午6:11, Jean-Philippe Brucker wrote:
On Fri, Apr 22, 2022 at 05:03:10PM +0800, [email protected] wrote:
[...]
Have tested, still got some issue with our openssl-engine.
1. If openssl-engine does not register rsa, nginx works well.
2. If openssl-engine register rsa, nginx also works, but ioasid is not
freed when nginx stop.
IMPLEMENT_DYNAMIC_BIND_FN(bind_fn)
bind_fn
ENGINE_set_RSA(e, rsa_methods())
destroy_fn
If ENGINE_set_RSA is set, nginx start and stop will NOT call destroy_fn.
Even rsa_methods is almost new via RSA_meth_new.
In 5.18-rcx, this caused ioasid not freed in nginx start and stop.
In 5.17, though destroy_fn is not called, but ioasid is freed when nginx
stop, so not noticed this issue before.
1. uacce_fops_release
In 5.16 or 5.17
In fact, we aslo has the issue: openssl engine does not call destroy_fn ->
close(uacce_fd)
But system will automatically close all opened fd,
so uacce_fops_release is also called and free ioasid.
Have one experiment, not call close fd
log: open uacce fd but no close
[ 2583.471225] dump_backtrace+0x0/0x1a0
[ 2583.474876] show_stack+0x20/0x30
[ 2583.478178] dump_stack_lvl+0x8c/0xb8
[ 2583.481825] dump_stack+0x18/0x34
[ 2583.485126] uacce_fops_release+0x44/0xdc
[ 2583.489117] __fput+0x78/0x240
[ 2583.492159] ____fput+0x18/0x28
[ 2583.495288] task_work_run+0x88/0x160
[ 2583.498936] do_notify_resume+0x214/0x490
[ 2583.502927] el0_svc+0x58/0x70
[ 2583.505968] el0t_64_sync_handler+0xb0/0xb8
[ 2583.510132] el0t_64_sync+0x1a0/0x1a4
[ 2583.582292] uacce_fops_release q=00000000d6674128
In 5.18, since refcount was add.
The opened uacce fd was not closed automatically by system
So we see the issue.
log: open uacce fd but no close
[ 106.360140] uacce_fops_open q=00000000ccc38d74
[ 106.364929] ioasid_alloc ioasid=1
[ 106.368585] iommu_sva_alloc_pasid pasid=1
[ 106.372943] iommu_sva_bind_device handle=000000006cca298a
// ioasid is not free
I'm trying to piece together what happens from the kernel point of view.
* master process with mm A opens a queue fd through uacce, which calls
iommu_sva_bind_device(dev, A) -> PASID 1
* master forks and exits. Child (daemon) gets mm B, inherits the queue fd.
The device is still bound to mm A with PASID 1, since the queue fd is
still open.
We discussed this before, but I don't remember where we left off. The
child can't use the queue because its mappings are not copied on fork(),
and the queue is still bound to the parent mm A. The child either needs to
open a new queue or take ownership of the old one with a new uacce ioctl.
Yes, currently nginx aligned with the case.
Child process (worker process) reopen uacce,
Master process (do init) open uacce, iommu_sva_bind_device(dev, A) ->
PASID 1
Master process fork Child (daemon) and exit.
Child (daemon) does not use PASID 1 any more, only fork and manage
worker process.
Worker process reopen uacce, iommu_sva_bind_device(dev, B) PASID 2
So it is expected.
Is that the "IMPLEMENT_DYNAMIC_BIND_FN()" you mention, something out of
tree? This operation should unbind from A before binding to B, no?
Otherwise we leak PASID 1.
In 5.16 PASID 1 from master is hold until nginx service stop.
nginx start
master:
iommu_sva_alloc_pasid mm->pasid=1 // master process
lynx https start:
iommu_sva_alloc_pasid mm->pasid=2 //worker process
nginx stop: from fops_release
iommu_sva_free_pasid mm->pasid=2 // worker process
iommu_sva_free_pasid mm->pasid=1 // master process
Have one silly question.
kerne driver
fops_open
iommu_sva_bind_device
fops_release
iommu_sva_unbind_device
application
main()
fd = open
return;
Application exit but not close(fd), is it expected fops_release will be
called automatically by system?
On 5.17
fops_release is called automatically, as well as iommu_sva_unbind_device.
On 5.18-rc1.
fops_release is not called, have to manually call close(fd)
Since nginx may have a issue, it does not call close(fd) when nginx -s quit.
Thanks
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
