[Kernel-packages] [Bug 1831566] Re: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset
These patch sets are quite invasive for a stable kernel. It looks like the default behavior is untouched if cq_cpumask is zero. However, I would like to see some discussion with upstream as to the efficacy of IRQ affinity with this device. Mykyta - can you develop and send a patch for current upstream review ? ** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1831566 Title: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset Status in linux package in Ubuntu: Confirmed Status in linux source package in Bionic: New Bug description: We're running realtime application on Ubuntu 16.04 with linux-image 4.15 and found it impossible to get rid of jitter introduced by Intel NVMe IRQs. I'm providing here a patch which solved the issue for us. The realtime application is bound to isolated CPUs (one thread per CPU, nohz_full= in kernel cmdline, all IRQs moved to housekeeping CPUs), application doesn't use any linux kernel syscalls except in startup phase so we don't expect any interruptions of the application from the kernel or HW. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1831566] Re: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset
Sorry for the late action. I made a patch (attached) adding the nitpicks we discussed above, and a couple others. Please take a look and let me know what you think. If all is well, I'll submit this all to the mailing list to be merged. ** Patch added: "0001-nvme-add-nitpicks-for-cq_cpulist-module-parameter.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+attachment/5334962/+files/0001-nvme-add-nitpicks-for-cq_cpulist-module-parameter.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1831566 Title: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset Status in linux package in Ubuntu: Confirmed Bug description: We're running realtime application on Ubuntu 16.04 with linux-image 4.15 and found it impossible to get rid of jitter introduced by Intel NVMe IRQs. I'm providing here a patch which solved the issue for us. The realtime application is bound to isolated CPUs (one thread per CPU, nohz_full= in kernel cmdline, all IRQs moved to housekeeping CPUs), application doesn't use any linux kernel syscalls except in startup phase so we don't expect any interruptions of the application from the kernel or HW. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1831566] Re: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset
I agree that it is wise to intersect cq_cpumask with online_cpumask in cq_cpulist_set(), and if cpuset_weight(cq_cpumask & online_cpumask) == 0 -- ignore cq_cpumask and stick with default behavior. I don't think cpu offlining is problematic here. If CPU is offlined -- the IRQ (and queue) will be served by another online CPU. Yes, it will break IRQ affinity and make cq_cpumask setting pointless, but this is system administrator's problem. The parameter is only useful for fine tuning and some resource planning in advance is required. I don't think that this side effect is enough to motivate "unsafe" tag. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1831566 Title: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset Status in linux package in Ubuntu: Confirmed Bug description: We're running realtime application on Ubuntu 16.04 with linux-image 4.15 and found it impossible to get rid of jitter introduced by Intel NVMe IRQs. I'm providing here a patch which solved the issue for us. The realtime application is bound to isolated CPUs (one thread per CPU, nohz_full= in kernel cmdline, all IRQs moved to housekeeping CPUs), application doesn't use any linux kernel syscalls except in startup phase so we don't expect any interruptions of the application from the kernel or HW. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1831566] Re: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset
@mykizi-ericsson Could I get an update for my last comment? If CPU hotplugging can't be addressed then perhaps it'd be alright to just mark the new module parameter as unsafe... -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1831566 Title: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset Status in linux package in Ubuntu: Confirmed Bug description: We're running realtime application on Ubuntu 16.04 with linux-image 4.15 and found it impossible to get rid of jitter introduced by Intel NVMe IRQs. I'm providing here a patch which solved the issue for us. The realtime application is bound to isolated CPUs (one thread per CPU, nohz_full= in kernel cmdline, all IRQs moved to housekeeping CPUs), application doesn't use any linux kernel syscalls except in startup phase so we don't expect any interruptions of the application from the kernel or HW. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1831566] Re: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset
Thanks for the clarification, that makes sense to me now. It doesn't look like the patch handles CPU hotplugging though; a cpumask filled with *only* offline CPUs can be passed as a parameter, or all of the CPUs in the cpumask can be offlined after the setup is finished. This isn't a problem with the default behavior because num_possible_cpus() guarantees that there will always be at least one online CPU for the nvme queue (since the CPU hotplug code doesn't allow all CPUs to be offlined). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1831566 Title: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset Status in linux package in Ubuntu: Confirmed Bug description: We're running realtime application on Ubuntu 16.04 with linux-image 4.15 and found it impossible to get rid of jitter introduced by Intel NVMe IRQs. I'm providing here a patch which solved the issue for us. The realtime application is bound to isolated CPUs (one thread per CPU, nohz_full= in kernel cmdline, all IRQs moved to housekeeping CPUs), application doesn't use any linux kernel syscalls except in startup phase so we don't expect any interruptions of the application from the kernel or HW. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1831566] Re: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset
The permission is set to 0 on a purpose -- to prevent runtime changes. The parameter can be still used on modprobe/insmod or on kernel command line. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1831566 Title: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset Status in linux package in Ubuntu: Confirmed Bug description: We're running realtime application on Ubuntu 16.04 with linux-image 4.15 and found it impossible to get rid of jitter introduced by Intel NVMe IRQs. I'm providing here a patch which solved the issue for us. The realtime application is bound to isolated CPUs (one thread per CPU, nohz_full= in kernel cmdline, all IRQs moved to housekeeping CPUs), application doesn't use any linux kernel syscalls except in startup phase so we don't expect any interruptions of the application from the kernel or HW. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1831566] Re: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset
It looks like the supplied patch is a no-op. A module parameter is introduced with a .set function, but the parameter does not have any write permission specified so the .set function can never execute (since userspace won't have permission to write to the module parameter file). module_param_cb(cq_cpulist, _cpulist_ops, NULL, 0); ~~^~ This 0 provided to module_param_cb() is the octal UNIX permissions argument to module_param_cb(). The correct value should be 0200 in this case, to specify write-only permission. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1831566 Title: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset Status in linux package in Ubuntu: Confirmed Bug description: We're running realtime application on Ubuntu 16.04 with linux-image 4.15 and found it impossible to get rid of jitter introduced by Intel NVMe IRQs. I'm providing here a patch which solved the issue for us. The realtime application is bound to isolated CPUs (one thread per CPU, nohz_full= in kernel cmdline, all IRQs moved to housekeeping CPUs), application doesn't use any linux kernel syscalls except in startup phase so we don't expect any interruptions of the application from the kernel or HW. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1831566] Re: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset
** Tags added: patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1831566 Title: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset Status in linux package in Ubuntu: Confirmed Bug description: We're running realtime application on Ubuntu 16.04 with linux-image 4.15 and found it impossible to get rid of jitter introduced by Intel NVMe IRQs. I'm providing here a patch which solved the issue for us. The realtime application is bound to isolated CPUs (one thread per CPU, nohz_full= in kernel cmdline, all IRQs moved to housekeeping CPUs), application doesn't use any linux kernel syscalls except in startup phase so we don't expect any interruptions of the application from the kernel or HW. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1831566] Re: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset
** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1831566 Title: [realtime app] not possible to redirect drivers/nvme IRQs from realtime cpuset Status in linux package in Ubuntu: Confirmed Bug description: We're running realtime application on Ubuntu 16.04 with linux-image 4.15 and found it impossible to get rid of jitter introduced by Intel NVMe IRQs. I'm providing here a patch which solved the issue for us. The realtime application is bound to isolated CPUs (one thread per CPU, nohz_full= in kernel cmdline, all IRQs moved to housekeeping CPUs), application doesn't use any linux kernel syscalls except in startup phase so we don't expect any interruptions of the application from the kernel or HW. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1831566/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
