Imply and set strict CPU affinity for WQ_RT workqueues for even lower
worst case execution latency.
With this, a benchmark which measures the GPU submit latency against a
real-time Vulkan queue when faced with CPU saturation, highlights a
dramatic reduction in maximum observed latencies and further reduction in
the median, and 95- and 98-the percentiles:
us 7.3.0 WQ_RT +strict_affinity
median 30 21 11
min 2 3 2
max 1038 941 108
95pct 2384 49 28
98pct 2922 63 51
Signed-off-by: Tvrtko Ursulin <[email protected]>
Cc: Boris Brezillon <[email protected]>
Cc: Chia-I Wu <[email protected]>
Cc: Liviu Dudau <[email protected]>
Cc: Matthew Brost <[email protected]>
Cc: Steven Price <[email protected]>
Cc: Tejun Heo <[email protected]>
---
Documentation/core-api/workqueue.rst | 5 +++--
kernel/workqueue.c | 13 ++++++++++---
2 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/Documentation/core-api/workqueue.rst
b/Documentation/core-api/workqueue.rst
index 54697907f2b0..d699c3832b19 100644
--- a/Documentation/core-api/workqueue.rst
+++ b/Documentation/core-api/workqueue.rst
@@ -226,8 +226,9 @@ resources, scheduled and executed.
implements concurrency management among its workers.
``WQ_RT``
- Real-time priority workqueues must be created as unbound. Their worker
threads
- use the FIFO scheduling policy with the lowest applicable priority.
+ Real-time priority workqueues must be created as unbound and will be
+ configured with the strict CPU affinity set. Their worker threads use the
FIFO
+ scheduling policy with the lowest applicable priority.
To be used sparingly for use cases such as the real-time GPU rendering
contexts accessible to privileged clients.
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 637240667cf7..26a26831bee7 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -5723,8 +5723,12 @@ static void
sanitize_immutable_workqueue_attrs(struct workqueue_struct *wq,
struct workqueue_attrs *attrs)
{
- if (wq->flags & WQ_RT)
+ if (wq->flags & WQ_RT) {
attrs->prio = RT_PRIORITY;
+ /* Strict CPU affinity for low latency execution. */
+ attrs->affn_scope = WQ_AFFN_CPU;
+ attrs->affn_strict = true;
+ }
if (wq->flags & __WQ_ORDERED)
attrs->ordered = true;
@@ -7847,8 +7851,11 @@ static umode_t wq_sysfs_unbound_group_visible(struct
kobject *kobj,
if (!(wq->flags & WQ_UNBOUND))
return 0;
- /* Do not allow priority changes for RT workers. */
- if ((wq->flags & WQ_RT) && (attr == &dev_attr_nice.attr))
+ /* Do not allow priority and affinity changes for RT workers. */
+ if ((wq->flags & WQ_RT) &&
+ (attr == &dev_attr_nice.attr ||
+ attr == &dev_attr_affinity_scope.attr ||
+ attr == &dev_attr_affinity_strict.attr))
return 0444;
return attr->mode;
--
2.56.0
