On 2020/11/24 23:08, Joel Fernandes wrote:
>>>
>>> Core-Scheduling
>>> ===============
>>> Enclosed is series v9 of core scheduling.
>>> v9 is rebased on tip/master (fe4adf6f92c4 ("Merge branch 'irq/core'"))..
>>> I hope that this version is acceptable to be merged (pending any new review
>>> comments that arise) as the main issues in the past are all resolved:
>>> 1. Vruntime comparison.
>>> 2. Documentation updates.
>>> 3. CGroup and per-task interface developed by Google and Oracle.
>>> 4. Hotplug fixes.
>>> Almost all patches also have Reviewed-by or Acked-by tag. See below for full
>>> list of changes in v9.
>>>
>>> Introduction of feature
>>> =======================
>>> Core scheduling is a feature that allows only trusted tasks to run
>>> concurrently on cpus sharing compute resources (eg: hyperthreads on a
>>> core). The goal is to mitigate the core-level side-channel attacks
>>> without requiring to disable SMT (which has a significant impact on
>>> performance in some situations). Core scheduling (as of v7) mitigates
>>> user-space to user-space attacks and user to kernel attack when one of
>>> the siblings enters the kernel via interrupts or system call.
>>>
>>> By default, the feature doesn't change any of the current scheduler
>>> behavior. The user decides which tasks can run simultaneously on the
>>> same core (for now by having them in the same tagged cgroup). When a tag
>>> is enabled in a cgroup and a task from that cgroup is running on a
>>> hardware thread, the scheduler ensures that only idle or trusted tasks
>>> run on the other sibling(s). Besides security concerns, this feature can
>>> also be beneficial for RT and performance applications where we want to
>>> control how tasks make use of SMT dynamically.
>>>
>>> Both a CGroup and Per-task interface via prctl(2) are provided for
>>> configuring
>>> core sharing. More details are provided in documentation patch. Kselftests
>>> are
>>> provided to verify the correctness/rules of the interface.
>>>
>>> Testing
>>> =======
>>> ChromeOS testing shows 300% improvement in keypress latency on a Google
>>> docs key press with Google hangout test (the maximum latency drops from
>>> 150ms
>>> to 50ms for keypresses).
>>>
>>> Julien: TPCC tests showed improvements with core-scheduling as below. With
>>> kernel
>>> protection enabled, it does not show any regression. Possibly ASI will
>>> improve
>>> the performance for those who choose kernel protection (can be toggled
>>> through
>>> sched_core_protect_kernel sysctl).
>>> average stdev diff
>>> baseline (SMT on) 1197.272 44.78312824
>>> core sched ( kernel protect) 412.9895 45.42734343 -65.51%
>>> core sched (no kernel protect) 686.6515 71.77756931 -42.65%
>>> nosmt 408.667 39.39042872 -65.87%
>>> (Note these results are from v8).
>>>
>>> Vineeth tested sysbench and does not see any regressions.
>>> Hong and Aubrey tested v9 and see results similar to v8. There is a known
>>> issue
>>> with uperf that does regress. This appears to be because of ksoftirq heavily
>>> contending with other tasks on the core. The consensus is this can be
>>> improved
>>> in the future.
>>>
>>> Changes in v9
>>> =============
>>> - Note that the vruntime snapshot change is written in 2 patches to show the
>>> progression of the idea and prevent merge conflicts:
>>> sched/fair: Snapshot the min_vruntime of CPUs on force idle
>>> sched: Improve snapshotting of min_vruntime for CGroups
>>> Same with the RT priority inversion change:
>>> sched: Fix priority inversion of cookied task with sibling
>>> sched: Improve snapshotting of min_vruntime for CGroups
>>> - Disable coresched on certain AMD HW.
>>>
Adding workloads and negative case test results for core scheduling v9 posted:
- kernel under test:
-- coresched community v9 posted from
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=sched/coresched-v9-posted
(tag: sched/coresched-v9-posted)
-- latest commit: d48636e429de (HEAD -> coresched-v9-posted, tag:
sched/coresched-v9-posted) sched: Debug bits...
-- coresched=on kernel parameter applied
- workloads:
-- A. sysbench cpu (192 threads) + sysbench cpu (192 threads)
-- B. sysbench cpu (192 threads) + sysbench mysql (192 threads, mysqld
forced into the same cgroup)
-- C. uperf netperf.xml (192 threads over TCP or UDP protocol
separately)
-- D. will-it-scale context_switch via pipe (192 threads)
- negative case:
-- A. continuously toggle cpu.core_tag, during full loading uperf
workload running with cs_on
-- B. continuously toggle smt setting via
/sys/devices/system/cpu/smt/control, during full loading uperf workload running
with cs_on
-- C. continuously cgroup switch between cs_on cgroup and cs_off cgroup
via cgclassify, during full loading uperf workload running
- test machine setup:
CPU(s): 192
On-line CPU(s) list: 0-191
Thread(s) per core: 2
Core(s) per socket: 48
Socket(s): 2
NUMA node(s): 4
- test results of workloads, no obvious performance drop compared to community
v8 build:
-- workload A:
+----------------------+------+----------------------+------------------------+
| workloads | ** | sysbench cpu * 192 | sysbench cpu *
192 |
+======================+======+======================+========================+
| cgroup | ** | cg_sysbench_cpu_0 |
cg_sysbench_cpu_1 |
+----------------------+------+----------------------+------------------------+
| record_item | ** | Tput_avg (events/s) | Tput_avg
(events/s) |
+----------------------+------+----------------------+------------------------+
| coresched_normalized | ** | 0.97 | 1.02
|
+----------------------+------+----------------------+------------------------+
| default_normalized | ** | 1.00 | 1.00
|
+----------------------+------+----------------------+------------------------+
| smtoff_normalized | ** | 0.60 | 0.60
|
+----------------------+------+----------------------+------------------------+
-- workload B:
+----------------------+------+----------------------+------------------------+
| workloads | ** | sysbench cpu * 192 | sysbench mysql *
192 |
+======================+======+======================+========================+
| cgroup | ** | cg_sysbench_cpu_0 |
cg_sysbench_mysql_0 |
+----------------------+------+----------------------+------------------------+
| record_item | ** | Tput_avg (events/s) | Tput_avg
(events/s) |
+----------------------+------+----------------------+------------------------+
| coresched_normalized | ** | 0.94 | 0.88
|
+----------------------+------+----------------------+------------------------+
| default_normalized | ** | 1.00 | 1.00
|
+----------------------+------+----------------------+------------------------+
| smtoff_normalized | ** | 0.56 | 0.84
|
+----------------------+------+----------------------+------------------------+
-- workload C:
+----------------------+------+---------------------------+---------------------------+
| workloads | ** | uperf netperf TCP * 192 | uperf
netperf UDP * 192 |
+======================+======+===========================+===========================+
| cgroup | ** | cg_uperf | cg_uperf
|
+----------------------+------+---------------------------+---------------------------+
| record_item | ** | Tput_avg (Gb/s) | Tput_avg
(Gb/s) |
+----------------------+------+---------------------------+---------------------------+
| coresched_normalized | ** | 0.64 | 0.68
|
+----------------------+------+---------------------------+---------------------------+
| default_normalized | ** | 1.00 | 1.00
|
+----------------------+------+---------------------------+---------------------------+
| smtoff_normalized | ** | 0.92 | 0.89
|
+----------------------+------+---------------------------+---------------------------+
-- workload D:
+----------------------+------+-------------------------------+
| workloads | ** | will-it-scale * 192 |
| | | (pipe based context_switch) |
+======================+======+===============================+
| cgroup | ** | cg_will-it-scale |
+----------------------+------+-------------------------------+
| record_item | ** | threads_avg |
+----------------------+------+-------------------------------+
| coresched_normalized | ** | 0.30 |
+----------------------+------+-------------------------------+
| default_normalized | ** | 1.00 |
+----------------------+------+-------------------------------+
| smtoff_normalized | ** | 0.87 |
+----------------------+------+-------------------------------+
-- notes on record_item:
* coresched_normalized: smton, cs enabled, test result normalized by
default value
* default_normalized: smton, cs disabled, test result normalized by
default value
* smtoff_normalized: smtoff, test result normalized by default value
- test results of negative case, all as expected, no kernel panic or system
hang observed
Hongyu
