On Wed, Sep 15, 2021 at 5:45 AM <anton.iva...@cambridgegreys.com> wrote:
>
> From: Anton Ivanov <anton.iva...@cambridgegreys.com>
>
> Restore parallel build with dp groups using rwlock instead
> of per row locking as an underlying mechanism.
>
> This provides improvement ~ 10% end-to-end on ovn-heater
> under virutalization despite awakening some qemu gremlin
> which makes qemu climb to silly CPU usage. The gain on
> bare metal is likely to be higher.
>
Hi Anton,
I am trying to see the benefit of parallel_build, but encountered
unexpected performance result when running the perf tests with command:
make check-perf TESTSUITEFLAGS="--rebuild"
It shows significantly worse performance than without parallel_build. For
dp_group = no cases, it is better, but still ~30% slower than without
parallel_build. I have 24 cores, but each thread is not consuming much CPU
except the main thread. I also tried hardcode the number of thread to just
4, which end up with slightly better results, but still far behind "without
parallel_build".
no parallel |
parallel (24 pool threads) | parallel with (4 pool
threads)
|
|
1: ovn-northd basic scale test -- 200 Hypervisors, 200 | 1:
ovn-northd basic scale test -- 200 Hypervisors, 200 | 1: ovn-northd
basic scale test -- 200 Hypervisors, 200
--- | ---
| ---
Maximum (NB in msec): 1058 | Maximum (NB
in msec): 4269 | Maximum (NB in msec): 4097
Average (NB in msec): 836.941167 | Average (NB
in msec): 3697.253931 | Average (NB in msec):
3498.311525
Maximum (SB in msec): 30 | Maximum (SB
in msec): 30 | Maximum (SB in msec): 28
Average (SB in msec): 25.934011 | Average (SB
in msec): 26.001840 | Average (SB in msec):
25.685091
Maximum (northd-loop in msec): 1204 | Maximum
(northd-loop in msec): 4379 | Maximum (northd-loop in
msec): 4251
Average (northd-loop in msec): 1005.330078 | Average
(northd-loop in msec): 4233.871504 | Average (northd-loop in
msec): 4022.774208
|
|
2: ovn-northd basic scale test -- 200 Hypervisors, 200 | 2:
ovn-northd basic scale test -- 200 Hypervisors, 200 | 2: ovn-northd
basic scale test -- 200 Hypervisors, 200
--- | ---
| ---
Maximum (NB in msec): 1124 | Maximum (NB
in msec): 1480 | Maximum (NB in msec): 1331
Average (NB in msec): 892.403405 | Average (NB
in msec): 1206.189287 | Average (NB in msec):
1089.378455
Maximum (SB in msec): 29 | Maximum (SB
in msec): 31 | Maximum (SB in msec): 30
Average (SB in msec): 26.922632 | Average (SB
in msec): 26.636706 | Average (SB in msec):
25.657484
Maximum (northd-loop in msec): 1275 | Maximum
(northd-loop in msec): 1639 | Maximum (northd-loop in
msec): 1495
Average (northd-loop in msec): 1074.917873 | Average
(northd-loop in msec): 1458.152327 | Average (northd-loop in
msec): 1301.057201
|
|
5: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5:
ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5: ovn-northd
basic scale test -- 500 Hypervisors, 50
--- | ---
| ---
Maximum (NB in msec): 768 | Maximum (NB
in msec): 3086 | Maximum (NB in msec): 2876
Average (NB in msec): 614.491938 | Average (NB
in msec): 2681.688365 | Average (NB in msec):
2531.255444
Maximum (SB in msec): 18 | Maximum (SB
in msec): 17 | Maximum (SB in msec): 18
Average (SB in msec): 16.347526 | Average (SB
in msec): 15.955263 | Average (SB in msec):
16.278075
Maximum (northd-loop in msec): 889 | Maximum
(northd-loop in msec): 3247 | Maximum (northd-loop in
msec): 3031
Average (northd-loop in msec): 772.083572 | Average
(northd-loop in msec): 3117.504297 | Average (northd-loop in
msec): 2833.182361
|
|
6: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6:
ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6: ovn-northd
basic scale test -- 500 Hypervisors, 50
--- | ---
| ---
Maximum (NB in msec): 1046 | Maximum (NB
in msec): 1371 | Maximum (NB in msec): 1262
Average (NB in msec): 827.735852 | Average (NB
in msec): 1135.514228 | Average (NB in msec):
970.544792
Maximum (SB in msec): 19 | Maximum (SB
in msec): 18 | Maximum (SB in msec): 19
Average (SB in msec): 16.828127 | Average (SB
in msec): 16.083914 | Average (SB in msec):
15.602525
Maximum (northd-loop in msec): 1163 | Maximum
(northd-loop in msec): 1545 | Maximum (northd-loop in
msec): 1411
Average (northd-loop in msec): 972.567407 | Average
(northd-loop in msec): 1328.617583 | Average (northd-loop in
msec): 1207.667100
I didn't debug yet, but do you have any clue what could be the reason? I am
using the upstream commit 9242f27f63 which already included this patch.
Below is my change to the perf-northd.at file just to enable parallel_build:
diff --git a/tests/perf-northd.at b/tests/perf-northd.at
index 74b69e9d4..9328c2e21 100644
--- a/tests/perf-northd.at
+++ b/tests/perf-northd.at
@@ -191,6 +191,7 @@ AT_SETUP([ovn-northd basic scale test -- 200
Hypervisors, 200 Logical Ports/Hype
PERF_RECORD_START()
ovn_start
+ovn-nbctl set nb_global . options:use_parallel_build=true
BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(200, 200))
@@ -203,9 +204,10 @@ AT_SETUP([ovn-northd basic scale test -- 500
Hypervisors, 50 Logical Ports/Hyper
PERF_RECORD_START()
ovn_start
+ovn-nbctl set nb_global . options:use_parallel_build=true
BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(500, 50))
Thanks,
Han
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
