I need to have a look.
I use the ovn-heater end-to-end test, that was showing a substantial
improvement.
What are you running this on?
A.
On 30/09/2021 00:56, Han Zhou wrote:
On Wed, Sep 15, 2021 at 5:45 AM <[email protected]
<mailto:[email protected]>> wrote:
>
> From: Anton Ivanov <[email protected]
<mailto:[email protected]>>
>
> Restore parallel build with dp groups using rwlock instead
> of per row locking as an underlying mechanism.
>
> This provides improvement ~ 10% end-to-end on ovn-heater
> under virutalization despite awakening some qemu gremlin
> which makes qemu climb to silly CPU usage. The gain on
> bare metal is likely to be higher.
>
Hi Anton,
I am trying to see the benefit of parallel_build, but encountered
unexpected performance result when running the perf tests with command:
make check-perf TESTSUITEFLAGS="--rebuild"
It shows significantly worse performance than without parallel_build.
For dp_group = no cases, it is better, but still ~30% slower than
without parallel_build. I have 24 cores, but each thread is not
consuming much CPU except the main thread. I also tried hardcode the
number of thread to just 4, which end up with slightly better results,
but still far behind "without parallel_build".
no parallel | parallel
(24 pool threads) | parallel with (4 pool threads)
| |
1: ovn-northd basic scale test -- 200 Hypervisors, 200 | 1:
ovn-northd basic scale test -- 200 Hypervisors, 200 | 1: ovn-northd
basic scale test -- 200 Hypervisors, 200
--- | ---
| ---
Maximum (NB in msec): 1058 |
Maximum (NB in msec): 4269 | Maximum
(NB in msec): 4097
Average (NB in msec): 836.941167 |
Average (NB in msec): 3697.253931 | Average
(NB in msec): 3498.311525
Maximum (SB in msec): 30 |
Maximum (SB in msec): 30 | Maximum
(SB in msec): 28
Average (SB in msec): 25.934011 |
Average (SB in msec): 26.001840 | Average
(SB in msec): 25.685091
Maximum (northd-loop in msec): 1204 |
Maximum (northd-loop in msec): 4379 | Maximum
(northd-loop in msec): 4251
Average (northd-loop in msec): 1005.330078 |
Average (northd-loop in msec): 4233.871504 | Average
(northd-loop in msec): 4022.774208
|
|
2: ovn-northd basic scale test -- 200 Hypervisors, 200 | 2:
ovn-northd basic scale test -- 200 Hypervisors, 200 | 2: ovn-northd
basic scale test -- 200 Hypervisors, 200
--- | ---
| ---
Maximum (NB in msec): 1124 |
Maximum (NB in msec): 1480 | Maximum
(NB in msec): 1331
Average (NB in msec): 892.403405 |
Average (NB in msec): 1206.189287 | Average
(NB in msec): 1089.378455
Maximum (SB in msec): 29 |
Maximum (SB in msec): 31 | Maximum
(SB in msec): 30
Average (SB in msec): 26.922632 |
Average (SB in msec): 26.636706 | Average
(SB in msec): 25.657484
Maximum (northd-loop in msec): 1275 |
Maximum (northd-loop in msec): 1639 | Maximum
(northd-loop in msec): 1495
Average (northd-loop in msec): 1074.917873 |
Average (northd-loop in msec): 1458.152327 | Average
(northd-loop in msec): 1301.057201
|
|
5: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5:
ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5: ovn-northd
basic scale test -- 500 Hypervisors, 50
--- | ---
| ---
Maximum (NB in msec): 768 |
Maximum (NB in msec): 3086 | Maximum
(NB in msec): 2876
Average (NB in msec): 614.491938 |
Average (NB in msec): 2681.688365 | Average
(NB in msec): 2531.255444
Maximum (SB in msec): 18 |
Maximum (SB in msec): 17 | Maximum
(SB in msec): 18
Average (SB in msec): 16.347526 |
Average (SB in msec): 15.955263 | Average
(SB in msec): 16.278075
Maximum (northd-loop in msec): 889 |
Maximum (northd-loop in msec): 3247 | Maximum
(northd-loop in msec): 3031
Average (northd-loop in msec): 772.083572 |
Average (northd-loop in msec): 3117.504297 | Average
(northd-loop in msec): 2833.182361
|
|
6: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6:
ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6: ovn-northd
basic scale test -- 500 Hypervisors, 50
--- | ---
| ---
Maximum (NB in msec): 1046 |
Maximum (NB in msec): 1371 | Maximum
(NB in msec): 1262
Average (NB in msec): 827.735852 |
Average (NB in msec): 1135.514228 | Average
(NB in msec): 970.544792
Maximum (SB in msec): 19 |
Maximum (SB in msec): 18 | Maximum
(SB in msec): 19
Average (SB in msec): 16.828127 |
Average (SB in msec): 16.083914 | Average
(SB in msec): 15.602525
Maximum (northd-loop in msec): 1163 |
Maximum (northd-loop in msec): 1545 | Maximum
(northd-loop in msec): 1411
Average (northd-loop in msec): 972.567407 |
Average (northd-loop in msec): 1328.617583 | Average
(northd-loop in msec): 1207.667100
I didn't debug yet, but do you have any clue what could be the reason?
I am using the upstream commit 9242f27f63 which already included this
patch.
Below is my change to the perf-northd.at <http://perf-northd.at> file
just to enable parallel_build:
diff --git a/tests/perf-northd.at <http://perf-northd.at>
b/tests/perf-northd.at <http://perf-northd.at>
index 74b69e9d4..9328c2e21 100644
--- a/tests/perf-northd.at <http://perf-northd.at>
+++ b/tests/perf-northd.at <http://perf-northd.at>
@@ -191,6 +191,7 @@ AT_SETUP([ovn-northd basic scale test -- 200
Hypervisors, 200 Logical Ports/Hype
PERF_RECORD_START()
ovn_start
+ovn-nbctl set nb_global . options:use_parallel_build=true
BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(200, 200))
@@ -203,9 +204,10 @@ AT_SETUP([ovn-northd basic scale test -- 500
Hypervisors, 50 Logical Ports/Hyper
PERF_RECORD_START()
ovn_start
+ovn-nbctl set nb_global . options:use_parallel_build=true
BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(500, 50))
Thanks,
Han
--
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
