Thanks Anton for checking. I am using: Intel(R) Core(TM) i9-7920X CPU @
2.90GHz, 24 cores.
It is weird why my result is so different. I also verified with a scale
test script that creates a large scale NB/SB with 800 nodes of simulated
k8s setup. And then just run:
ovn-nbctl --print-wait-time --wait=sb sync
Without parallel:
ovn-northd completion: 7807ms
With parallel:
ovn-northd completion: 41267ms
I suspected the hmap size problem but I tried changing the initial size to
64k buckets and it didn't help. I will find some time to check the "perf"
reports.
Thanks,
Han
On Wed, Sep 29, 2021 at 11:31 PM Anton Ivanov <
[email protected]> wrote:
> On 30/09/2021 07:16, Anton Ivanov wrote:
>
> Results on a Ryzen 5 3600 - 6 cores 12 threads
>
> I will also have a look into the "maximum" measurement for multi-thread.
>
> It does not tie up with the drop in average across the board.
>
> A.
>
>
> Without
>
>
> 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical
> Ports/Hypervisor -- ovn-northd -- dp-groups=yes
> ---
> Maximum (NB in msec): 1256
> Average (NB in msec): 679.463785
> Maximum (SB in msec): 25
> Average (SB in msec): 22.489798
> Maximum (northd-loop in msec): 1347
> Average (northd-loop in msec): 799.944878
>
> 2: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical
> Ports/Hypervisor -- ovn-northd -- dp-groups=no
> ---
> Maximum (NB in msec): 1956
> Average (NB in msec): 809.387285
> Maximum (SB in msec): 24
> Average (SB in msec): 21.649258
> Maximum (northd-loop in msec): 2011
> Average (northd-loop in msec): 961.718686
>
> 5: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical
> Ports/Hypervisor -- ovn-northd -- dp-groups=yes
> ---
> Maximum (NB in msec): 557
> Average (NB in msec): 474.010337
> Maximum (SB in msec): 15
> Average (SB in msec): 13.927192
> Maximum (northd-loop in msec): 1261
> Average (northd-loop in msec): 580.999122
>
> 6: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical
> Ports/Hypervisor -- ovn-northd -- dp-groups=no
> ---
> Maximum (NB in msec): 756
> Average (NB in msec): 625.614724
> Maximum (SB in msec): 15
> Average (SB in msec): 14.181048
> Maximum (northd-loop in msec): 1649
> Average (northd-loop in msec): 746.208332
>
>
> With
>
> 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical
> Ports/Hypervisor -- ovn-northd -- dp-groups=yes
> ---
> Maximum (NB in msec): 1140
> Average (NB in msec): 631.125000
> Maximum (SB in msec): 24
> Average (SB in msec): 21.453609
> Maximum (northd-loop in msec): 6080
> Average (northd-loop in msec): 759.718815
>
> 2: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical
> Ports/Hypervisor -- ovn-northd -- dp-groups=no
> ---
> Maximum (NB in msec): 1210
> Average (NB in msec): 673.000000
> Maximum (SB in msec): 27
> Average (SB in msec): 22.453125
> Maximum (northd-loop in msec): 6514
> Average (northd-loop in msec): 808.596842
>
> 5: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical
> Ports/Hypervisor -- ovn-northd -- dp-groups=yes
> ---
> Maximum (NB in msec): 798
> Average (NB in msec): 429.750000
> Maximum (SB in msec): 15
> Average (SB in msec): 12.998533
> Maximum (northd-loop in msec): 3835
> Average (northd-loop in msec): 564.875986
>
> 6: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical
> Ports/Hypervisor -- ovn-northd -- dp-groups=no
> ---
> Maximum (NB in msec): 1074
> Average (NB in msec): 593.875000
> Maximum (SB in msec): 14
> Average (SB in msec): 13.655273
> Maximum (northd-loop in msec): 4973
> Average (northd-loop in msec): 771.102605
>
> The only one slower is test 6 which I will look into.
>
> The rest are > 5% faster.
>
> A.
>
> On 30/09/2021 00:56, Han Zhou wrote:
>
>
>
> On Wed, Sep 15, 2021 at 5:45 AM <[email protected]> wrote:
> >
> > From: Anton Ivanov <[email protected]>
> >
> > Restore parallel build with dp groups using rwlock instead
> > of per row locking as an underlying mechanism.
> >
> > This provides improvement ~ 10% end-to-end on ovn-heater
> > under virutalization despite awakening some qemu gremlin
> > which makes qemu climb to silly CPU usage. The gain on
> > bare metal is likely to be higher.
> >
> Hi Anton,
>
> I am trying to see the benefit of parallel_build, but encountered
> unexpected performance result when running the perf tests with command:
> make check-perf TESTSUITEFLAGS="--rebuild"
>
> It shows significantly worse performance than without parallel_build. For
> dp_group = no cases, it is better, but still ~30% slower than without
> parallel_build. I have 24 cores, but each thread is not consuming much CPU
> except the main thread. I also tried hardcode the number of thread to just
> 4, which end up with slightly better results, but still far behind "without
> parallel_build".
>
> no parallel |
> parallel (24 pool threads) | parallel with (4 pool
> threads)
> |
> |
> 1: ovn-northd basic scale test -- 200 Hypervisors, 200 | 1:
> ovn-northd basic scale test -- 200 Hypervisors, 200 | 1: ovn-northd
> basic scale test -- 200 Hypervisors, 200
> --- | ---
> | ---
> Maximum (NB in msec): 1058 | Maximum
> (NB in msec): 4269 | Maximum (NB in msec):
> 4097
> Average (NB in msec): 836.941167 | Average
> (NB in msec): 3697.253931 | Average (NB in msec):
> 3498.311525
> Maximum (SB in msec): 30 | Maximum
> (SB in msec): 30 | Maximum (SB in msec):
> 28
> Average (SB in msec): 25.934011 | Average
> (SB in msec): 26.001840 | Average (SB in msec):
> 25.685091
> Maximum (northd-loop in msec): 1204 | Maximum
> (northd-loop in msec): 4379 | Maximum (northd-loop in
> msec): 4251
> Average (northd-loop in msec): 1005.330078 | Average
> (northd-loop in msec): 4233.871504 | Average (northd-loop in
> msec): 4022.774208
> |
> |
> 2: ovn-northd basic scale test -- 200 Hypervisors, 200 | 2:
> ovn-northd basic scale test -- 200 Hypervisors, 200 | 2: ovn-northd
> basic scale test -- 200 Hypervisors, 200
> --- | ---
> | ---
> Maximum (NB in msec): 1124 | Maximum
> (NB in msec): 1480 | Maximum (NB in msec):
> 1331
> Average (NB in msec): 892.403405 | Average
> (NB in msec): 1206.189287 | Average (NB in msec):
> 1089.378455
> Maximum (SB in msec): 29 | Maximum
> (SB in msec): 31 | Maximum (SB in msec):
> 30
> Average (SB in msec): 26.922632 | Average
> (SB in msec): 26.636706 | Average (SB in msec):
> 25.657484
> Maximum (northd-loop in msec): 1275 | Maximum
> (northd-loop in msec): 1639 | Maximum (northd-loop in
> msec): 1495
> Average (northd-loop in msec): 1074.917873 | Average
> (northd-loop in msec): 1458.152327 | Average (northd-loop in
> msec): 1301.057201
> |
> |
> 5: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5:
> ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5: ovn-northd
> basic scale test -- 500 Hypervisors, 50
> --- | ---
> | ---
> Maximum (NB in msec): 768 | Maximum
> (NB in msec): 3086 | Maximum (NB in msec):
> 2876
> Average (NB in msec): 614.491938 | Average
> (NB in msec): 2681.688365 | Average (NB in msec):
> 2531.255444
> Maximum (SB in msec): 18 | Maximum
> (SB in msec): 17 | Maximum (SB in msec):
> 18
> Average (SB in msec): 16.347526 | Average
> (SB in msec): 15.955263 | Average (SB in msec):
> 16.278075
> Maximum (northd-loop in msec): 889 | Maximum
> (northd-loop in msec): 3247 | Maximum (northd-loop in
> msec): 3031
> Average (northd-loop in msec): 772.083572 | Average
> (northd-loop in msec): 3117.504297 | Average (northd-loop in
> msec): 2833.182361
> |
> |
> 6: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6:
> ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6: ovn-northd
> basic scale test -- 500 Hypervisors, 50
> --- | ---
> | ---
> Maximum (NB in msec): 1046 | Maximum
> (NB in msec): 1371 | Maximum (NB in msec):
> 1262
> Average (NB in msec): 827.735852 | Average
> (NB in msec): 1135.514228 | Average (NB in msec):
> 970.544792
> Maximum (SB in msec): 19 | Maximum
> (SB in msec): 18 | Maximum (SB in msec):
> 19
> Average (SB in msec): 16.828127 | Average
> (SB in msec): 16.083914 | Average (SB in msec):
> 15.602525
> Maximum (northd-loop in msec): 1163 | Maximum
> (northd-loop in msec): 1545 | Maximum (northd-loop in
> msec): 1411
> Average (northd-loop in msec): 972.567407 | Average
> (northd-loop in msec): 1328.617583 | Average (northd-loop in
> msec): 1207.667100
>
> I didn't debug yet, but do you have any clue what could be the reason? I
> am using the upstream commit 9242f27f63 which already included this patch.
> Below is my change to the perf-northd.at file just to enable
> parallel_build:
>
> diff --git a/tests/perf-northd.at b/tests/perf-northd.at
> index 74b69e9d4..9328c2e21 100644
> --- a/tests/perf-northd.at
> +++ b/tests/perf-northd.at
> @@ -191,6 +191,7 @@ AT_SETUP([ovn-northd basic scale test -- 200
> Hypervisors, 200 Logical Ports/Hype
> PERF_RECORD_START()
>
> ovn_start
> +ovn-nbctl set nb_global . options:use_parallel_build=true
>
> BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(200, 200))
>
> @@ -203,9 +204,10 @@ AT_SETUP([ovn-northd basic scale test -- 500
> Hypervisors, 50 Logical Ports/Hyper
> PERF_RECORD_START()
>
> ovn_start
> +ovn-nbctl set nb_global . options:use_parallel_build=true
>
> BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(500, 50))
>
> Thanks,
> Han
>
>
> --
> Anton R. Ivanov
> Cambridgegreys Limited. Registered in England. Company Number
> 10273661https://www.cambridgegreys.com/
>
>
> --
> Anton R. Ivanov
> Cambridgegreys Limited. Registered in England. Company Number
> 10273661https://www.cambridgegreys.com/
>
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
