On Thu, Sep 30, 2021 at 12:08 AM Anton Ivanov < [email protected]> wrote:
> After quickly adding some more prints into the testsuite. > > Test 1: > > Without > > 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical > Ports/Hypervisor -- ovn-northd -- dp-groups=yes > --- > Maximum (NB in msec): 1130 > Average (NB in msec): 620.375000 > Maximum (SB in msec): 23 > Average (SB in msec): 21.468759 > Maximum (northd-loop in msec): 6002 > Minimum (northd-loop in msec): 0 > Average (northd-loop in msec): 914.760417 > Long term average (northd-loop in msec): 104.799340 > > With > > 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical > Ports/Hypervisor -- ovn-northd -- dp-groups=yes > --- > Maximum (NB in msec): 1148 > Average (NB in msec): 630.250000 > Maximum (SB in msec): 24 > Average (SB in msec): 21.468744 > Maximum (northd-loop in msec): 6090 > Minimum (northd-loop in msec): 0 > Average (northd-loop in msec): 762.101565 > Long term average (northd-loop in msec): 80.735192 > > The metric which actually matters and which SHOULD me measured - long term > average is better by 20%. Using short term average instead of long term in > the test suite is actually a BUG. > Good catch! > Are you running yours under some sort of virtualization? > No, I am testing on a bare-metal. > A. > On 30/09/2021 07:52, Han Zhou wrote: > > Thanks Anton for checking. I am using: Intel(R) Core(TM) i9-7920X CPU @ > 2.90GHz, 24 cores. > It is weird why my result is so different. I also verified with a scale > test script that creates a large scale NB/SB with 800 nodes of simulated > k8s setup. And then just run: > ovn-nbctl --print-wait-time --wait=sb sync > > Without parallel: > ovn-northd completion: 7807ms > > With parallel: > ovn-northd completion: 41267ms > > I suspected the hmap size problem but I tried changing the initial size to > 64k buckets and it didn't help. I will find some time to check the "perf" > reports. > > Thanks, > Han > > On Wed, Sep 29, 2021 at 11:31 PM Anton Ivanov < > [email protected]> wrote: > >> On 30/09/2021 07:16, Anton Ivanov wrote: >> >> Results on a Ryzen 5 3600 - 6 cores 12 threads >> >> I will also have a look into the "maximum" measurement for multi-thread. >> >> It does not tie up with the drop in average across the board. >> >> A. >> >> >> Without >> >> >> 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical >> Ports/Hypervisor -- ovn-northd -- dp-groups=yes >> --- >> Maximum (NB in msec): 1256 >> Average (NB in msec): 679.463785 >> Maximum (SB in msec): 25 >> Average (SB in msec): 22.489798 >> Maximum (northd-loop in msec): 1347 >> Average (northd-loop in msec): 799.944878 >> >> 2: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical >> Ports/Hypervisor -- ovn-northd -- dp-groups=no >> --- >> Maximum (NB in msec): 1956 >> Average (NB in msec): 809.387285 >> Maximum (SB in msec): 24 >> Average (SB in msec): 21.649258 >> Maximum (northd-loop in msec): 2011 >> Average (northd-loop in msec): 961.718686 >> >> 5: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical >> Ports/Hypervisor -- ovn-northd -- dp-groups=yes >> --- >> Maximum (NB in msec): 557 >> Average (NB in msec): 474.010337 >> Maximum (SB in msec): 15 >> Average (SB in msec): 13.927192 >> Maximum (northd-loop in msec): 1261 >> Average (northd-loop in msec): 580.999122 >> >> 6: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical >> Ports/Hypervisor -- ovn-northd -- dp-groups=no >> --- >> Maximum (NB in msec): 756 >> Average (NB in msec): 625.614724 >> Maximum (SB in msec): 15 >> Average (SB in msec): 14.181048 >> Maximum (northd-loop in msec): 1649 >> Average (northd-loop in msec): 746.208332 >> >> >> With >> >> 1: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical >> Ports/Hypervisor -- ovn-northd -- dp-groups=yes >> --- >> Maximum (NB in msec): 1140 >> Average (NB in msec): 631.125000 >> Maximum (SB in msec): 24 >> Average (SB in msec): 21.453609 >> Maximum (northd-loop in msec): 6080 >> Average (northd-loop in msec): 759.718815 >> >> 2: ovn-northd basic scale test -- 200 Hypervisors, 200 Logical >> Ports/Hypervisor -- ovn-northd -- dp-groups=no >> --- >> Maximum (NB in msec): 1210 >> Average (NB in msec): 673.000000 >> Maximum (SB in msec): 27 >> Average (SB in msec): 22.453125 >> Maximum (northd-loop in msec): 6514 >> Average (northd-loop in msec): 808.596842 >> >> 5: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical >> Ports/Hypervisor -- ovn-northd -- dp-groups=yes >> --- >> Maximum (NB in msec): 798 >> Average (NB in msec): 429.750000 >> Maximum (SB in msec): 15 >> Average (SB in msec): 12.998533 >> Maximum (northd-loop in msec): 3835 >> Average (northd-loop in msec): 564.875986 >> >> 6: ovn-northd basic scale test -- 500 Hypervisors, 50 Logical >> Ports/Hypervisor -- ovn-northd -- dp-groups=no >> --- >> Maximum (NB in msec): 1074 >> Average (NB in msec): 593.875000 >> Maximum (SB in msec): 14 >> Average (SB in msec): 13.655273 >> Maximum (northd-loop in msec): 4973 >> Average (northd-loop in msec): 771.102605 >> >> The only one slower is test 6 which I will look into. >> >> The rest are > 5% faster. >> >> A. >> >> On 30/09/2021 00:56, Han Zhou wrote: >> >> >> >> On Wed, Sep 15, 2021 at 5:45 AM <[email protected]> wrote: >> > >> > From: Anton Ivanov <[email protected]> >> > >> > Restore parallel build with dp groups using rwlock instead >> > of per row locking as an underlying mechanism. >> > >> > This provides improvement ~ 10% end-to-end on ovn-heater >> > under virutalization despite awakening some qemu gremlin >> > which makes qemu climb to silly CPU usage. The gain on >> > bare metal is likely to be higher. >> > >> Hi Anton, >> >> I am trying to see the benefit of parallel_build, but encountered >> unexpected performance result when running the perf tests with command: >> make check-perf TESTSUITEFLAGS="--rebuild" >> >> It shows significantly worse performance than without parallel_build. For >> dp_group = no cases, it is better, but still ~30% slower than without >> parallel_build. I have 24 cores, but each thread is not consuming much CPU >> except the main thread. I also tried hardcode the number of thread to just >> 4, which end up with slightly better results, but still far behind "without >> parallel_build". >> >> no parallel | >> parallel (24 pool threads) | parallel with (4 pool >> threads) >> | >> | >> 1: ovn-northd basic scale test -- 200 Hypervisors, 200 | 1: >> ovn-northd basic scale test -- 200 Hypervisors, 200 | 1: ovn-northd >> basic scale test -- 200 Hypervisors, 200 >> --- | --- >> | --- >> Maximum (NB in msec): 1058 | Maximum >> (NB in msec): 4269 | Maximum (NB in msec): >> 4097 >> Average (NB in msec): 836.941167 | Average >> (NB in msec): 3697.253931 | Average (NB in msec): >> 3498.311525 >> Maximum (SB in msec): 30 | Maximum >> (SB in msec): 30 | Maximum (SB in msec): >> 28 >> Average (SB in msec): 25.934011 | Average >> (SB in msec): 26.001840 | Average (SB in msec): >> 25.685091 >> Maximum (northd-loop in msec): 1204 | Maximum >> (northd-loop in msec): 4379 | Maximum (northd-loop in >> msec): 4251 >> Average (northd-loop in msec): 1005.330078 | Average >> (northd-loop in msec): 4233.871504 | Average (northd-loop in >> msec): 4022.774208 >> | >> | >> 2: ovn-northd basic scale test -- 200 Hypervisors, 200 | 2: >> ovn-northd basic scale test -- 200 Hypervisors, 200 | 2: ovn-northd >> basic scale test -- 200 Hypervisors, 200 >> --- | --- >> | --- >> Maximum (NB in msec): 1124 | Maximum >> (NB in msec): 1480 | Maximum (NB in msec): >> 1331 >> Average (NB in msec): 892.403405 | Average >> (NB in msec): 1206.189287 | Average (NB in msec): >> 1089.378455 >> Maximum (SB in msec): 29 | Maximum >> (SB in msec): 31 | Maximum (SB in msec): >> 30 >> Average (SB in msec): 26.922632 | Average >> (SB in msec): 26.636706 | Average (SB in msec): >> 25.657484 >> Maximum (northd-loop in msec): 1275 | Maximum >> (northd-loop in msec): 1639 | Maximum (northd-loop in >> msec): 1495 >> Average (northd-loop in msec): 1074.917873 | Average >> (northd-loop in msec): 1458.152327 | Average (northd-loop in >> msec): 1301.057201 >> | >> | >> 5: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5: >> ovn-northd basic scale test -- 500 Hypervisors, 50 L| 5: ovn-northd >> basic scale test -- 500 Hypervisors, 50 >> --- | --- >> | --- >> Maximum (NB in msec): 768 | Maximum >> (NB in msec): 3086 | Maximum (NB in msec): >> 2876 >> Average (NB in msec): 614.491938 | Average >> (NB in msec): 2681.688365 | Average (NB in msec): >> 2531.255444 >> Maximum (SB in msec): 18 | Maximum >> (SB in msec): 17 | Maximum (SB in msec): >> 18 >> Average (SB in msec): 16.347526 | Average >> (SB in msec): 15.955263 | Average (SB in msec): >> 16.278075 >> Maximum (northd-loop in msec): 889 | Maximum >> (northd-loop in msec): 3247 | Maximum (northd-loop in >> msec): 3031 >> Average (northd-loop in msec): 772.083572 | Average >> (northd-loop in msec): 3117.504297 | Average (northd-loop in >> msec): 2833.182361 >> | >> | >> 6: ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6: >> ovn-northd basic scale test -- 500 Hypervisors, 50 L| 6: ovn-northd >> basic scale test -- 500 Hypervisors, 50 >> --- | --- >> | --- >> Maximum (NB in msec): 1046 | Maximum >> (NB in msec): 1371 | Maximum (NB in msec): >> 1262 >> Average (NB in msec): 827.735852 | Average >> (NB in msec): 1135.514228 | Average (NB in msec): >> 970.544792 >> Maximum (SB in msec): 19 | Maximum >> (SB in msec): 18 | Maximum (SB in msec): >> 19 >> Average (SB in msec): 16.828127 | Average >> (SB in msec): 16.083914 | Average (SB in msec): >> 15.602525 >> Maximum (northd-loop in msec): 1163 | Maximum >> (northd-loop in msec): 1545 | Maximum (northd-loop in >> msec): 1411 >> Average (northd-loop in msec): 972.567407 | Average >> (northd-loop in msec): 1328.617583 | Average (northd-loop in >> msec): 1207.667100 >> >> I didn't debug yet, but do you have any clue what could be the reason? I >> am using the upstream commit 9242f27f63 which already included this patch. >> Below is my change to the perf-northd.at file just to enable >> parallel_build: >> >> diff --git a/tests/perf-northd.at b/tests/perf-northd.at >> index 74b69e9d4..9328c2e21 100644 >> --- a/tests/perf-northd.at >> +++ b/tests/perf-northd.at >> @@ -191,6 +191,7 @@ AT_SETUP([ovn-northd basic scale test -- 200 >> Hypervisors, 200 Logical Ports/Hype >> PERF_RECORD_START() >> >> ovn_start >> +ovn-nbctl set nb_global . options:use_parallel_build=true >> >> BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(200, 200)) >> >> @@ -203,9 +204,10 @@ AT_SETUP([ovn-northd basic scale test -- 500 >> Hypervisors, 50 Logical Ports/Hyper >> PERF_RECORD_START() >> >> ovn_start >> +ovn-nbctl set nb_global . options:use_parallel_build=true >> >> BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(500, 50)) >> >> Thanks, >> Han >> >> >> -- >> Anton R. Ivanov >> Cambridgegreys Limited. Registered in England. Company Number >> 10273661https://www.cambridgegreys.com/ >> >> >> -- >> Anton R. Ivanov >> Cambridgegreys Limited. Registered in England. Company Number >> 10273661https://www.cambridgegreys.com/ >> >> -- > Anton R. Ivanov > Cambridgegreys Limited. Registered in England. Company Number > 10273661https://www.cambridgegreys.com/ > > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
