> On 14 Nov 2019, at 22:33, Ludwig Krispenz <[email protected]> wrote: > > > On 11/14/2019 12:17 PM, William Brown wrote: >> >>> On 14 Nov 2019, at 19:06, Ludwig Krispenz <[email protected]> wrote: >>> >>> >>> On 11/14/2019 09:29 AM, William Brown wrote: >>>>> On 14 Nov 2019, at 18:22, Ludwig Krispenz <[email protected]> wrote: >>>>> >>>>> Hi William, >>>>> >>>>> before further thinking about this, I need some clarification, or maybe I >>>>> just missed this. When you talk about 1..16 threads do you mean worker >>>>> threads ? >>>> Server worker threads. ldclt is set to only use 10 client threads - which >>>> is surprising that with 10 client threads we see a decline when workers > >>>> 10 (one would assume it should stabilise). >>>> >>>>> Or concurrent client connection threads in ldclt/rsearch/.... - how many >>>>> concurrent connections do you have and how does varying this number >>>>> change results ? >>>> I will add more tests to this to allow varying the ldclt numbers. >>> ok, and I assume that you are using a version with nunc-stans removed, >>> could you please also verify the effect of tubo-mode on/off ? >> Correct, I'm using git master. Yes I'll check that also. I plan to add >> permutations like this to the test harness so it's easier for us to repeat >> in the future when we make changes. >> >> I also need to find a way to wire in perf/stap so we can generate >> flamegraphs from each test run too for later analysis. >> >> Thanks for the great ideas :) > Thanks, and one more idea ;-) > Can you separate the client and the server on two different machines, I've > seen ldclt or other clients impacting cpu usage a lot, there will be some > network overhead, but this should be ok (and more realistic)
That was the original goal, but I can't seperate it (yet) because we restart to change settings ... I'm not sure what's the best way to do it - have the tests maybe act as a generator and then you have to run the ldclt from a seperate machine? Not sure really .... I need to think what it should look like. I know viktor did some work on pytest over multiple hosts so perhaps that could help here too to coordinate? I think they also were speaking about ansible as well ... maybe he should comment if he has ideas. >> >>>>> Regards, >>>>> Ludwig >>>>> >>>>> On 11/14/2019 03:34 AM, William Brown wrote: >>>>>> Hi all, >>>>>> >>>>>> After our catch up, we were discussing performance matters. I decided to >>>>>> start on this while waiting for some of my tickets to be reviewed and to >>>>>> see what's going on. >>>>>> >>>>>> These tests were carried out on a virtual machine configured in search 6 >>>>>> to have access to 6 CPU's, and search 12 with 12 CPU. Both machines had >>>>>> access to 8GB of ram. >>>>>> >>>>>> The hardware is an i7 2.2GHz with 6 cores (12 threads) and 32GB of ram, >>>>>> with NVME storage provided. >>>>>> >>>>>> The rows are the VM CPU's available, and the columns are the number of >>>>>> threads in nsslapd-threadnumber. No other variables were changed. The >>>>>> database has 6000 users and 4000 groups. The instance was restarted >>>>>> before each test. The search was a randomised uid equality test with a >>>>>> single result. I provided the thread 6 and 12 columns to try to match >>>>>> the VM and host specs rather than just the traditional base 2 sequence >>>>>> we see. >>>>>> >>>>>> I've attached a screen shot of the results, but I have some initial >>>>>> thoughts to provide on this. What's interesting is our initial 1 thread >>>>>> performance and how steeply it ramps up towards 4 thread. This in mind >>>>>> it's not a linear increase. Per thread on s6 we go from ~3800 to ~2500 >>>>>> ops per second, and a similar ratio exists in s12. What is stark is that >>>>>> after t4 we immediately see a per thread *decline* despite the greater >>>>>> amount of available computer resources. This indicates that it is poor >>>>>> locking and thread coordination causing a rapid decline in performance. >>>>>> This was true on both s6 and s12. The decline intesifies rapidly once we >>>>>> exceed the CPU avail on the host (s6 between t6 to t12), but still >>>>>> declines even when we do have the hardware threads available in s12. >>>>>> >>>>>> I will perform some testing between t1 and t6 versions to see if I can >>>>>> isolate which functions are having a growth in time consumption. >>>>>> >>>>>> For now an early recommendation is that we alter our default CPU >>>>>> auto-tuning. Currently we use a curve which starts at 16 threads from 1 >>>>>> to 4 cores, and then tapering down to 512 cores to 512 threads - however >>>>>> in almost all of these autotuned threads we have threads greater than >>>>>> our core count. This from this graph would indicate that this decision >>>>>> only hurts our performance rather than improving it. I suggest we change >>>>>> our thread autotuning to be 1 to 1 ratio of threads to cores to prevent >>>>>> over contention on lock resources. >>>>>> >>>>>> Thanks, more to come once I setup this profiling on a real machine so I >>>>>> can generate flamegraphs. >>>>>> >>>>>> <Mail Attachment.png> >>>>>> >>>>>> — >>>>>> Sincerely, >>>>>> >>>>>> William Brown >>>>>> >>>>>> Senior Software Engineer, 389 Directory Server >>>>>> SUSE Labs >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> 389-devel mailing list -- >>>>>> [email protected] >>>>>> >>>>>> To unsubscribe send an email to >>>>>> [email protected] >>>>>> >>>>>> Fedora Code of Conduct: >>>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>>> >>>>>> List Guidelines: >>>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>>> >>>>>> List Archives: >>>>>> https://lists.fedoraproject.org/archives/list/[email protected] >>>>> -- >>>>> Red Hat GmbH, >>>>> http://www.de.redhat.com/ >>>>> , Registered seat: Grasbrunn, >>>>> Commercial register: Amtsgericht Muenchen, HRB 153243, >>>>> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, >>>>> Eric Shander >>>>> >>>>> _______________________________________________ >>>>> 389-devel mailing list -- [email protected] >>>>> To unsubscribe send an email to [email protected] >>>>> Fedora Code of Conduct: >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>> List Archives: >>>>> https://lists.fedoraproject.org/archives/list/[email protected] >>>> — >>>> Sincerely, >>>> >>>> William Brown >>>> >>>> Senior Software Engineer, 389 Directory Server >>>> SUSE Labs >>>> _______________________________________________ >>>> 389-devel mailing list -- [email protected] >>>> To unsubscribe send an email to [email protected] >>>> Fedora Code of Conduct: >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: >>>> https://lists.fedoraproject.org/archives/list/[email protected] >>> -- >>> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, >>> Commercial register: Amtsgericht Muenchen, HRB 153243, >>> Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, >>> Eric Shander >>> _______________________________________________ >>> 389-devel mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/[email protected] >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server >> SUSE Labs >> _______________________________________________ >> 389-devel mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/[email protected] > > -- > Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, > Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, > Eric Shander > _______________________________________________ > 389-devel mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
