It could be interesting to support snapshots of attribute trees here. We could do something with a concurrent b+tree within the entry where we have:
attribute tree: pinned-initial-state: write-state: That way we only copy-on-write what we need into the write state, and we have an easy revert path (remove the write state), less mallocs etc. This would save a fair bit of copying I think, and would allow us to have those changes in the entry as we go. It'd especially pay off on large attributes/entries with huge amounts of attributes or many values in a single attribute. But I think yeah, some more thought needed here .... but good research :) > On 8 May 2020, at 05:31, Mark Reynolds <[email protected]> wrote: > > As we've been discussing performance recently, I did a little checking on how > many times we duplicate a slapi_entry when doing a single modification. I > updated an entry like this (which is actually pretty small): > > dn: uid=mareynol,ou=people,dc=example,dc=com > uid: mareynol > objectClass: top > objectClass: person > objectClass: organizationalperson > objectClass: inetOrgPerson > objectClass: ntUser > ntUserDeleteAccount: true > sn: Reynolds > givenName: Mark > initials: T > cn: Mark T. Reynolds > ntUserCodePage: 0 > ntUserAcctExpires: 9223372036854775807 > ntUserDomainId: Mareynol > ntUniqueId: 76ae38ecf89b884cbfe2b713082f16a3 > > > So every time the server duplicates this particular entry it triggers 159 > mallocs and 64 frees. The more attributes the entry has the more more > mallocs would be required to copy it. I found that when I just change a > single attribute in this entry that the server calls slapi_entry_dup() 8 > times! (that's over 1200 mallocs and 500 frees) The entry is duplicated two > times by the MEP plugin for just being enabled out of the box. We also call > backentry_dup() two times which is even more expensive, but I don't think we > can avoid that one as it's needed for backend plugin transactions. Below are > the stack traces from each "duplication" in order. Surely I would hope we > can reduce this number of entry dups for a single modify operation ... > > > #0 slapi_entry_dup (e=0x7fc881864240) at > ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8b062528b in internal_plugin_search_entry_callback > (e=0x7fc881864240, callback_data=0x7fc8855fcc20) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:531 > #2 0x00007fc8b0624549 in internal_srch_entry_callback (be=0x7fc8aee457e0, > conn=0x0, op=0x7fc88263b400, e=0x7fc881864240) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:99 > #3 0x00007fc8b063a053 in send_ldap_search_entry_ext (pb=0x7fc8825fe120, > e=0x7fc881864240, ectrls=0x0, attrs=0x0, attrsonly=0, > send_result=0, nentries=0, urls=0x0) at > ../389-ds-base/ldap/servers/slapd/result.c:1507 > #4 0x00007fc8b0639194 in send_ldap_search_entry (pb=0x7fc8825fe120, > e=0x7fc881864240, ectrls=0x0, attrs=0x0, attrsonly=0) > at ../389-ds-base/ldap/servers/slapd/result.c:1050 > #5 0x00007fc8b060ded1 in send_entry (pb=0x7fc8825fe120, e=0x7fc881864240, > operation=0x7fc88263b400, attrs=0x0, attrsonly=0, > pnentries=0x7fc8855f855c) at > ../389-ds-base/ldap/servers/slapd/opshared.c:1132 > #6 0x00007fc8b060e497 in iterate (pb=0x7fc8825fe120, be=0x7fc8aee457e0, > send_result=1, pnentries=0x7fc8855f855c, pagesize=-1, > pr_statp=0x7fc8855f854c) at > ../389-ds-base/ldap/servers/slapd/opshared.c:1463 > #7 0x00007fc8b060e8a5 in send_results_ext (pb=0x7fc8825fe120, send_result=1, > nentries=0x7fc8855f855c, pagesize=-1, > pr_stat=0x7fc8855f854c) at > ../389-ds-base/ldap/servers/slapd/opshared.c:1680 > #8 0x00007fc8b060d444 in op_shared_search (pb=0x7fc8825fe120, send_result=1) > at ../389-ds-base/ldap/servers/slapd/opshared.c:875 > #9 0x00007fc8b06258c8 in search_internal_callback_pb (pb=0x7fc8825fe120, > callback_data=0x7fc8855fcc20, > prc=0x7fc8b062534c <internal_plugin_result_callback>, psec=0x7fc8b0625257 > <internal_plugin_search_entry_callback>, > prec=0x7fc8b06252e2 <internal_plugin_search_referral_callback>) at > ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:730 > #10 0x00007fc8b062542e in search_internal_pb (pb=0x7fc8825fe120) at > ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:594 > #11 0x00007fc8b06251ca in slapi_search_internal_pb (pb=0x7fc8825fe120) at > ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:504 > #12 0x00007fc8b0625cb8 in slapi_search_internal_get_entry (dn=0x7fc8855fccf0, > attrs=0x0, ret_entry=0x7fc8855fccd8, > component_identity=0x7fc8ac713320) at > ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:864 > #13 0x00007fc8b0632eea in get_entry (pb=0x7fc8825fe000, dn=0x7fc8abe82fa0 > "uid=mareynol,ou=people,dc=example,dc=com") > at ../389-ds-base/ldap/servers/slapd/pw_retry.c:210 > #14 0x00007fc8b060405f in op_shared_modify (pb=0x7fc8825fe000, pw_change=0, > old_pw=0x0) > at ../389-ds-base/ldap/servers/slapd/modify.c:726 > #15 0x00007fc8b06033bb in do_modify (pb=0x7fc8825fe000) at > ../389-ds-base/ldap/servers/slapd/modify.c:380 > #16 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acf6c0, > op=0x7fc8ac85a000, pb=0x7fc8825fe000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #17 0x000000000041ad49 in connection_threadmain () at > ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #18 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #19 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #20 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc880c68000) at > ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8b0625d24 in slapi_search_internal_get_entry (dn=0x7fc8825facf0, > attrs=0x0, ret_entry=0x7fc8825facd8, component_identity=0x7fc8ac713320) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:873 > #2 0x00007fc8b0632eea in get_entry (pb=0x7fc880bff000, dn=0x7fc880c2c320 > "uid=mareynol,ou=people,dc=example,dc=com") > at ../389-ds-base/ldap/servers/slapd/pw_retry.c:210 > #3 0x00007fc8b060405f in op_shared_modify (pb=0x7fc880bff000, pw_change=0, > old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:726 > #4 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/modify.c:380 > #5 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, > op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #6 0x000000000041ad49 in connection_threadmain () at > ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #7 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #8 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #9 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc882667180) at > ../389-ds-base/ldap/servers/slapd/entry.c:2057 > #1 0x00007fc8ac53b501 in ldbm_back_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:614 > #2 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, > old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #3 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/modify.c:380 > #4 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, > op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #5 0x000000000041ad49 in connection_threadmain () at > ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #6 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #7 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #8 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc882667180) at > ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8ac4fe456 in backentry_dup (e=0x7fc88267ebc0) at > ../389-ds-base/ldap/servers/slapd/back-ldbm/backentry.c:86 > #2 0x00007fc8ac53b572 in ldbm_back_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:622 > #3 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, > old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #4 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/modify.c:380 > #5 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, > op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #6 0x000000000041ad49 in connection_threadmain () at > ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #7 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #8 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #9 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc880c68180) at > ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8ac4fe456 in backentry_dup (e=0x7fc880c7d000) at > ../389-ds-base/ldap/servers/slapd/back-ldbm/backentry.c:86 > #2 0x00007fc8ac53b846 in ldbm_back_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:680 > #3 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, > old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #4 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/modify.c:380 > #5 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, > op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #6 0x000000000041ad49 in connection_threadmain () at > ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #7 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #8 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #9 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc882667180) at > ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8b062528b in internal_plugin_search_entry_callback > (e=0x7fc882667180, callback_data=0x7fc8825fa9b0) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:531 > #2 0x00007fc8b0624549 in internal_srch_entry_callback (be=0x7fc8aee457e0, > conn=0x0, op=0x7fc880c3c400, e=0x7fc882667180) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:99 > #3 0x00007fc8b063a053 in send_ldap_search_entry_ext (pb=0x7fc880bff120, > e=0x7fc882667180, ectrls=0x0, attrs=0x0, attrsonly=0, send_result=0, > nentries=0, > urls=0x0) at ../389-ds-base/ldap/servers/slapd/result.c:1507 > #4 0x00007fc8b0639194 in send_ldap_search_entry (pb=0x7fc880bff120, > e=0x7fc882667180, ectrls=0x0, attrs=0x0, attrsonly=0) > at ../389-ds-base/ldap/servers/slapd/result.c:1050 > #5 0x00007fc8b060ded1 in send_entry (pb=0x7fc880bff120, e=0x7fc882667180, > operation=0x7fc880c3c400, attrs=0x0, attrsonly=0, pnentries=0x7fc8825f62ec) > at ../389-ds-base/ldap/servers/slapd/opshared.c:1132 > #6 0x00007fc8b060e497 in iterate (pb=0x7fc880bff120, be=0x7fc8aee457e0, > send_result=1, pnentries=0x7fc8825f62ec, pagesize=-1, pr_statp=0x7fc8825f62dc) > at ../389-ds-base/ldap/servers/slapd/opshared.c:1463 > #7 0x00007fc8b060e8a5 in send_results_ext (pb=0x7fc880bff120, send_result=1, > nentries=0x7fc8825f62ec, pagesize=-1, pr_stat=0x7fc8825f62dc) > at ../389-ds-base/ldap/servers/slapd/opshared.c:1680 > #8 0x00007fc8b060d444 in op_shared_search (pb=0x7fc880bff120, send_result=1) > at ../389-ds-base/ldap/servers/slapd/opshared.c:875 > #9 0x00007fc8b06258c8 in search_internal_callback_pb (pb=0x7fc880bff120, > callback_data=0x7fc8825fa9b0, prc=0x7fc8b062534c > <internal_plugin_result_callback>, > psec=0x7fc8b0625257 <internal_plugin_search_entry_callback>, > prec=0x7fc8b06252e2 <internal_plugin_search_referral_callback>) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:730 > #10 0x00007fc8b062542e in search_internal_pb (pb=0x7fc880bff120) at > ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:594 > #11 0x00007fc8b06251ca in slapi_search_internal_pb (pb=0x7fc880bff120) at > ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:504 > #12 0x00007fc8b0625cb8 in slapi_search_internal_get_entry (dn=0x7fc8abe90240, > attrs=0x0, ret_entry=0x7fc8825faa68, component_identity=0x7fc8ac6985e0) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:864 > #13 0x00007fc8ac4ca221 in mep_pre_op (pb=0x7fc880bff000, modop=4) at > ../389-ds-base/ldap/servers/plugins/mep/mep.c:2169 > #14 0x00007fc8ac4ca64f in mep_mod_pre_op (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/plugins/mep/mep.c:2303 > #15 0x00007fc8b061ed1c in plugin_call_func (list=0x7fc8ac6c3e00, > operation=461, pb=0x7fc880bff000, call_one=0) at > ../389-ds-base/ldap/servers/slapd/plugin.c:2030 > #16 0x00007fc8b061eb85 in plugin_call_list (list=0x7fc8aef6fe00, > operation=461, pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/plugin.c:1973 > #17 0x00007fc8b061b912 in plugin_call_plugins (pb=0x7fc880bff000, > whichfunction=461) at ../389-ds-base/ldap/servers/slapd/plugin.c:442 > #18 0x00007fc8ac53b87c in ldbm_back_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:687 > #19 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, > old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #20 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/modify.c:380 > #21 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, > op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #22 0x000000000041ad49 in connection_threadmain () at > ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #23 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #24 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #25 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc880c68300) at > ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8b0625d24 in slapi_search_internal_get_entry (dn=0x7fc8abe90240, > attrs=0x0, ret_entry=0x7fc8825faa68, component_identity=0x7fc8ac6985e0) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:873 > #2 0x00007fc8ac4ca221 in mep_pre_op (pb=0x7fc880bff000, modop=4) at > ../389-ds-base/ldap/servers/plugins/mep/mep.c:2169 > #3 0x00007fc8ac4ca64f in mep_mod_pre_op (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/plugins/mep/mep.c:2303 > #4 0x00007fc8b061ed1c in plugin_call_func (list=0x7fc8ac6c3e00, > operation=461, pb=0x7fc880bff000, call_one=0) at > ../389-ds-base/ldap/servers/slapd/plugin.c:2030 > #5 0x00007fc8b061eb85 in plugin_call_list (list=0x7fc8aef6fe00, > operation=461, pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/plugin.c:1973 > #6 0x00007fc8b061b912 in plugin_call_plugins (pb=0x7fc880bff000, > whichfunction=461) at ../389-ds-base/ldap/servers/slapd/plugin.c:442 > #7 0x00007fc8ac53b87c in ldbm_back_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:687 > #8 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, > old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #9 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/modify.c:380 > #10 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, > op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #11 0x000000000041ad49 in connection_threadmain () at > ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #12 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #13 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #14 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc880c68180) at > ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8ac53bfb6 in ldbm_back_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:844 > #2 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, > old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #3 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at > ../389-ds-base/ldap/servers/slapd/modify.c:380 > #4 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, > op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #5 0x000000000041ad49 in connection_threadmain () at > ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #6 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #7 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #8 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > -- > > 389 Directory Server Development Team > _______________________________________________ > 389-devel mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-devel mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
