On Tue, Dec 15, 2020 at 01:56:57AM +0000, Sanjay Agrawal wrote: > Thanks Sumit, it worked for us. For some if queries still taking too long. I > have collected all the logs. They are pretty large. Is there w site I can > upload to ?
Hi, if you have a place where you can upload them you can send me the link directly. Otherwise you can try to open a ticket a bugzilla.redhat.com and try to upload the logs there but I do not know which limit applies. bye, Sumit > > Thanks,Sanjay Agrawal > > On Friday, December 4, 2020, 07:24:31 AM EST, Sumit Bose > <sb...@redhat.com> wrote: > > On Thu, Dec 03, 2020 at 10:44:09PM +0000, Sanjay Agrawal wrote: > > Thanks for response. I re-ran test after adding index as suggested. Which > > seems to fix the underlying issue and results are inline with older > > version. So will this index would be added in newer version of sssd ? Is > > there an easier way to add this index w/o opening an editor ? > > Hi, > > something like: > > ldbmodify -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b > '@INDEXLIST' << END_END > dn: @INDEXLIST > changetype: modify > add: @IDXATTR > @IDXATTR: originalADgidNumber > > END_END > > should work as well or you put ldif data into a file and add the > filename to the command line. > > > > > older version > > sssd-20201203163147/analysis.out:'Initgroups by name' count: 102 avg: > > 0.596225137254902 median 0.035176 min: 0.001218 max: 22.037332 > > sssd-20201203164328/analysis.out:'Initgroups by name' count: 201 avg: > > 0.313984855721393 median 0.039057 min: 0.000394 max: 23.138423 > > sssd-20201203165234/analysis.out:'Initgroups by name' count: 100 avg: > > 0.59339562 median 0.0362155 min: 0.000412 max: 21.559991 > > > > With new version + adding index as suggested above. > > sssd-20201203145950/analysis.out:'Initgroups by name' count: 103 avg: > > 0.5026502815533981 median 0.03847 min: 0.000435 max: 21.532709 > > sssd-20201203152524/analysis.out:'Initgroups by name' count: 90 avg: > > 1.1182167555555556 median 0.0349585 min: 0.000536 max: 33.604103 > > sssd-20201203160837/analysis.out:'Initgroups by name' count: 102 avg: > > 0.4324193431372549 median 0.0338985 min: 0.000479 max: 19.493188 > > > > Above output is by parsing the sssd_nss.log with debug_level 9. As you can > > see from both older/newer versions, we are seeing a very high max value. > > All these are when request is being sent from nss->be for resolution. Is is > > possible similar issue is happening at BE side as well when updating ldb ? > > Please let me know, if you need additional data. > > Yes, the max times are most likely related to looking up fresh data from > the server via the backend. It is not unusual the Initgroups request > will take some time, e.g. if you user belongs to many groups and the > cache groups entries are expired as well and must be refreshed as well. > But to better understand what is the reason for the long delay in your > case we would need the backend logs with debug_level=9. There are some > code paths known to be slow, Alexey is e.g. currently working on one of > them. But there might be other reasons for delay as well, e.g. slowly > responding servers, failover to a different server etc. > > bye, > Sumit > > > > > Thanks > > > > > > Sanjay Agrawal > > > > On Thursday, December 3, 2020, 01:32:32 PM EST, Sumit Bose > ><sb...@redhat.com> wrote: > > > > On Thu, Dec 03, 2020 at 05:50:31PM +0000, Sanjay Agrawal wrote: > > > resending > > > Sanjay Agrawal > > > > > >  On Wednesday, December 2, 2020, 03:41:36 PM EST, Sanjay Agrawal > > ><sanjayagra...@yahoo.com> wrote: > > > > > > Hi, > > > > > > we are seeing an issue with newer version of sssd with centos 7.9 sssd > > > version 1.16.5-10.el7_9.5.x86_64, where Initgroups is taking much longer > > > compared to > > > previous version. Can you please look into it. Following are details, > > > including a sample program to reproduce the issue with old and new > > > version. > > > > > > Sample Program: get_user_groups.py > > >   - it just call getgrouplist of a user supplied using libc > > > > > > Old env - > > > OS Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Wed Aug 26 11:48:49 BST 2020 > > > x86_64 x86_64 x86_64 GNU/Linux > > > SSSD Version   - 1.16.4-37.el7_8.4.x86_64 > > > Sample log file - sssd_nss-old.log > > > Flamegraph    - flamgraph_sssd_nss-old.svg > > > ID  879 'Initgroups by name' testuser1@example > > >   starttime Tue Dec  1 18:36:42:092541 2020 > > >   endtime Tue Dec  1 18:36:42:124463 2020 > > >   lookup_time 0.031922 sec > > > > > > New env > > > OS Linux 3.10.0-1160.6.1.el7.x86_64 #1 SMP Wed Nov 18 22:40:48 GMT 2020 > > > x86_64 x86_64 x86_64 GNU/Linux > > > SSSD Version   - 1.16.5-10.el7_9.5.x86_64 > > > Sample log file - sssd_nss-new.log > > > Flamegraph    - flamgraph_sssd_nss-new.svg > > > ID  776 'Initgroups by name' testuser1@example > > >   starttime 2020-12-01 17:31:24:328419 > > >   endtime 2020-12-01 17:31:24:451778 > > >   lookup_time 0.123359 sec > > > > > > > > > It seem to be due to addtional .1 second taken during following two trace > > > (from sssd_nss-new.log) > > > (2020-12-01 17:31:24:355458): [nss] [cache_req_done] (0x0400): CR #776: > > > Finished: Success > > > (2020-12-01 17:31:24:451727): [nss] [sysdb_search_group_by_id] (0x0400): > > > No such entry > > > (2020-12-01 17:31:24:451757): [nss] [nss_protocol_fill_initgr] (0x0080): > > > Unable to find primary gid [2]: No such file or directory > > > > > > It may be related to following change, which seems to ref to > > > sysdb_search_group_by_id > > > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71 > > > > Hi, > > > > thanks for the analysis and I guess you might be right. The patch adds a > > search where one attribute in the filter does not have an index. As a > > result the search has to run over the whole cache. > > > > I wonder if you would like to try to add such index? To do this please > > stop SSSD and call: > > > >   ldbedit -H /var/lib/sss/db/cache_YOUR.DOMAIN.NAME.ldb -s base -b > > '@INDEXLIST' > > > > which should open an editor showing something similar to > > > > > > # editing 1 records > > # record 1 > > dn: @INDEXLIST > > @IDXATTR: cn > > @IDXATTR: objectclass > > @IDXATTR: member > > @IDXATTR: memberof > > @IDXATTR: name > > @IDXATTR: uidNumber > > @IDXATTR: gidNumber > > @IDXATTR: lastUpdate > > @IDXATTR: dataExpireTimestamp > > @IDXATTR: originalDN > > @IDXATTR: nameAlias > > @IDXATTR: servicePort > > @IDXATTR: serviceProtocol > > @IDXATTR: sudoUser > > @IDXATTR: sshKnownHostsExpire > > @IDXATTR: objectSIDString > > @IDXATTR: ghost > > @IDXATTR: userPrincipalName > > @IDXATTR: canonicalUserPrincipalName > > @IDXATTR: uniqueID > > @IDXATTR: mail > > @IDXATTR: userMappedCertificate > > @IDXATTR: ccacheFile > > @IDXATTR: ipHostNumber > > @IDXATTR: ipNetworkNumber > > distinguishedName: @INDEXLIST > > > > > > Please add in the @IDXATTR list a line > > > >   @IDXATTR: originalADgidNumber > > > > and exit the editor, the databased will be reindexed at this point. > > Finally start SSSD again and rerun the test. > > > > HTH > > > > bye, > > Sumit > > > > > > > > | > > > | > > > | > > > | | | > > > > > > | > > > > > > | > > > | > > > | | > > > nss: use real primary gid if the value is overriden · SSSD/sssd@80e6f71 > > > > > > SYSDB_PRIMARY_GROUP_GIDNUM contains original primary group id from AD > > > because any possible override may not be k... > > > | > > > > > > | > > > > > > | > > > > > > > > > > > > > > > > > > Sanjay Agrawal > > > > > _______________________________________________ > > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > > > Fedora Code of Conduct: > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: > > > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > > _______________________________________________ > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > > > > > _______________________________________________ > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > _______________________________________________ > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > > _______________________________________________ > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
