On Wed, Oct 4, 2023 at 11:40 AM Alexey Tikhonov <[email protected]> wrote:
> > > On Tue, Oct 3, 2023 at 11:22 PM Spike White <[email protected]> > wrote: > >> Alexey, >> >> Yes I see that now. That every time it starts a new LDAP connection, it >> starts by querying rootDSE. So I have to look further in the logs. >> >> I think I have discerned a pattern. It appears that on each hour and >> half-hour, it's querying the members of the simple_allow_groups line. >> > > A cron job run under a corresponding user? > > In the domain log, next line after "Got request for" there should be line > like > ``` > DP Request [Account #2]: REQ_TRACE: New request. [sssd.nss CID #1] > ``` > -- this tells you what service ('sssd_nss' in this case) requested this > lookup and what is request ID in its logs ('#1' in this case). > > Next you can grep sssd_nss.log to figure out who app triggered this: > ``` > (2023-04-14 14:02:08): [nss] [get_client_cred] (0x4000): Client > [0x55eb3ac27760][27] creds: euid[0] egid[0] pid[181089] cmd_line['id']. > (2023-04-14 14:02:08): [nss] [setup_client_idle_timer] (0x4000): Idle > timer re-set for client [0x55eb3ac27760][27] > (2023-04-14 14:02:08): [nss] [accept_fd_handler] (0x0400): [CID#1] Client > [cmd id][uid 0][0x55eb3ac27760][27] connected! > ``` > -- 'id' in this case. > You could also give a try `sssctl analyze request list` and `sssctl analyze --logdir . request show ...` > > > >> I have examined this on 5 different servers in different >> geographical locations, it holds true for each server. >> >> for example, in /var/log/sssd dir: >> >> # grep -A 4 'sbus_dispatch.*Dispatching.' sssd_amer.corp.com.log | grep >> 'name=' | grep BE_REQ_GROUP >> >> Here's the output. Each ellipsis is 10 - 20 lines omitted that occurs in >> the same second. >> >> (2023-10-03 10:07:50): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> (2023-10-03 10:07:50): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> ... >> (2023-10-03 10:30:02): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> ... >> (2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> (2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> (2023-10-03 10:30:03): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> (2023-10-03 11:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 11:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 11:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> ... >> (2023-10-03 11:30:02): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 11:30:02): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 11:30:06): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 11:30:06): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> ... >> (2023-10-03 12:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 12:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 12:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 12:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 12:00:06): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> ... >> (2023-10-03 12:30:02): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 12:30:02): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 12:30:04): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> ... >> (2023-10-03 12:41:16): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 12:41:16): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 12:41:16): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> ... >> (2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> (2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> (2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> (2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> (2023-10-03 12:49:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][[email protected]] >> ... >> (2023-10-03 13:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 13:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 13:00:01): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> ... >> (2023-10-03 13:26:18): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 13:26:18): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> ... >> (2023-10-03 13:30:02): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 13:30:02): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 13:30:02): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> (2023-10-03 13:30:02): [be[amer.corp.com]] [dp_get_account_info_send] >> (0x0200): Got request for [0x2][BE_REQ_GROUP][name= >> [email protected]] >> ... >> and it continues on, each and every half-hour. >> >> So it appears that something is waking up every half-hour and validating >> memberships in the simple_allow_groups. I don't claim that's all that's >> being performed on this half-hour wake-up, but it is clear this is occuring. >> >> Different servers have different simple_allow_group memberships; it's >> always the memberships for that specific server that's being queried. >> >> Spike >> >> >> On Mon, Oct 2, 2023 at 12:23 PM Alexey Tikhonov <[email protected]> >> wrote: >> >>> On Mon, Oct 2, 2023 at 7:01 PM Spike White <[email protected]> >>> wrote: >>> > >>> > So the idea to turn on debug_level = 9 on the client and view the logs >>> was inspired. We turned on debug level 9 on 4 clients; >>> > >>> > 2 in the list (that we got from AD team of servers in that AMERAustin >>> site hitting the non-AMER Austin AD DCs). >>> > 2 not in their list. (1 in another AMER site). >>> > >>> > Consistently, we see them querying the rootDSE for all these domains >>> on the hour and the half-hour. Querying the local AMER rootDSE in Austin >>> is not a problem; they have beaucoup AD DCs in Austin. Querying the other >>> domains' rootDSEs in Austin is a problem; they typically only have 2 AD >>> DCs from each region. >>> > >>> > Here’s an example from the client logs. First client: >>> > >>> > >>> > >>> > (2023-10-03 0:30:06): [be[amer.corp.com]] >>> [sdap_ldap_connect_callback_add] (0x1000): New LDAP connection to [ldap:// >>> ausdc16corp05.corp.com:389/??base] with fd [30]. >>> > >>> > (2023-10-03 0:30:06): [be[amer.corp.com]] [sdap_get_rootdse_send] >>> (0x4000): Getting rootdse >>> > >>> > … >>> > >>> > (2023-10-03 0:41:18): [be[amer.corp.com]] >>> [sdap_ldap_connect_callback_add] (0x1000): New LDAP connection to [ldap:// >>> AUSDC16ROAMER01.amer.corp.com:389/??base] with fd [20]. >>> > >>> > (2023-10-03 0:41:18): [be[amer.corp.com]] [sdap_get_rootdse_send] >>> (0x4000): Getting rootdse >>> > >>> > >>> > >>> > Another client: >>> > >>> > >>> > >>> > (2023-10-02 11:30:02): [be[amer.corp.com]] >>> [sdap_ldap_connect_callback_add] (0x4000): [RID#48] New connection to >>> [ldap://ausdc16amer33.amer.corp.com:3268/??base] with fd [25] >>> > >>> > (2023-10-02 11:30:02): [be[amer.corp.com]] [sdap_get_rootdse_send] >>> (0x4000): [RID#48] Getting rootdse >>> > >>> >>> What goes next for this 'RID#48', after 'rootdse' is read? >>> >>> SSSD doesn't query 'rootdse' on its own. It is being read (as a first >>> operation) when a new connection is established. >>> You need to see what happens next to figure out *why* this new >>> connection is established. >>> >>> As a practical side note, you can also increase >>> `ldap_connection_expire_timeout` to keep connections longer. >>> But I would figure out the reason first. >>> >>> >>> > -- >>> > >>> > (2023-10-02 11:30:04): [be[amer.corp.com]] >>> [sdap_ldap_connect_callback_add] (0x4000): [RID#49] New connection to >>> [ldap://ausdc16emea05.emea.corp.com:389/??base] with fd [26] >>> > >>> > (2023-10-02 11:30:04): [be[amer.corp.com]] [sdap_get_rootdse_send] >>> (0x4000): [RID#49] Getting rootdse >>> > >>> > -- >>> > >>> > (2023-10-02 11:30:05): [be[amer.corp.com]] >>> [sdap_ldap_connect_callback_add] (0x4000): [RID#50] New connection to >>> [ldap://ausdc16apac06.apac.corp.com:389/??base] with fd [27] >>> > >>> > (2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_get_rootdse_send] >>> (0x4000): [RID#50] Getting rootdse >>> > >>> > -- >>> > >>> > (2023-10-02 11:30:05): [be[amer.corp.com]] >>> [sdap_ldap_connect_callback_add] (0x4000): [RID#51] New connection to >>> [ldap://AUSDC16JAPN02.japn.corp.com:389/??base] with fd [28] >>> > >>> > (2023-10-02 11:30:05): [be[amer.corp.com]] [sdap_get_rootdse_send] >>> (0x4000): [RID#51] Getting rootdse >>> > >>> > -- >>> > >>> > (2023-10-02 11:32:52): [be[amer.corp.com]] >>> [sdap_ldap_connect_callback_add] (0x4000): [RID#84] New connection to >>> [ldap://ausdc16emea05.emea.corp.com:389/??base] with fd [26] >>> > >>> > (2023-10-02 11:32:52): [be[amer.corp.com]] [sdap_get_rootdse_send] >>> (0x4000): [RID#84] Getting rootdse >>> > >>> > -- >>> > >>> > >>> > >>> > BTW, this seems to occurs on both RHEL7 and RHEL8. (Haven't looked at >>> our RHEL9 builds yet). It's occurring on all servers to all rootDSEs, but >>> only a problem for AMERAustin, since Austin is such a heavily-populated. >>> > >>> > >>> > These rootDSEs change almost never. Any way to have it query not as >>> frequently, or randomize when servers query these rootDSEs. >>> > >>> > >>> > Spike >>> > >>> > On Mon, Oct 2, 2023 at 2:37 AM Alexey Tikhonov <[email protected]> >>> wrote: >>> >> >>> >> Hi, >>> >> >>> >> On Mon, Oct 2, 2023 at 6:20 AM Spike White <[email protected]> >>> wrote: >>> >>> >>> >>> All, >>> >>> >>> >>> Is there anything in sssd's RHEL and RHEL-like Linux server OS >>> settings that perform LDAP binds or connections to AD every 30 minutes? >>> >>> >>> >>> What our AD team is seeing is all of the DCs in our biggest AMER AD >>> site peak with LDAP sessions for about 10 minutes at the top of the hour >>> then again at the bottom of the hour. No other AD site in the world >>> appears to see this behavior not even other AD sites in this metro area. >>> >>> >>> >>> The reason they noticed is that our non-amer DCs in this biggest AD >>> site hit their 5k LDAP client session limit during those 10 minutes every >>> 30 minutes. Meaning any clients attempting to establish a LDAP session >>> past 5000 are dropped by the DC. In their research they see thousands LDAP >>> Binds by RHEL Linux servers against two specific non-AMER AD DCs in a short >>> period of time after digging through some LDAP log samples that they pulled >>> from these DCs. >>> >> >>> >> >>> >> Can they also say what operations are being performed by those >>> connections? >>> >> Or can you check SSSD logs on the client side? >>> >> >>> >> I wonder if this could be `ldap_sudo_smart_refresh_interval`... >>> >> >>> >> >>> >>> >>> >>> >>> >>> In this major AD sites, we have dozens and dozens of AMER AD DCs. >>> So there's enough preferred AD DCs to spread the load. But typically for >>> the non-AMER regions, the AD team puts 2 of each regions DCs in a site. >>> For instance, for APAC they would be put two APAC DCs in this AMER major >>> site. Thus all AMER RHEL servers in this site would randomly hit dozens of >>> AMER DCs, but concentrate on these two preferred APAC DCs. (preferred >>> because they're in this locatiion). >>> >>> >>> >>> I know our older AD integration product used to hit AD every 30 mins >>> to check GPOs, but we're not implementing GPOs with sssd. >>> >>> >>> >>> Spike >>> >>> _______________________________________________ >>> >>> sssd-users mailing list -- [email protected] >>> >>> To unsubscribe send an email to >>> [email protected] >>> >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> >>> List Archives: >>> https://lists.fedorahosted.org/archives/list/[email protected] >>> >>> Do not reply to spam, report it: >>> https://pagure.io/fedora-infrastructure/new_issue >>> >> >>> >> _______________________________________________ >>> >> sssd-users mailing list -- [email protected] >>> >> To unsubscribe send an email to >>> [email protected] >>> >> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> >> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> >> List Archives: >>> https://lists.fedorahosted.org/archives/list/[email protected] >>> >> Do not reply to spam, report it: >>> https://pagure.io/fedora-infrastructure/new_issue >>> > >>> > _______________________________________________ >>> > sssd-users mailing list -- [email protected] >>> > To unsubscribe send an email to >>> [email protected] >>> > Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> > List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> > List Archives: >>> https://lists.fedorahosted.org/archives/list/[email protected] >>> > Do not reply to spam, report it: >>> https://pagure.io/fedora-infrastructure/new_issue >>> _______________________________________________ >>> sssd-users mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedorahosted.org/archives/list/[email protected] >>> Do not reply to spam, report it: >>> https://pagure.io/fedora-infrastructure/new_issue >>> >> _______________________________________________ >> sssd-users mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedorahosted.org/archives/list/[email protected] >> Do not reply to spam, report it: >> https://pagure.io/fedora-infrastructure/new_issue >> >
_______________________________________________ sssd-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
