Thanks for responding. what you said was not exactly my situation, but it got me poking around and finally I got the configuration working.
I find this interesting item when looking at various sssd subcommands; [root@spikerealmd02 ~]# sssctl domain-list amer.dell.com apac.dell.com emea.dell.com japn.dell.com dell.com apac.dell.com emea.dell.com japn.dell.com *Why are the remote subdomains duplicated? * Ultimately, this was my problem. When I clear my logs: sssctl remove-logs and II do a: getent passwd admjesse_chan which still fails. via 'grep -il admjesse_chan sssd_*.log', I find string only in sssd_amer.dell.com.log and sssd_nss.log. sssd_apac.dell.com.log does the usual site-awareness lookup and identified primary LDAP server optimally. So this sssd_be subprocess is working correctly. Yet this log file never receives a query request for ' admjesse_c...@apac.dell.com'. sssd_nss.log gets a query request for admjesse_chan. It knows about remote subdomains: (Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_set_plugin] (0x2000): CR #1753: Setting "User by name" plugin (Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_send] (0x0400): CR #1753: New request 'User by name' (Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_process_input] (0x0400): CR #1753: Parsing input name [admjesse_chan] *(Wed Jul 4 01:21:34 2018) [sssd[nss]] [sss_domain_get_state] (0x1000): Domain apac.dell.com <http://apac.dell.com> is Active* *(Wed Jul 4 01:21:34 2018) [sssd[nss]] [sss_domain_get_state] (0x1000): Domain emea.dell.com <http://emea.dell.com> is Active* *(Wed Jul 4 01:21:34 2018) [sssd[nss]] [sss_domain_get_state] (0x1000): Domain japn.dell.com <http://japn.dell.com> is Active* Since admjesse_chan was specified with domain, it performs a multi-domain search. it searches local domain first. It searches cache for admjesse_c...@amer.dell.com and then data_provider first: (Wed Jul 4 01:21:34 2018) [sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'admjesse_chan' matched without domain, user is admjesse_chan (Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_set_name] (0x0400): CR #1753: Setting name [admjesse_chan] (Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_select_domains] (0x0400): CR #1753: Performing a multi-domain search (Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_search_domains] (0x0400): CR #1753: Search will check the cache and check the data provider ... (Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #1753: Object [admjesse_c...@amer.dell.com] was not found in cache (Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_search_dp] (0x0400): CR #1753: Looking up [admjesse_c...@amer.dell.com] in data provider It dispatches to data_provider (child process sssd_be for amer.dell.com) and receives successful response back (no records found): (Wed Jul 4 01:21:34 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [amer.dell.com ][0x1][BE_REQ_USER][name=admjesse_c...@amer.dell.com:-] (Wed Jul 4 01:21:34 2018) [sssd[nss]] [sbus_add_timeout] (0x2000): 0x55f844a5cac0 (Wed Jul 4 01:21:34 2018) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x55f842febb50:1:admjesse_c...@amer.dell.com@amer.dell.com ] (Wed Jul 4 01:21:34 2018) [sssd[nss]] [sbus_remove_timeout] (0x2000): 0x55f844a5cac0 (Wed Jul 4 01:21:34 2018) [sssd[nss]] [sbus_dispatch] (0x4000): dbus conn: 0x55f844a330f0 *(Wed Jul 4 01:21:34 2018) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching.* *(Wed Jul 4 01:21:34 2018) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success* So next, it searches the next domain -- emea.dell.com: (Wed Jul 4 01:21:34 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [emea.dell.com ][0x1][BE_REQ_USER][name=admjesse_c...@emea.dell.com:-] ... (Wed Jul 4 01:21:34 2018) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching. *(Wed Jul 4 01:21:34 2018) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 3 errno: 5 error message: Input/output error* *(Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_common_dp_recv] (0x0040): CR #1753: Data Provider Error: 3, 5, Input/output error* *(Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_common_dp_recv] (0x0400): CR #1753: Due to an error we will return cached data* (Wed Jul 4 01:21:34 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #1753: Looking up [admjesse_c...@emea.dell.com] in cache It catches the same I/O error when dispatching to the other data_providers: apac.dell.com, japn.dell.com So I focused on -- why are there duplicate entries in sssctl domain-list? Might it be trying to dispatch to some "ghost" data_provider (non-existent sssd_be child process)? Thus I focused on my sssd.conf file. i realized I don't need this line: [sssd] debug_level = 6 domains = amer.dell.com,apac.dell.com,emea.dell.com,japn.dell.com *domain_resolution_order = amer.dell.com <http://amer.dell.com>, emea.dell.com <http://emea.dell.com>, apac.dell.com <http://apac.dell.com>, japn.dell.com <http://japn.dell.com>* As without it, it will search domains in the order listed in "domains" line. Tried that fix, made no difference. Next I realized that https://lists.fedorahosted.org/pipermail/sssd-users/2015-February/002648.html doesn't have these lines in each of his [domain/XXX] stanzas. ad_enabled_domains = amer.dell.com,apac.dell.com,emea.dell.com,japn.dell.com ,dell.com So I removed that line from each [domain/XXX] stanza and did a: sssctl cache-remove (which restarts sssd service) Now sssctl domain-list shows me no dups: [root@spikerealmd02 sssd]# sssctl domain-list amer.dell.com apac.dell.com emea.dell.com japn.dell.com Now the dispatches out of sssd_nss.log to the various data_providers succeed. And I see remote users: [root@spikerealmd02 sssd]# id admjesse_chan uid=525641(admjesse_c...@apac.dell.com) gid=525641( admjesse_c...@apac.dell.com) groups=525641(admjesse_c...@apac.dell.com ),1008(apacunixus...@apac.dell.com),1000(apaclinux...@apac.dell.com),1001( apaclinux...@apac.dell.com) I still don't see all expected groups for these users, so my ldap_use_tokengroups is still not 100% right. Or maybe these missing groups are UNIVERSAL groups and I should be additionally querying dell.com. Oh, well -problem for another day. At least now x-subdomain auth is again working and tokengroups is (mostly) working. Spike On Tue, Jul 3, 2018 at 12:21 AM Sumit Bose <sb...@redhat.com> wrote: > On Mon, Jul 02, 2018 at 09:52:17AM -0500, Spike White wrote: > > Thanks for prompt reply. > > > > Yes, user name is strewn throughout sssd_nss.log. Here's the last > little > > bit: > > > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] > (0x0400): > > CR #1653: Checking negative cache for [admjesse_c...@japn.dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_ncache_check_str] (0x2000): > > Checking negative cache for [NCE/USER/ > > japn.dell.com/admjesse_c...@japn.dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] > (0x0400): > > CR #1653: [admjesse_c...@japn.dell.com] is not present in negative cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Looking up [admjesse_c...@japn.dell.com] in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Object [admjesse_c...@japn.dell.com] was not found in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_dp] (0x0400): CR > > #1653: Looking up [admjesse_c...@japn.dell.com] in data provider > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_issue_request] (0x0400): > > Issuing request for [0x55f842febb50:1:admjesse_c...@japn.dell.com@ > > japn.dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): > > Creating request for [japn.dell.com > > ][0x1][BE_REQ_USER][name=admjesse_c...@japn.dell.com:-] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_internal_get_send] > (0x0400): > > Entering request [0x55f842febb50:1:admjesse_c...@japn.dell.com@ > japn.dell.com > > ] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_req_destructor] (0x0400): > > Deleting request: [0x55f842febb50:1:admjesse_c...@apac.dell.com@ > > apac.dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Looking up [admjesse_c...@japn.dell.com] in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Object [admjesse_c...@japn.dell.com] was not found in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_send] (0x0400): > CR > > #1653: Looking up admjesse_c...@dell.com > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] > (0x0400): > > CR #1653: Checking negative cache for [admjesse_c...@dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_ncache_check_str] (0x2000): > > Checking negative cache for [NCE/USER/dell.com/admjesse_c...@dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] > (0x0400): > > CR #1653: [admjesse_c...@dell.com] is not present in negative cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Looking up [admjesse_c...@dell.com] in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Object [admjesse_c...@dell.com] was not found in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_dp] (0x0400): CR > > #1653: Looking up [admjesse_c...@dell.com] in data provider > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_issue_request] (0x0400): > > Issuing request for [0x55f842febb50:1:admjesse_c...@dell.com@dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): > > Creating request for [dell.com > > ][0x1][BE_REQ_USER][name=admjesse_c...@dell.com:-] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_internal_get_send] > (0x0400): > > Entering request [0x55f842febb50:1:admjesse_c...@dell.com@dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_req_destructor] (0x0400): > > Deleting request: [0x55f842febb50:1:admjesse_c...@japn.dell.com@ > > japn.dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Looking up [admjesse_c...@dell.com] in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Object [admjesse_c...@dell.com] was not found in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_req_destructor] (0x0400): > > Deleting request: [0x55f842febb50:1:admjesse_c...@dell.com@dell.com] > > > > Specifically, this user is in apac.dell.com, so I want to make sure it's > > querying that subdomain. It is. > > > > [root@spikerealmd02 sssd]# grep 'admjesse_chan.*apac' sssd_nss.log > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_send] (0x0400): > CR > > #1653: Looking up admjesse_c...@apac.dell.com > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] > (0x0400): > > CR #1653: Checking negative cache for [admjesse_c...@apac.dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_ncache_check_str] (0x2000): > > Checking negative cache for [NCE/USER/ > > apac.dell.com/admjesse_c...@apac.dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] > (0x0400): > > CR #1653: [admjesse_c...@apac.dell.com] is not present in negative cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Looking up [admjesse_c...@apac.dell.com] in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Object [admjesse_c...@apac.dell.com] was not found in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_dp] (0x0400): CR > > #1653: Looking up [admjesse_c...@apac.dell.com] in data provider > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_issue_request] (0x0400): > > Issuing request for [0x55f842febb50:1:admjesse_c...@apac.dell.com@ > > apac.dell.com] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): > > Creating request for [apac.dell.com > > ][0x1][BE_REQ_USER][name=admjesse_c...@apac.dell.com:-] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_internal_get_send] > (0x0400): > > Entering request [0x55f842febb50:1:admjesse_c...@apac.dell.com@ > apac.dell.com > > ] > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Looking up [admjesse_c...@apac.dell.com] in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): > > CR #1653: Object [admjesse_c...@apac.dell.com] was not found in cache > > (Sun Jul 1 15:21:12 2018) [sssd[nss]] [sss_dp_req_destructor] (0x0400): > > Deleting request: [0x55f842febb50:1:admjesse_c...@apac.dell.com@ > > apac.dell.com] > > > > Yes, sss is in /etc/nsswitch.conf: > > > > [root@spikerealmd02 sssd]# grep sss /etc/nsswitch.conf > > passwd: files sss > > shadow: files sss > > group: files sss > > #initgroups: files sss > > #netgroup: files sss > > > > But I knew that already, as users in the local domain get resolved fine: > > > > [root@spikerealmd02 sssd]# id admspike_white > > uid=2025431(admspike_wh...@amer.dell.com) gid=2025431( > > admspike_wh...@amer.dell.com) groups=2025431( > admspike_wh...@amer.dell.com > > ),1002(amerlinux...@amer.dell.com),1041( > linux-core-engineer...@amer.dell.com > > ) > > > > Curiously, I see these local account lookups both in sssd_nss.log and > > in sssd_amer.dell.com.log (AMER.DELL.COM is the local domain). But for > > that APAC user, I see the entries (above) in sssd_nss.log, but no > equiv > > entries in sssd_apac.dell.com.log. > > You should check what happens in teh sssd_apac log around the '(Sun Jul > 1 15:21:12 2018)' timestamp. Maybe it is offline ('sssctl domain-status > apac.dell.com.log' might show this as well). There should be a debug > message with an offline error-code in sssd_nss.log as well. > > HTH > > bye, > Sumit > > > > Here's the systemctl status for sssd: > > > > [root@spikerealmd02 sssd]# systemctl status sssd > > ● sssd.service - System Security Services Daemon > > Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor > > preset: disabled) > > Active: active (running) since Mon 2018-06-25 00:05:10 CDT; 1 weeks 0 > > days ago > > Main PID: 82591 (sssd) > > CGroup: /system.slice/sssd.service > > ├─82591 /usr/sbin/sssd -i --logger=files > > ├─82592 /usr/libexec/sssd/sssd_be --domain amer.dell.com > --uid 0 > > --gid 0 --logger=files > > ├─82593 /usr/libexec/sssd/sssd_be --domain apac.dell.com > --uid 0 > > --gid 0 --logger=files > > ├─82594 /usr/libexec/sssd/sssd_be --domain emea.dell.com > --uid 0 > > --gid 0 --logger=files > > ├─82595 /usr/libexec/sssd/sssd_be --domain japn.dell.com > --uid 0 > > --gid 0 --logger=files > > ├─82596 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 > --logger=files > > └─82597 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 > --logger=files > > > > It's as if sssd_nss is properly invoking sub sssd process 82592 for > > amer.dell.com lookups. But not invoking sub sssd process 82593 for > > apac.dell.com lookups. > > > > Spike > > > > > > On Mon, Jul 2, 2018 at 2:32 AM Sumit Bose <sb...@redhat.com> wrote: > > > > > On Sun, Jul 01, 2018 at 03:25:26PM -0500, Spike White wrote: > > > > sssd subject matter experts, > > > > > > > > Why is my sssd deployment not doing cross-subdomain AD > authentication? > > > > > > > > > > > > > > > > *Background:* > > > > > > > > I have a parent AD domain DELL.COM with trusted subdomains > AMER.DELL.COM > > > , > > > > APAC.DELL.COM, EMEA.DELL.COM and JAPN.DELL.COM Each subdomain has a > > > > transitive trust with DELL.COM. > > > > > > > > So all subdomains trust each other. > > > > > > > > I set up a first test VM deployment using sssd. I set up the cross > > > > subdomain auth as in: > > > > > > > > > > > > https://lists.fedorahosted.org/pipermail/sssd-users/2015-February/002648.html > > > > > > > > It worked great – allowed cross subdomain authentication. The only > > > thing > > > > it would not do was use tokengroups. That is, the VM was fully > > > functional, > > > > but I had to add ‘ldap_use_tokengroups = false’ to the sssd.conf > file. > > > > > > > > My AD experts have advised me that ‘tokengroups’ are an important AD > > > > optimization and I should use them, if at all possible. > > > > > > > > Using ldapsearch, I was able to verify that machine account didn’t > have > > > the > > > > necessary privileges to query a user’s tokengroups. Thus, the fault > was > > > > mine – that this first sssd deployment couldn’t use tokengroups. > > > > > > > > So I did another sssd deployment, using another test VM. > Apparently, I > > > did > > > > the realm join command correct this time, as it’s able to use > > > tokengroups. > > > > > > > > BUT! This second test VM is not allowing cross subdomain > authentication > > > > and login. How do I fix this so that I have use of both > tokengroups > > > and > > > > cross subdomain authentication? > > > > > > > > > > ... > > > > > > > > > > > If I do the same query on the bad VM (spikerealmd02): > > > > > > > > > > > > > > > > [root@spikerealmd02 sssd]# id admjesse_chan > > > > > > > > id: admjesse_chan: no such user > > > > > > How do you know that tokengroups are working if the request fails and > > > the lookup is not recorded in the logs? > > > > > > > > > > > > > > > > > > > and I see nothing in the sssd_apac.dell.com.log file: > > > > > > > > > > > > > > > > [root@spikerealmd02 sssd]# grep -i admjesse_chan > sssd_apac.dell.com.log > > > > > > > > [root@spikerealmd02 sssd]# > > > > > > Do you see the user name in sssd_nss.log? If not, is 'sss' listed in > the > > > passwd line of /etc/nsswitch.conf? > > > > > > bye, > > > Sumit > > > > > > > > > > > > > > > > > > > Please help, > > > > > > > > Spike > > > > > > > _______________________________________________ > > > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > > > To unsubscribe send an email to > sssd-users-le...@lists.fedorahosted.org > > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > List Archives: > > > > https://lists.fedoraproject.org/archives/list/sssd-users@lists.fedorahosted.org/message/HHNJ6AMXTSKF4UW3PLS6Y5MY5ADF6VCV/ > > > _______________________________________________ > > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > > To unsubscribe send an email to > sssd-users-le...@lists.fedorahosted.org > > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: > > > > https://lists.fedoraproject.org/archives/list/sssd-users@lists.fedorahosted.org/message/BASM657T2U6AVKBKVBQVJQQ7PNVU7UC7/ > > > > > > _______________________________________________ > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedoraproject.org/archives/list/sssd-users@lists.fedorahosted.org/message/BKO5ZSU5AORKKVINJS4O6CZHRDXSQOF3/ > _______________________________________________ > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/sssd-users@lists.fedorahosted.org/message/X43BWGUCFVNLTAT346CQCPJZZ5QUGX2Z/ >
_______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/sssd-users@lists.fedorahosted.org/message/YDU6YH2JXX53I72AVNXZPN4DF54D3AUY/
