Am Thu, Sep 02, 2021 at 11:41:47AM -0500 schrieb Spike White: > OK, > > That particular candidate seems like a very unusual end corner case. Where > someone cloned an existing VM, renamed it, re-IP'd and (incorrectly) > re-joined it to AD. > > I saw "incorrectly", because they did not clear the existing > /etc/krb5.keytab file prior to the re-join. Hence, the old bogus entries > in /etc/krb5.keytab with the original hostname -- which confused adcli. > > BTW, Sumit -- I completely understand what you're saying about AD and > TGTs. You can request TGT tickets only off your UPN. So our UPN is always > 'host/fqdn@REALM'. in AD, you cannot request TGT tickets off your SPNs. > In other Kerberos implementations, they allow you to request TGT tickets > off SPNs -- but not AD. AD does not allow that. So I completely get the > difference between kinit -k and adcli. > > I think a better command-line test for us to troubleshoot sssd would be > request a Kerberos service ticket, not a Kerberos TGT ticket. But I > don't know how to do that on the command line (kinit only requests TGT > tickets and renews tickets). We use 'adcli testjoin' to simulate sssd > Kerberos initialization.
Hi, you can use 'kvno' to request a service ticket on the command line. > > Anyway, back to our research. We have now found 8 candidates that has AD > attribute 'passwordLastSet' between 31 days and 40 days. (Actually, two > are at 40 days, so probably AD has locked those machine accounts.) > > On this first candidate, we think it's another one-off for that particular > server. CPU pegged at 100% since Aug 9. We rebooted, set debug_level 9 > and it appears to have renewed this first attempt (based on timestamps). > We see this output in the sssd_amer.company.com.log > > (2021-09-01 14:44:36): [be[amer.company.com]] > [ad_machine_account_password_renewal_done] (0x1000): --- adcli output > start--- > ---adcli output end--- > (2021-09-01 14:44:36): [be[amer.company.com]] [be_ptask_done] (0x0400): > Task [AD machine account password renewal]: finished successfully > (2021-09-01 14:44:36): [be[amer.company.com]] [be_ptask_schedule] (0x0400): > Task [AD machine account password renewal]: scheduling task 86400 seconds > from last execution time [1630608275] > > We saw a second update attempt later (today?) with a lot of adcli output, > but it said: > > ...--- adcli output start--- > ... > * Password not too old, no change needed > ... > ---adcli output end--- > > So we suspect this candidate is yet another edge-corner case (CPU at 100% > -- not able to adcli update). > > Looking at the next couple of candidates, these are a more interesting (& > seemingly more common) case. They updated their entry in AD, but the local > /etc/krb5.keytab file was not updated. These happen to be OL7 servers (but > we have a RHEL7 candidate at 40 days non-check-in). Because these are > *L7, it's not the Kerberos UDP kpasswd problem (that's only on RHEL6/OL6). I recently came across a similar issue where an older SELinux policy was installed which prevented 'adcli' called from SSSD under SSSD's SELinux context to write to the keytab. bye, Sumit > > Let's take the first one. casnlrritpgm206. According to AD, it has a > 'passwordLastSet' of 7/28/2021. > > PS C:\Users\spike_white> get-adcomputer casnlrritpgm206 -properties > 'passwordLastSet' > > DistinguishedName : > CN=CASNLRRITPGM206,OU=Servers,OU=UNIX,DC=amer,DC=company,DC=com > DNSHostName : casnlrritpgm206.us.company.com > Enabled : True > Name : CASNLRRITPGM206 > ObjectClass : computer > ObjectGUID : f9fa2b5b-75b6-434b-8e94-477599d1afca > PasswordLastSet : 7/28/2021 10:04:49 PM > SamAccountName : CASNLRRITPGM206$ > SID : S-1-5-21-1802859667-647903414-1863928812-3091065 > UserPrincipalName : host/casnlrritpgm206.us.company....@amer.company.com > > It has a 'msDS-KeyVersionNumber' of 7. > > but in the local /etc/krb5.keytab file, it has a last KVNO of 6 with a > timestamp of 6/28: > > Keytab name: FILE:/etc/krb5.keytab > > KVNO Timestamp Principal > > ---- ------------------- > ------------------------------------------------------ > > 6 06/28/2021 06:53:30 CASNLRRITPGM206$@AMER.COMPANY.COM (arcfour-hmac) > > 6 06/28/2021 06:53:30 CASNLRRITPGM206$@AMER.COMPANY.COM > (aes128-cts-hmac-sha1-96) > > 6 06/28/2021 06:53:30 CASNLRRITPGM206$@AMER.COMPANY.COM > (aes256-cts-hmac-sha1-96) > > 6 06/28/2021 06:53:30 host/ > casnlrritpgm206.us.company....@amer.company.com (arcfour-hmac) > > 6 06/28/2021 06:53:30 host/ > casnlrritpgm206.us.company....@amer.company.com (aes128-cts-hmac-sha1-96) > > 6 06/28/2021 06:53:30 host/ > casnlrritpgm206.us.company....@amer.company.com (aes256-cts-hmac-sha1-96) > > 6 06/28/2021 06:53:30 host/casnlrritpgm...@amer.company.com > (arcfour-hmac) > > 6 06/28/2021 06:53:30 host/casnlrritpgm...@amer.company.com > (aes128-cts-hmac-sha1-96) > > 6 06/28/2021 06:53:30 host/casnlrritpgm...@amer.company.com > (aes256-cts-hmac-sha1-96) > > 6 06/28/2021 06:53:30 RestrictedKrbHost/casnlrritpgm...@amer.company.com > (arcfour-hmac) > > 6 06/28/2021 06:53:31 RestrictedKrbHost/casnlrritpgm...@amer.company.com > (aes128-cts-hmac-sha1-96) > > 6 06/28/2021 06:53:31 RestrictedKrbHost/casnlrritpgm...@amer.company.com > (aes256-cts-hmac-sha1-96) > > 6 06/28/2021 06:53:31 RestrictedKrbHost/ > casnlrritpgm206.us.company....@amer.company.com (arcfour-hmac) > > 6 06/28/2021 06:53:31 RestrictedKrbHost/ > casnlrritpgm206.us.company....@amer.company.com (aes128-cts-hmac-sha1-96) > > 6 06/28/2021 06:53:31 RestrictedKrbHost/ > casnlrritpgm206.us.company....@amer.company.com (aes256-cts-hmac-sha1-96) > > 5 05/29/2021 02:08:37 CASNLRRITPGM206$@AMER.COMPANY.COM (arcfour-hmac) > > 5 05/29/2021 02:08:37 CASNLRRITPGM206$@AMER.COMPANY.COM > (aes128-cts-hmac-sha1-96) > > 5 05/29/2021 02:08:37 CASNLRRITPGM206$@AMER.COMPANY.COM > (aes256-cts-hmac-sha1-96) > > > So 7/28 is 30 days after 6/28. It appears on 7/28 that sssd invoked > adcli update to update the machine account password in AD. adcli update > updated it in AD, but not in the local /etc/krb5.keytab file. > > > The next candidate is similar. It has a KVNO in AD one more than in > /etc/krb5.keytab file with a timestamp 30 days past the timestamp of the > latest entry in /etc/krb5.keytab file. > > > This sure seems similar to the Kerberos kpasswd UDP problem. But it's not > -- krb5-libs quit using UDP for kpasswd after RHEL6/OL6. > > > We know how to remediate when we hit such a candidate. adcli update with > the valid user principal and valid login ccache of a principal that have AD > privs to update these machine accounts. > > > So -- this is the ultimate question? *Why* is this happening? Why is the > adcli update (called by sssd) updating the passwd in AD and > the msDS-KeyVersionNumber, but not updating /etc/krb5.keytab? > > > I think this is the common case that we're seeing -- that these other cases > (plus one other) are the unusual end-corner cases. > > > Spike > > On Thu, Sep 2, 2021 at 12:49 AM Sumit Bose <sb...@redhat.com> wrote: > > > Am Wed, Sep 01, 2021 at 11:39:30AM -0500 schrieb Spike White: > > > So to respond to my own email, but a co-worker did finally find some > > > references to that bizarre name ZZZKBTDURBOL8. > > > > > > [root@nwpllv8bu100 post_install]# klist -kte > > > Keytab name: FILE:/etc/krb5.keytab > > > KVNO Timestamp Principal > > > ---- ------------------- > > > ------------------------------------------------------ > > > 2 07/13/2021 16:42:17 ZZZKBTDURBOL8$@AMER.COMPANY.COM > > > (aes128-cts-hmac-sha1-96) > > > 2 07/13/2021 16:42:17 ZZZKBTDURBOL8$@AMER.COMPANY.COM > > > (aes256-cts-hmac-sha1-96) > > > 2 07/13/2021 16:42:17 host/ > > > zzzkbtdurbol8.amer.company....@amer.company.com > > (aes128-cts-hmac-sha1-96) > > > 2 07/13/2021 16:42:17 host/ > > > zzzkbtdurbol8.amer.company....@amer.company.com > > (aes256-cts-hmac-sha1-96) > > > 2 07/13/2021 16:42:17 host/zzzkbtdurb...@amer.company.com > > > (aes128-cts-hmac-sha1-96) > > > 2 07/13/2021 16:42:17 host/zzzkbtdurb...@amer.company.com > > > (aes256-cts-hmac-sha1-96) > > > 2 07/13/2021 16:42:17 RestrictedKrbHost/ > > zzzkbtdurb...@amer.company.com > > > (aes128-cts-hmac-sha1-96) > > > 2 07/13/2021 16:42:17 RestrictedKrbHost/ > > zzzkbtdurb...@amer.company.com > > > (aes256-cts-hmac-sha1-96) > > > 2 07/13/2021 16:42:17 RestrictedKrbHost/ > > > zzzkbtdurbol8.amer.company....@amer.company.com > > (aes128-cts-hmac-sha1-96) > > > 2 07/13/2021 16:42:17 RestrictedKrbHost/ > > > zzzkbtdurbol8.amer.company....@amer.company.com > > (aes256-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM > > > (DEPRECATED:arcfour-hmac) > > > 3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM > > > (aes128-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM > > > (aes256-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 host/ > > nwpllv8bu100.amer.company....@amer.company.com > > > (DEPRECATED:arcfour-hmac) > > > 3 07/29/2021 13:38:27 host/ > > nwpllv8bu100.amer.company....@amer.company.com > > > (aes128-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 host/ > > nwpllv8bu100.amer.company....@amer.company.com > > > (aes256-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 host/nwpllv8bu...@amer.company.com > > > (DEPRECATED:arcfour-hmac) > > > 3 07/29/2021 13:38:27 host/nwpllv8bu...@amer.company.com > > > (aes128-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 host/nwpllv8bu...@amer.company.com > > > (aes256-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 RestrictedKrbHost/nwpllv8bu...@amer.company.com > > > (DEPRECATED:arcfour-hmac) > > > 3 07/29/2021 13:38:27 RestrictedKrbHost/nwpllv8bu...@amer.company.com > > > (aes128-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 RestrictedKrbHost/nwpllv8bu...@amer.company.com > > > (aes256-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 RestrictedKrbHost/ > > > nwpllv8bu100.amer.company....@amer.company.com (DEPRECATED:arcfour-hmac) > > > 3 07/29/2021 13:38:27 RestrictedKrbHost/ > > > nwpllv8bu100.amer.company....@amer.company.com (aes128-cts-hmac-sha1-96) > > > 3 07/29/2021 13:38:27 RestrictedKrbHost/ > > > nwpllv8bu100.amer.company....@amer.company.com (aes256-cts-hmac-sha1-96) > > > > > > I realize this is reflecting the literal entries in the /etc/krb5.keytab > > > file. So it appears that when this VM was born (on July 13th), it was > > > named zzzkbtdurbo18.amer.company.com. (I see other supporting evidence > > > for this). On or before July 29th, it was renamed to final FQDN > > > nwpllv8bu100.amer.company.com. /etc/hostname, /etc/hosts, > > > /etc/sysconfig/network etc were all updated and it was rejoined to AD. > > > > > > kinit -k works fine. It picks up the current hostname and apparently > > uses > > > host/nwpllv8bu100.amer.company....@amer.company.com as its service > > > principal. Since there's a valid entry in /etc/krb5.keytab file for > > this, > > > it uses this and all is good. (I'm guessing it uses the 14th or 15th > > > /etc/krb5.keytab file entry above.) > > > > > > sssd works, because it has this line: > > > > > > ldap_sasl_authid = host/nwpllv8bu100.amer.company....@amer.company.com > > > > > > But when sssd invokes adcli update to refresh the machine account > > password, > > > adcli update fails. > > > > > > Also, I see that adcli testjoin fails. > > > > > > [root@nwpllv8bu100 tmp]# adcli testjoin -D AMER.COMPANY.COM > > > adcli: couldn't connect to AMER.COMPANY.COM domain: Couldn't > > authenticate > > > as machine account: ZZZKBTDURBOL8: Preauthentication failed > > > [root@nwpllv8bu100 tmp]# > > > > > > From a strace of this adcli testjoin, it appears that adcli is opening > > the > > > /etc/krb5.keytab file to determine the default service principal to use > > and > > > is pulling the old server name. (instead of using the correct service > > > principal, as kinit -k somehow does.) > > > > Hi, > > > > it looks like your environment is a bit special. I guess you have added > > 'host/nwpllv8bu100.amer.company....@amer.company.com' to the > > 'userPrincipalName' LDAP attribute in the AD computer object for this > > host. > > > > # klist -k > > Keytab name: FILE:/etc/krb5.keytab > > KVNO Principal > > ---- > > -------------------------------------------------------------------------- > > 2 MASTER$@CHILD.AD.VM > > 2 MASTER$@CHILD.AD.VM > > 2 host/mas...@child.ad.vm > > 2 host/mas...@child.ad.vm > > 2 host/master.client...@child.ad.vm > > 2 host/master.client...@child.ad.vm > > 2 RestrictedKrbHost/mas...@child.ad.vm > > 2 RestrictedKrbHost/mas...@child.ad.vm > > 2 RestrictedKrbHost/master.client...@child.ad.vm > > 2 RestrictedKrbHost/master.client...@child.ad.vm > > # kdestroy -A > > # > > # > > # kinit -k > > kinit: Client 'host/master.client...@child.ad.vm' not found in Kerberos > > database while getting initial credentials > > # kinit -k 'MASTER$@CHILD.AD.VM' > > > > > > The reason is that 'kinit -k' constructs the principal by calling > > gethostname() or similar, adding the 'host/' prefix and the realm. But > > by default this principal in AD is only a service principal can cannot > > be used to request a TGT as kinit does. AD only allows user principals > > for request a TGT and this is by default 'SHORT$@AD.REALM'. If the > > userPrincipalName attribute is set, this principal given here is allowed > > as well. > > > > That's why adcli is checking the keytab for a principal with a '$' > > character and by default it uses the first it finds because it is > > expected there is only one. Adding some heuristics in case there are > > more '$' principals in the keytab like highest KVNO might help in some > > cases but would fail in other cases so I think just using the first is > > good enough. > > > > > > > > > > Maybe when sssd constructs this adcli update invocation, it's not passing > > > the ldap_sasl_authid, so the adcli update is doing the above logic to > > pull > > > the old server name? > > > > Adding a new option to adcli and using the value from ldap_sasl_authid > > might be a solution, I will think about it. > > > > HTH > > > > bye, > > Sumit > > > > > > > > Sounds like an adcli problem. Adcli should do as 'kinit -k' does when > > it's > > > passed no explicit service principal. Should dive into /etc/krb5.keytab > > > file and use the most recent set of entries (KVNO = 3 in above example). > > > Maybe derive the default service principal off the current FQDN and > > > Kerberos realm? > > > > > > Spike > > > PS As a general policy, we are not supposed to clone a VM and rename it > > to > > > another FQDN/IP address. I'll be trying to track down who did this and > > for > > > what reason. > > > > > > On Wed, Sep 1, 2021 at 10:08 AM Spike White <spikewhit...@gmail.com> > > wrote: > > > > > > > Ok, this is *very* illuminating! > > > > > > > > I see this in sssd_amer.company.com.log" > > > > > > > > (2021-09-01 3:44:46): [be[amer.company.com]] > > > > [ad_machine_account_password_renewal_done] (0x1000): --- adcli output > > > > start--- > > > > adcli: couldn't connect to amer.company.com domain: Couldn't > > authenticate > > > > as machine account: ZZZKBTDURBOL8: Preauthentication failed > > > > ---adcli output end--- > > > > > > > > However, I don't find that host name ZZZKBTDURBOL8 anywhere on the > > > > system. (By company convention, servers named ZZZ* are test servers > > that > > > > linux SEs spin up themselves). > > > > > > > > This server that's not renewing its creds is named: > > > > nwpllv8bu100.amer.company.com. it's a std dev server. in > > > > /etc/sssd/sssd.conf file, it has that as its sasl auth ID: > > > > > > > > [root@nwpllv8bu100 sssd]# grep sasl /etc/sssd/sssd.conf > > > > ldap_sasl_authid = host/nwpllv8bu100.amer.dell....@amer.company.com > > > > [root@nwpllv8bu100 sssd]# > > > > > > > > If I do 'kinit -k', the /etc/krb5.keytab file has that name as well: > > > > > > > > [root@nwpllv8bu100 sssd]# kinit -k > > > > [root@nwpllv8bu100 sssd]# klist > > > > Ticket cache: KCM:0 > > > > Default principal: host/nwpllv8bu100.amer.dell....@amer.company.com > > > > > > > > Valid starting Expires Service principal > > > > 09/01/2021 11:04:16 09/01/2021 21:04:16 krbtgt/ > > > > amer.dell....@amer.company.com > > > > renew until 09/08/2021 11:04:16 > > > > [root@nwpllv8bu100 sssd]# > > > > > > > > I searched /etc/sssd/sssd.conf -- no "zzz" or "ZZZ" string is anywhere > > in > > > > there. So where is sssd picking up this name ZZZKBTDURBOL8 and > > passing it > > > > to adcli update? > > > > > > > > Spike > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 1, 2021 at 2:46 AM Sumit Bose <sb...@redhat.com> wrote: > > > > > > > >> Am Tue, Aug 31, 2021 at 09:53:01PM +0200 schrieb Alexey Tikhonov: > > > >> > On Tue, Aug 31, 2021 at 6:47 PM Spike White <spikewhit...@gmail.com > > > > > > >> wrote: > > > >> > > > > >> > > All, > > > >> > > > > > >> > > OK we have a query we run in AD for machine account passwords for > > a > > > >> > > certain age. In today's run, 31 - 32 days. Then we verify it's > > > >> pingable. > > > >> > > > > > >> > > We have found such one such suspicious candidate today (two > > actually, > > > >> but > > > >> > > the other Linux server is quite sick). So one good research > > > >> candidate. > > > >> > > According to both AD and /etc/krb5.keytab file, the machine > > account > > > >> > > password was last set on 7/29. Today is 8/31, so that would be 32 > > > >> days. > > > >> > > This 'automatic machine account keytab renewal' background task > > > >> should > > > >> > > trigger again today. > > > >> > > > > > >> > > sssd service was last started 2 weeks ago and, by all appearances, > > > >> appears > > > >> > > healthy. sssctl domain-status <domain> shows online, connected > > to AD > > > >> > > servers (both domain and GC servers).. All logins and group > > > >> enumerations > > > >> > > working as expected. > > > >> > > > > > >> > > Just now, we dynamically set the debug level to 9 with 'sssctl > > > >> debug-level > > > >> > > 9'. This particular server is Oracle Linux 8.4, > > > >> > > running sssd-*-2.4.0-9.0.1.el8_4.1.x86_64. Installed July 13th, > > > >> 2021. So > > > >> > > -- very recent sssd version. (This problem occurs with both RHEL > > & OL > > > >> > > 6/7/8, it's just today's candidate happens to be OL8.) > > > >> > > > > > >> > > We can't keep debug level 9 up for a great many days; it swamps > > the > > > >> > > /var/log filesystem. But we can leave up for a few days. We > > > >> purposely did > > > >> > > not restart sssd server as we know that would trigger a machine > > > >> account > > > >> > > renewal. > > > >> > > > > > >> > > Speaking of that -- from Sumit's sssd source code in > > > >> > > ad_provider/ad_machine_pw_renewal.c, it appears that sssd is > > creating > > > >> a > > > >> > > back-end task to call external program /usr/sbin/adcli with > > certain > > > >> args. > > > >> > > What string can I look for in which sssd log file (now that I > > have > > > >> debug > > > >> > > level 9 enabled) to tell me when this 'adcli update' task (aka > > > >> 'automatic > > > >> > > machine account keytab renewal') is triggered? > > > >> > > > > > >> > > > > >> > It seems SSSD itself only logs in case of errors. I didn't find any > > > >> > explicit logs around `ad_machine_account_password_renewal_send()`. > > > >> > But perhaps there will be something like "[be_ptask_execute] > > (0x0400): > > > >> Task > > > >> > [AD machine account password renewal]: executing task" from generic > > > >> > be_ptask_* helpers in the sssd_$domain.log (I'm not sure). > > > >> > > > > >> > Also at this verbosity level `--verbose` should be supplied to adcli > > > >> itself > > > >> > and I guess output should be captured in sssd_$domain.log as well. > > I'm > > > >> not > > > >> > familiar with `adcli` internals, you can take a glance at > > > >> > https://gitlab.freedesktop.org/realmd/adcli to find its log > > messages. > > > >> > > > >> Hi, > > > >> > > > >> if SSSD's debug_level is 7 or higher the '--verbose' option is set > > > >> when calling adcli and the output is added to the backend logs. It > > will > > > >> start with log message "--- adcli output start---". > > > >> > > > >> HTH > > > >> > > > >> bye, > > > >> Sumit > > > >> > > > >> > > > > >> > > > > >> > > > > > >> > > I'm less certain now that we've surveyed our env that this > > background > > > >> > > 'adcli update' task is the reason behind 70 - 80 servers / month > > > >> dropping > > > >> > > off the domain. It might be a slight contributor, but I find > > only a > > > >> very > > > >> > > few pingable servers with machine account last renewal date > > between > > > >> 30 and > > > >> > > 40 days. > > > >> > > > > > >> > > Yes, I can disable this default 30 day automatic update and roll > > my > > > >> own > > > >> > > 'adcli update' cron. But that's a mass deployment, to fix what > > might > > > >> not > > > >> > > be the problem. I want to verify this is the actual culprit > > before > > > >> I take > > > >> > > those drastic steps. > > > >> > > > > > >> > > Spike > > > >> > > > > > >> > > > > > >> > > > >> > _______________________________________________ > > > >> > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > > >> > To unsubscribe send an email to > > sssd-users-le...@lists.fedorahosted.org > > > >> > Fedora Code of Conduct: > > > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > >> > List Guidelines: > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > >> > List Archives: > > > >> > > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > > > >> > Do not reply to spam on the list, report it: > > > >> https://pagure.io/fedora-infrastructure > > > >> _______________________________________________ > > > >> sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > > >> To unsubscribe send an email to > > sssd-users-le...@lists.fedorahosted.org > > > >> Fedora Code of Conduct: > > > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > >> List Guidelines: > > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > >> List Archives: > > > >> > > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > > > >> Do not reply to spam on the list, report it: > > > >> https://pagure.io/fedora-infrastructure > > > >> > > > > > > > > > _______________________________________________ > > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: > > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > > > Do not reply to spam on the list, report it: > > https://pagure.io/fedora-infrastructure > > _______________________________________________ > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > > Do not reply to spam on the list, report it: > > https://pagure.io/fedora-infrastructure > > > _______________________________________________ > sssd-users mailing list -- sssd-users@lists.fedorahosted.org > To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-le...@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
