Sumit, It took all day, but I finally got these RPMs on a test box:
libsss_simpleifp-2.4.0-9.el8_4.2sb1.x86_64 sssd-ipa-2.4.0-9.el8_4.2sb1.x86_64 sssd-client-2.4.0-9.el8_4.2sb1.x86_64 sssd-krb5-common-2.4.0-9.el8_4.2sb1.x86_64 libsss_idmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-common-2.4.0-9.el8_4.2sb1.x86_64 python3-sssdconfig-2.4.0-9.el8_4.2sb1.noarch sssd-common-pac-2.4.0-9.el8_4.2sb1.x86_64 sssd-ad-2.4.0-9.el8_4.2sb1.x86_64 libsss_certmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-dbus-2.4.0-9.el8_4.2sb1.x86_64 sssd-tools-2.4.0-9.el8_4.2sb1.x86_64 libipa_hbac-2.4.0-9.el8_4.2sb1.x86_64 libsss_nss_idmap-2.4.0-9.el8_4.2sb1.x86_64 sssd-kcm-2.4.0-9.el8_4.2sb1.x86_64 python3-sss-2.4.0-9.el8_4.2sb1.x86_64 samba-client-libs-4.13.3-5.el8_4 must be *brand new*; the latest we have in our mirrored repos is samba-client-libs-4.13.3-4.el8_4. Which caused no end of manual work and installing 2-3 of the above rpms with 'rpm -Uvh --nodeps'. Originally, all the bogus domains still appeared in sssctl domain-list and I was sad! But I realized they could have been cached from previously. Sure enough, when I shut down sssd and removed *all* cache: rm -rf /var/lib/sss/db/* rm -rf /var/lib/sss/mc/* Then when it started it found only the expected 5 AD domains. (1 parent + 4 regional child sub-domains). I'll let my fellow engineers poke at it in the next few days -- but I think it looks good. I think you've fixed it. Spike PS I git cloned your github.com source tree, checked out your ad_domain_filter branch and attempted to build according to build instructions. Using a RHEL7 test box since it's what a fellow engineer had handy. Using the build instructions. https://sssd.io/contrib/building-sssd.html It was an epic fail, even when I installed and enabled the devtoolset-9 SCL. I eventually generated binaries, but they installed into /usr/local/bin. I started to generate RPMs, but you had your RPMs ready for me by then and I switched over to a 8.4 test box, using your pre-build RPMs. Will the Fedora sssd build instructions on https://sssd.io/contrib/building-sssd.html ( dnf builddep sssd and following) work on RHEL8.4? On Fri, Oct 8, 2021 at 12:42 PM Sumit Bose <[email protected]> wrote: > Am Fri, Oct 08, 2021 at 08:51:55AM -0500 schrieb Spike White: > > Sumit, > > > > It would probably be faster for you to do a test build. I'd have to > fumble > > through pulling the source RPM, rpmbuild invocation, rpm install. You > > probably know those commands at your fingertips. > > > > We have ~20K servers with RHEL7, RHEL8, OL7 (RHCK) and OL8 (RHCK) > > exhibiting this behavior. All x86_64. We have test servers of each of > > those flavors on which we can test. Your call. > > > > We have beaucoup RHEL8/OL8 test boxes, so if that's convenient for you, > > it'll work for us. > > Hi, > > please have a look at > https://sbose.fedorapeople.org/sssd-2.4.0-9.el8_4.2sb1.tar.gz, a test > build for RHEL-8.4. You do not have to install all packages, calling > 'yum update *' in the new directory should just update the installed > packages. > > bye, > Sumit > > > > > It is super-easy for us to determine.if it's fixed or not. Previously > > 'sssctl domain-list' only showed the 5 trusted domains. Now with this > new > > sssd version (~July), 'sssctl domain-list' shows the expected 5 trusted > > domains and the 14 untrusted domains. > > > > Spike > > > > On Fri, Oct 8, 2021 at 1:01 AM Sumit Bose <[email protected]> wrote: > > > > > Am Thu, Oct 07, 2021 at 11:38:54AM -0500 schrieb Spike White: > > > > All (but particularly Sumit since he wrote the comments on > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1984591), > > > > > > > > > > Hi, > > > > > > jfyi, I'm currently working on a fix for this to filter out domains > from > > > other forests and untrusted domains. My WIP branch is at > > > https://github.com/sumit-bose/sssd/tree/ad_filter_domains. Can you do > a > > > test build of SSSD based on this or shall I try to create a test build > > > for you? For the latter, please tell me for which platform. > > > > > > bye, > > > Sumit > > > > > > > > > > > > > > > There are at least two problems created by this recently-introduced > sssd > > > > bug. One problem is solvable by the suggested work-around, the > other is > > > > not. The work-around suggested is: > > > > > > > > [domain/name.of.joined.domain] > > > > > > > > ad_enabled_domains = dom1.example.com, dom2.example.com, > > > > dom3.example.com > > > > > > > > In order to query only the desired AD domains. > > > > > > > > > > > > > > > > What is the bug? > > > > > > > > the sssd-ad man page says "The AD provider can be used to get user > > > > information and authenticate users from trusted domains. Currently > > > > only trusted domains in the same forest are recognized.". > > > > > > > > What is happening is that untrusted AD domains are being > discovered. A > > > > very specific type of untrusted domains. When the joined domain > has no > > > > trust with that other domain, but that other domain trusts the > original > > > > domain – that is a one-way trust (the wrong way). To the joined > domain, > > > > this is an untrusted domain and should not be discovered. > > > > > > > > This is actually very common in corporate environments. > > > > > > > > You may have a main AD domain, call it CORP.COMPANY.COM. Then for > > > testing > > > > and new production evaluation, you might have a test AD domain called > > > > LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled, with > full > > > > audits and corporate security. LAB-TEST.COMPANY.COM is a test AD > > > domain – > > > > it’s the wild, wild west! > > > > > > > > So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that > users > > > can > > > > log into this test domain with their CORP accounts). But > > > CORP.COMPANY.COM > > > > does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the > wild, > > > > wild west, doing so would compromise corporate security.) > > > > > > > > Thus, a server joined to domain CORP.COMPANY.COM should discover > > > > CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It > > > should > > > > *NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not > trust > > > > this domain. > > > > > > > > A server joined to LAB-TEST.COMPANY.COM should discover > > > LAB-TEST.COMPANY.COM > > > > and all domains trusted by LAB-TEST.COMPANY.COM. Including > > > CORP.COMPANY.COM, > > > > as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM. > > > > > > > > The bug is that a server joined to CORP.COMPANY.COM discovers > > > > LAB-TEST.COMPANY.COM, which it shouldn’t. > > > > > > > > > > > > > > > > What problems does this cause? > > > > > > > > Two problems. > > > > > > > > 1. Many of these untrusted discovered “lab” domains are > accessible > > > > only to specific network locations. That is, they’re firewalled off > to a > > > > particular lab. So sssd attempts to query these inaccessible AD > domains > > > > and takes a long time to time out. This problem can be worked > around by > > > > the suggested work-around in the Bugzilla: > > > > > > > > > > > > > > > > [domain/corp.company.com] > > > > > > > > ad_enabled_domains = corp.company.com > > > > > > > > > > > > > > > > So then, while LAB-TEST.COMPANY.COM is still erroneously discovered, > > > it is > > > > no longer searched. Sssd is again fast. > > > > > > > > > > > > > > > > 2. Bogus messages in /var/log/sssd_nss.log file. Even with no > > > debug > > > > level set in the [nss] stanza, these error messages appear multiple > > > times a > > > > second. It quickly fills up the /var/log filesystem. > > > > > > > > [root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data Provider > > > > returned an error" > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > > > > > > > > > From debug level 9, it is clear that this is arising from a query of > > > these > > > > erroneously-discovered untrusted domains. Here’s an example of one > > > > instance of above with debug level 9 turned on. So > > > > emeaicmd.geodll.company.com is one of these erroneously-discovered > > > > untrusted lab domains, that happens to be firewalled off from this > > > > particular AD client: > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply > from > > > > Data Provider - DP error code: 0 errno: 0 error message: Success > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR > #9: > > > > Looking up [[email protected]] in cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR > #9: > > > > Object [[email protected]] was not found in cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] > > > > (0x0400): CR #9: Adding [[email protected]] to negative cache > > > > > > > > (2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User > > > > [email protected] is a local user > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding > > > > [NCE/USER/company.com/[email protected]] to negative cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] > (0x2000): > > > > Request type POSIX-only for domain EMEAICMD.geodll.company.com type > > > POSIX > > > > is valid > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9: > > > Using > > > > domain [EMEAICMD.geodll.company.com] > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] > (0x0400): CR > > > > #9: Preparing input data for domain [EMEAICMD.geodll.company.com] > rules > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #9: > > > > Looking up [email protected] > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR > #9: > > > > Checking negative cache for [[email protected]] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): > Checking > > > > negative cache for [NCE/USER/ > > > > EMEAICMD.geodll.company.com/[email protected]] > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR > #9: [ > > > > [email protected]] is not present in negative cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR > #9: > > > > Looking up [[email protected]] in cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR > #9: > > > > Object [[email protected]] was not found in cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9: > > > Looking > > > > up [[email protected]] in data provider > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing > > > > request for [0x564d6be36a70:3:[email protected]@ > > > > EMEAICMD.geodll.company.com] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400): > Creating > > > > request for [EMEAICMD.geodll.company.com > > > > ][0x3][BE_REQ_INITGROUPS][[email protected]:-] > > > > > > > > (2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000): > 0x564d6ccd6670 > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400): > > > Entering > > > > request [0x564d6be36a70:3:[email protected]@ > > > > EMEAICMD.geodll.company.com] > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR > #12: > > > > Looking up [[email protected]] in cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR > #12: > > > > Object [[email protected]] was not found in cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain] > > > > (0x0400): CR #12: Adding [[email protected]] to negative cache > > > > > > > > (2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User > > > > [email protected] is a local user > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding > > > > [NCE/USER/company.com/[email protected]] to negative cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] > (0x2000): > > > > Request type POSIX-only for domain EMEAICMD.geodll.company.com type > > > POSIX > > > > is valid > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #12: > > > Using > > > > domain [EMEAICMD.geodll.company.com] > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] > (0x0400): CR > > > > #12: Preparing input data for domain [EMEAICMD.geodll.company.com] > rules > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR > #12: > > > > Looking up [email protected] > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR > #12: > > > > Checking negative cache for [[email protected]] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): > Checking > > > > negative cache for [NCE/USER/ > > > > EMEAICMD.geodll.company.com/[email protected]] > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR > #12: > > > [ > > > > [email protected]] is not present in negative cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR > #12: > > > > Looking up [[email protected]] in cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR > #12: > > > > Object [[email protected]] was not found in cache > > > > > > > > (2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12: > > > > Looking up [[email protected]] in data provider > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing > > > > request for [0x564d6be36a70:3:[email protected]@ > > > > EMEAICMD.geodll.company.com] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): > Identical > > > > request in progress: [ > 0x564d6be36a70:3:[email protected] > > > @ > > > > EMEAICMD.geodll.company.com] > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400): > Deleting > > > > request: [0x564d6be36a70:3:[email protected]@company.com] > > > > > > > > (2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000): > > > 0x564d6ccd6670 > > > > > > > > (2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn: > > > > 0x564d6ccc9300 > > > > > > > > (2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching. > > > > > > > > (2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data > > > Provider > > > > returned an error [org.freedesktop.DBus.Error.Failed] > > > > > > > > > > > > > > > > The suggested work-around does not resolve problem #2. > > > > > > > > BTW, here is a listing of the domains discovered on that sssd client: > > > > > > > > [root@auspdfdlobv01 ~]# sssctl domain-list > > > > > > > > amer.company.com > > > > > > > > company.com > > > > > > > > japn.company.com > > > > > > > > emea.company.com > > > > > > > > apac.company.com > > > > > > > > EMEAICMD.geodll.company.com > > > > > > > > geodll.company.com > > > > > > > > EMEAICM.GEODLL.COMPANY.COM > > > > > > > > alienware.com > > > > > > > > corp.svcs > > > > > > > > perotsystems.net > > > > > > > > companyservices.dmz > > > > > > > > Beer.Town > > > > > > > > production.online.company.com > > > > > > > > jp-poclab.companypoc.com > > > > > > > > emea-poclab.companypoc.com > > > > > > > > oldev.preol.company.com > > > > > > > > olqa.preol.company.com > > > > > > > > ap-poclab.companypoc.com > > > > > > > > [root@auspdfdlobv01 ~]# > > > > > > > > > > > > > > > > This sssd client is joined to amer.company.com, so the only trusted > > > domains > > > > are the first 5. The parent domain and the 4 regional domains. > All > > > > those other domains below that are untrusted domains. More > specifically, > > > > they trust company.com, but company.com does not trust them. (one > way > > > > trust – the wrong way.) Some look like the real wild wild west > > > (Beer.Town > > > > ?). > > > > > > > > > > > > > > > > Spike > > > > > > > _______________________________________________ > > > > sssd-users mailing list -- [email protected] > > > > To unsubscribe send an email to > [email protected] > > > > Fedora Code of Conduct: > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > > List Archives: > > > > https://lists.fedorahosted.org/archives/list/[email protected] > > > > Do not reply to spam on the list, report it: > > > https://pagure.io/fedora-infrastructure > > > _______________________________________________ > > > sssd-users mailing list -- [email protected] > > > To unsubscribe send an email to > [email protected] > > > Fedora Code of Conduct: > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: > > > > https://lists.fedorahosted.org/archives/list/[email protected] > > > Do not reply to spam on the list, report it: > > > https://pagure.io/fedora-infrastructure > > > > > > _______________________________________________ > > sssd-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > _______________________________________________ > sssd-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/[email protected] > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure >
_______________________________________________ sssd-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/[email protected] Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
