Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
On Fri, 2023-08-04 at 11:42 +0100, Adam Williamson wrote: > > > > This is a big deal, because even users who don't use modularity at all (but > > have not uninstalled fedora-repos-modular) will not be able to upgrade to > > Fedora 39+ without reaching for help. > > > > Adam outlined 3 options to solve this problem in the bugzilla where he > > reported > > this: https://bugzilla.redhat.com/2228827 > > So an update to this, thanks to Miro for double-checking me: I had > forgotten that the openQA tests edit the dnf config to point to the > compose tree (in order to make sure we're testing the right thing - > there's an ordering problem if we just test the actual 'rawhide' > location on the mirror system, it might not have been synced by the > time the tests run). > > It looks like the public 'rawhide' location *does* still have a Modular > tree: > > https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/ > > but there's still a problem there, because...it's now just stale data. > That is the Modular tree from the 20230802.n.0 compose, and unless > someone does something about it, it always *will* be. Keeping the last > set of modular repos frozen in amber forever doesn't seem like the best > permanent situation :) > > So this is still an issue, but at least it doesn't actually immediately > break upgrades to Rawhide using the default dnf config. I'll downgrade > the bug severity appropriately. Update to the update: now F39 has branched, this does indeed look broken. Upgrades from F38 to F39 (or F37 to F39) fail because of the non-existent modular repos: we get 404s from mirrormanager trying to find 'fedora-modular-39' in the repo set, which doesn't exist. And there is no "Modular" tree at https://dl.fedoraproject.org/pub/fedora/linux/development/39/ currently. I will update the bug. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
On Fri, 2023-08-04 at 11:13 -0400, Stephen Smoogen wrote: > 2. We need to work out 1-2 methods we 'support' for upgrading which of the > ones below I would say D and A where certain tooling will check to see if > the upgrade is possible and then alert if it isn't and try a method of > turning things off if ok. Well, this is, effectively, exactly already the case. We already have specific supported methods of upgrading: https://docs.fedoraproject.org/en-US/quick-docs/upgrading/ (basically, graphical upgrade for Workstation; dnf system-upgrade for everything else; both using the default stock repository configuration). The tooling that checks to see if the upgrade is possible and alert if it isn't is...well...the upgrade process. It does that. The issue as described affected the "supported" methods of upgrading, except that (as described in the correction) it doesn't actually cause immediate issues in real-world cases because of the 'stale' tree that remains on the official repo location. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
On Fri, 4 Aug 2023 at 06:16, Miro Hrončok wrote: > Hello folks, > > With the retirement of modularity, the modular dnf repositories for Fedora > 39 > no longer exist. However, this will introduce a problem during upgrades. > When > users try to upgrade from previous Fedora releases with > fedora-repos-modular > installed, they will hit fatal errors that will probably look like this: > > Errors during downloading metadata for repository 'fedora-modular': >- Status code: 404 for > > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) >- Status code: 404 for > > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) >- Status code: 404 for > > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) > Error: Failed to download metadata for repo 'fedora-modular': Cannot > prepare > internal mirrorlist: Status code: 404 for > > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) > > Or: > > Error: Failed to download metadata for repository 'fedora-modular': Cannot > download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were > tried > > (The actual error might differ depending on the exact state of the removal > of > the modular repos and their mirrorlist etc.) > > > This is caused by the combination of the following facts: > > - the modular repo configuration in Fedora 37/38 has > skip_if_unavailable=False > - when the releasever is set to 39, the URLs of the repos give error 404 > > This is a big deal, because even users who don't use modularity at all > (but > have not uninstalled fedora-repos-modular) will not be able to upgrade to > Fedora 39+ without reaching for help. > > Adam outlined 3 options to solve this problem in the bugzilla where he > reported > this: https://bugzilla.redhat.com/2228827 > > I slept on it and have found a fourth option (but I don't think it's very > viable). I'll present all 4 options here: > > I think we are going to need to do multiple of these solutions because there are too many ways people upgrade their systems (usually because someone told them there method would work years ago or they just found a doc which was written years ago, etc). As Petr Pisar points out elsewhere this is not a new problem. We regularly see someone on IRC for years asking why an upgrade didn't work and it comes down to something like skip_if_unavailable=true or some similar issue where a repository is not properly set up (say a copr or third party repo) for the current release. 1. We need to document that this failure could and can occur. We need to point out that we are only testing N methods for upgrades and anything else will need manual work arounds. The way to maybe find those will be to ask on . 2. We need to work out 1-2 methods we 'support' for upgrading which of the ones below I would say D and A where certain tooling will check to see if the upgrade is possible and then alert if it isn't and try a method of turning things off if ok. [If that is even possible with code paths not yet written.] 3. 4. Profit. I don't like B because it could be the most fragile as it will need work in pungi to make sure that the repositories are ALWAYS properly created in a way that doesn't cause problems. [They can't just be 'empty'. They need to be empty with repodata that allows for updates to always work. If this requires hacking pungi then any upgrade of the source code on the releng boxes could cause breakage weeks later or releases later.] It is also hard to ever turn this off. As much as we like to have people upgrade from 36->37 or 36->38, there is a large enough 'hump' of people who do a 36->40 turn that means we would be doing this til maybe 44. Of course if it is just a simple pungi config which doesn't make other things break, B may be the easiest.. > > A. Set skip_if_unavailable=True in the modular repo configuration on old > Fedora > > > > B. Mirror empty modular repositories for the time being > > > > C. Document this in the upgrade documentation > > > > D. Change our upgrade tools to disable modular repos on upgrades to F39+ > > > > > What do you think we should do? My preference would be to go with option > (B) > and fallback to option (A) if (B) isn't finished in reasonable time. > > > -- > Miro Hrončok > -- > Phone: +420777974800 > IRC: mhroncok > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue > -- Stephen Smoogen, Red Hat Automotive
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
V Fri, Aug 04, 2023 at 12:16:14PM +0200, Miro Hrončok napsal(a): > With the retirement of modularity, the modular dnf repositories for Fedora > 39 no longer exist. However, this will introduce a problem during upgrades. > When users try to upgrade from previous Fedora releases with > fedora-repos-modular installed, they will hit fatal errors that will > probably look like this: > > Errors during downloading metadata for repository 'fedora-modular': > - Status code: 404 for > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) > - Status code: 404 for > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) > - Status code: 404 for > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) > Error: Failed to download metadata for repo 'fedora-modular': Cannot prepare > internal mirrorlist: Status code: 404 for > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) > > Or: > > Error: Failed to download metadata for repository 'fedora-modular': Cannot > download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were > tried > > (The actual error might differ depending on the exact state of the removal > of the modular repos and their mirrorlist etc.) > > > This is caused by the combination of the following facts: > > - the modular repo configuration in Fedora 37/38 has > skip_if_unavailable=False > - when the releasever is set to 39, the URLs of the repos give error 404 > This is a general problem of removing a repository from the distribution. Unrelated to Modularity. I propose this procedure: Keep Fedora <= 38 as it is. For Fedora 39 and 40 keep an empty repository on mirrors and uninstall the repository configuration by obsoleting fedora-repos-modular. Then in Fedora 41 remove the repository from mirrors. That way users of currently relased Fedoras <= 38 will have access to their modules, and users upgrading to Fedora 39 or 40 won't get the 404 HTTP error. After this upgrade to 39 or 40 the repository definition became nonexistant thanks to the obsolete. Fedora >= 41 can stop mirroring the repository because upgrade guidelines only support jump to N+1 and N+2 distribution release. -- Petr signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
Once upon a time, Miro Hrončok said: > On 04. 08. 23 14:45, Chris Adams wrote: > >Could the repo config be removed by having fedora-repos Obsolete: > >fedora-repos-moduler <= 39? > > That has already been done. But when the upgrade happens, the repo > config is still there, to be removed in the transaction that fails. Ah, that makes sense, sorry. :) -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
On 04. 08. 23 14:45, Chris Adams wrote: Once upon a time, Miro Hrončok said: With the retirement of modularity, the modular dnf repositories for Fedora 39 no longer exist. However, this will introduce a problem during upgrades. When users try to upgrade from previous Fedora releases with fedora-repos-modular installed, they will hit fatal errors that will probably look like this: Could the repo config be removed by having fedora-repos Obsolete: fedora-repos-moduler <= 39? That has already been done. But when the upgrade happens, the repo config is still there, to be removed in the transaction that fails. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
Once upon a time, Miro Hrončok said: > With the retirement of modularity, the modular dnf repositories for > Fedora 39 no longer exist. However, this will introduce a problem > during upgrades. When users try to upgrade from previous Fedora > releases with fedora-repos-modular installed, they will hit fatal > errors that will probably look like this: Could the repo config be removed by having fedora-repos Obsolete: fedora-repos-moduler <= 39? -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
On Fri, Aug 04, 2023 at 12:25:34PM +0100, Adam Williamson wrote: > On Fri, 2023-08-04 at 12:16 +0100, Daniel P. Berrangé wrote: > > > > > > It looks like the public 'rawhide' location *does* still have a Modular > > > tree: > > > > > > https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/ > > > > > > but there's still a problem there, because...it's now just stale data. > > > That is the Modular tree from the 20230802.n.0 compose, and unless > > > someone does something about it, it always *will* be. Keeping the last > > > set of modular repos frozen in amber forever doesn't seem like the best > > > permanent situation :) > > > > Do we still have the problem where dnf will preferrentially pick > > content from a modular repo, even if it has older NEVR than the > > same package name in a non-modular repo ? > > > > IOW, would the existance of this stale modular content, prevent > > the upgrade tools from correctly bringing in content from the > > new release ? > > Well, I think only if you have a module explicitly enabled. We dropped > the whole "ooh, if this is in a module, pull it in and enable the > module!" thing aaages ago, it only lasted like one release I think. > IIRC, anyway. Ok, so for most people it should be fairly harmless having the stale content in the modular repo. During upgrade dnf will look at the repo and ignore all the outdated RPMs. Them presumably something will have an "Obsoletes: fedora-modular-repos" so the repo config files all get erased during the upgrade, and thus the modular content will never be consulted thereafter. IOW, just leavnig the stale repo there forever wouldn't look to be terrible, though from a conceptual POV it would be nicer to just have an empty repo at the URL instead. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
On Fri, 2023-08-04 at 12:16 +0100, Daniel P. Berrangé wrote: > > > > It looks like the public 'rawhide' location *does* still have a Modular > > tree: > > > > https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/ > > > > but there's still a problem there, because...it's now just stale data. > > That is the Modular tree from the 20230802.n.0 compose, and unless > > someone does something about it, it always *will* be. Keeping the last > > set of modular repos frozen in amber forever doesn't seem like the best > > permanent situation :) > > Do we still have the problem where dnf will preferrentially pick > content from a modular repo, even if it has older NEVR than the > same package name in a non-modular repo ? > > IOW, would the existance of this stale modular content, prevent > the upgrade tools from correctly bringing in content from the > new release ? Well, I think only if you have a module explicitly enabled. We dropped the whole "ooh, if this is in a module, pull it in and enable the module!" thing aaages ago, it only lasted like one release I think. IIRC, anyway. But yes, if you have any modules enabled, the experience is going to be bad. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
On Fri, Aug 04, 2023 at 11:42:07AM +0100, Adam Williamson wrote: > On Fri, 2023-08-04 at 12:16 +0200, Miro Hrončok wrote: > > Hello folks, > > > > With the retirement of modularity, the modular dnf repositories for Fedora > > 39 > > no longer exist. However, this will introduce a problem during upgrades. > > When > > users try to upgrade from previous Fedora releases with > > fedora-repos-modular > > installed, they will hit fatal errors that will probably look like this: > > > > Errors during downloading metadata for repository 'fedora-modular': > >- Status code: 404 for > > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > > > > (IP: ...) > >- Status code: 404 for > > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > > > > (IP: ...) > >- Status code: 404 for > > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > > > > (IP: ...) > > Error: Failed to download metadata for repo 'fedora-modular': Cannot > > prepare > > internal mirrorlist: Status code: 404 for > > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > > > > (IP: ...) > > > > Or: > > > > Error: Failed to download metadata for repository 'fedora-modular': Cannot > > download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were > > tried > > > > (The actual error might differ depending on the exact state of the removal > > of > > the modular repos and their mirrorlist etc.) > > > > > > This is caused by the combination of the following facts: > > > > - the modular repo configuration in Fedora 37/38 has > > skip_if_unavailable=False > > - when the releasever is set to 39, the URLs of the repos give error 404 > > > > This is a big deal, because even users who don't use modularity at all (but > > have not uninstalled fedora-repos-modular) will not be able to upgrade to > > Fedora 39+ without reaching for help. > > > > Adam outlined 3 options to solve this problem in the bugzilla where he > > reported > > this: https://bugzilla.redhat.com/2228827 > > So an update to this, thanks to Miro for double-checking me: I had > forgotten that the openQA tests edit the dnf config to point to the > compose tree (in order to make sure we're testing the right thing - > there's an ordering problem if we just test the actual 'rawhide' > location on the mirror system, it might not have been synced by the > time the tests run). > > It looks like the public 'rawhide' location *does* still have a Modular > tree: > > https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/ > > but there's still a problem there, because...it's now just stale data. > That is the Modular tree from the 20230802.n.0 compose, and unless > someone does something about it, it always *will* be. Keeping the last > set of modular repos frozen in amber forever doesn't seem like the best > permanent situation :) Do we still have the problem where dnf will preferrentially pick content from a modular repo, even if it has older NEVR than the same package name in a non-modular repo ? IOW, would the existance of this stale modular content, prevent the upgrade tools from correctly bringing in content from the new release ? With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Removal of Modular repos broke upgrades to Fedora 39: What now?
On Fri, 2023-08-04 at 12:16 +0200, Miro Hrončok wrote: > Hello folks, > > With the retirement of modularity, the modular dnf repositories for Fedora 39 > no longer exist. However, this will introduce a problem during upgrades. When > users try to upgrade from previous Fedora releases with fedora-repos-modular > installed, they will hit fatal errors that will probably look like this: > > Errors during downloading metadata for repository 'fedora-modular': >- Status code: 404 for > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) >- Status code: 404 for > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) >- Status code: 404 for > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) > Error: Failed to download metadata for repo 'fedora-modular': Cannot prepare > internal mirrorlist: Status code: 404 for > https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39=x86_64 > (IP: ...) > > Or: > > Error: Failed to download metadata for repository 'fedora-modular': Cannot > download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were > tried > > (The actual error might differ depending on the exact state of the removal of > the modular repos and their mirrorlist etc.) > > > This is caused by the combination of the following facts: > > - the modular repo configuration in Fedora 37/38 has > skip_if_unavailable=False > - when the releasever is set to 39, the URLs of the repos give error 404 > > This is a big deal, because even users who don't use modularity at all (but > have not uninstalled fedora-repos-modular) will not be able to upgrade to > Fedora 39+ without reaching for help. > > Adam outlined 3 options to solve this problem in the bugzilla where he > reported > this: https://bugzilla.redhat.com/2228827 So an update to this, thanks to Miro for double-checking me: I had forgotten that the openQA tests edit the dnf config to point to the compose tree (in order to make sure we're testing the right thing - there's an ordering problem if we just test the actual 'rawhide' location on the mirror system, it might not have been synced by the time the tests run). It looks like the public 'rawhide' location *does* still have a Modular tree: https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/ but there's still a problem there, because...it's now just stale data. That is the Modular tree from the 20230802.n.0 compose, and unless someone does something about it, it always *will* be. Keeping the last set of modular repos frozen in amber forever doesn't seem like the best permanent situation :) So this is still an issue, but at least it doesn't actually immediately break upgrades to Rawhide using the default dnf config. I'll downgrade the bug severity appropriately. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
