Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Sat, Apr 1, 2023 at 3:44 PM Alexander Ploumistos wrote: > > Something is glitching on my provider's side and it's impossible to > create new VMs. I've opened a ticket, but I doubt anyone is going to > deal with it before Monday. It has taken them over a month to (sort of) resolve the problem, but in any case, I was able to set up another VM to test. I can confirm that with the backported fix in 251.14-2.fc37, 2GB of RAM is more than enough to perform the upgrade. Sorry for the delay. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Fri, Mar 31, 2023 at 7:25 PM Alexander Ploumistos wrote: > > On Fri, Mar 31, 2023 at 7:09 PM Zbigniew Jędrzejewski-Szmek > wrote: > > > > I will submit systemd-tests-251.14-2.fc37 with the patch. > > I doubt that it solves the issue completely, but it shouldn't make > > it worse, so let's at least do this for now. > > Thank you, I see it's still being built in koji. I will spin a VM with > the same configuration over the weekend and try the upgrade again. Something is glitching on my provider's side and it's impossible to create new VMs. I've opened a ticket, but I doubt anyone is going to deal with it before Monday. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Fri, Mar 31, 2023 at 7:09 PM Zbigniew Jędrzejewski-Szmek wrote: > > I will submit systemd-tests-251.14-2.fc37 with the patch. > I doubt that it solves the issue completely, but it shouldn't make > it worse, so let's at least do this for now. Thank you, I see it's still being built in koji. I will spin a VM with the same configuration over the weekend and try the upgrade again. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Thu, Mar 30, 2023 at 02:37:16PM +0200, Alexander Ploumistos wrote: > On Thu, Mar 30, 2023 at 2:33 PM Zbigniew Jędrzejewski-Szmek > wrote: > > > > On Thu, Mar 30, 2023 at 02:25:33PM +0200, Alexander Ploumistos wrote: > > > On Thu, Mar 30, 2023 at 2:10 PM Zbigniew Jędrzejewski-Szmek > > > wrote: > > > > > > > > On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote: > > > > > On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos > > > > > wrote: > > > > > > > > > > > > Hello, > > > > > > > > > > > > TL;DR: > > > > > > DNF memory usage during upgrades from F37 to F38 on a couple of > > > > > > Fedora > > > > > > Cloud images (with 2 GB of RAM each) led to oomd kicking in and > > > > > > killing the upgrade process. It might be worth looking into before > > > > > > the > > > > > > final (also beta?) release. > > > > > > > > Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39. > > > > > > > > > Hi Zbigniew, is F37 getting that version too or are you talking about > > > a future F38 → F39 upgrade? > > > > That version — no. The patch for oomd config — yes (if it proves to be > > beneficial in F38/rawhide). > > Could you please bump this thread when it's done? I will submit systemd-tests-251.14-2.fc37 with the patch. I doubt that it solves the issue completely, but it shouldn't make it worse, so let's at least do this for now. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Thu, Mar 30, 2023 at 2:33 PM Zbigniew Jędrzejewski-Szmek wrote: > > On Thu, Mar 30, 2023 at 02:25:33PM +0200, Alexander Ploumistos wrote: > > On Thu, Mar 30, 2023 at 2:10 PM Zbigniew Jędrzejewski-Szmek > > wrote: > > > > > > On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote: > > > > On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos > > > > wrote: > > > > > > > > > > Hello, > > > > > > > > > > TL;DR: > > > > > DNF memory usage during upgrades from F37 to F38 on a couple of Fedora > > > > > Cloud images (with 2 GB of RAM each) led to oomd kicking in and > > > > > killing the upgrade process. It might be worth looking into before the > > > > > final (also beta?) release. > > > > > > Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39. > > > > > > Hi Zbigniew, is F37 getting that version too or are you talking about > > a future F38 → F39 upgrade? > > That version — no. The patch for oomd config — yes (if it proves to be > beneficial in F38/rawhide). Could you please bump this thread when it's done? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Thu, Mar 30, 2023 at 02:25:33PM +0200, Alexander Ploumistos wrote: > On Thu, Mar 30, 2023 at 2:10 PM Zbigniew Jędrzejewski-Szmek > wrote: > > > > On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote: > > > On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos > > > wrote: > > > > > > > > Hello, > > > > > > > > TL;DR: > > > > DNF memory usage during upgrades from F37 to F38 on a couple of Fedora > > > > Cloud images (with 2 GB of RAM each) led to oomd kicking in and > > > > killing the upgrade process. It might be worth looking into before the > > > > final (also beta?) release. > > > > Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39. > > > Hi Zbigniew, is F37 getting that version too or are you talking about > a future F38 → F39 upgrade? That version — no. The patch for oomd config — yes (if it proves to be beneficial in F38/rawhide). Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Thu, Mar 30, 2023 at 2:10 PM Zbigniew Jędrzejewski-Szmek wrote: > > On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote: > > On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos > > wrote: > > > > > > Hello, > > > > > > TL;DR: > > > DNF memory usage during upgrades from F37 to F38 on a couple of Fedora > > > Cloud images (with 2 GB of RAM each) led to oomd kicking in and > > > killing the upgrade process. It might be worth looking into before the > > > final (also beta?) release. > > Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39. Hi Zbigniew, is F37 getting that version too or are you talking about a future F38 → F39 upgrade? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Thu, Mar 30, 2023 at 12:08:03AM +0200, Fabio Valentini wrote: > On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos > wrote: > > > > Hello, > > > > TL;DR: > > DNF memory usage during upgrades from F37 to F38 on a couple of Fedora > > Cloud images (with 2 GB of RAM each) led to oomd kicking in and > > killing the upgrade process. It might be worth looking into before the > > final (also beta?) release. Please try with systemd-253.2-1.fc38 / systemd-253.2-1.fc39. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Wed, Mar 29, 2023, at 6:08 PM, Fabio Valentini wrote: > > I don't really want to throw money out the window just because DNF > eats up all the memory it can :( Everyone needs to internalize: This has nothing to do with DNF, really. It's about the *size of the repository metadata*. Every single time someone adds a new package into the single "Fedora" rpm-md repo, that makes clients going OOM a bit more likely. Every single time there's a new package, it makes it more likely that a client will hit a timeout downloading the repodata and fail to get that critical kernel security update. Even doing the obvious thing and splitting off e.g. a `fedora-devel` rpm-md repo (as is done kind of done in RHEL with CRB) that had all the -devel and Go/Rust etc. packages would likely create a huge amount of savings. Also as I said in an earlier thread, relating to SBCs but also small cloud instances - we also have image-based editions; these are Fedora systems too. rpm-ostree will not download the rpm-md by default, so is immune to this. And particularly after https://fedoraproject.org/wiki/Changes/OstreeNativeContainerStable - the UX for managing these is the same as podman/docker style application containers. You don't pay the cost of downloading the rpm-md on each client. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Thu, Mar 30, 2023 at 10:49:33AM +0200, Miroslav Suchý wrote: > Dne 30. 03. 23 v 0:08 Fabio Valentini napsal(a): > > Additionally, the smallest offerings of popular VPS providers have > > just 1 or 2 GB of RAM, is Fedora Server no longer supported on systems > > like these? > > At first place - upgrades in cloud environment should be discouraged. The > standard should be 1) describe your system in Ansible playbook 2) terminate > machine 3) change in your playbook the name of new image 4) run your > playbook. This is how we upgrade machines in cloud in fedora-infra. I don't think it is realistic to expect users to follow such steps in the general case as plenty of cloud VMs are considered precious and not simply able to be thrown away and replaced by freshly built system that way. > And not sure if it worth a work (likely not easy) as the upgrades from F39 > to F40 will be done by DNF5 with totally different (smaller I hope) > footprint. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
Dne 30. 03. 23 v 0:08 Fabio Valentini napsal(a): Additionally, the smallest offerings of popular VPS providers have just 1 or 2 GB of RAM, is Fedora Server no longer supported on systems like these? At first place - upgrades in cloud environment should be discouraged. The standard should be 1) describe your system in Ansible playbook 2) terminate machine 3) change in your playbook the name of new image 4) run your playbook. This is how we upgrade machines in cloud in fedora-infra. And not sure if it worth a work (likely not easy) as the upgrades from F39 to F40 will be done by DNF5 with totally different (smaller I hope) footprint. Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Thu, Mar 30, 2023 at 12:09 AM Fabio Valentini wrote: > > Additionally, the smallest offerings of popular VPS providers have > just 1 or 2 GB of RAM, is Fedora Server no longer supported on systems > like these? > Do we need to update the documentation for system requirements? Ping > cloud hosting providers that they shouldn't offer installing Fedora on > configurations with too little RAM? > There's also lots of SBCs with limited RAM - are Fedora users on these > devices supposed to reinstall Fedora every six months, since in-place > upgrades might no longer be possible? Another solution could be to tweak systemd-oomd not to touch dnf when running a system upgrade (perhaps through python3-dnf-plugin-system-upgrade). When I repeated the upgrade process with the second VM and I kept an eye out for memory usage, I saw that there were more than 300 MB available when oomd stepped in. Even installations requiring much larger transactions should be fine as long as there's a swap. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: DNF Sytem Upgrade requirements for an F37 → F38 upgrade
On Mon, Mar 6, 2023 at 4:09 AM Alexander Ploumistos wrote: > > Hello, > > TL;DR: > DNF memory usage during upgrades from F37 to F38 on a couple of Fedora > Cloud images (with 2 GB of RAM each) led to oomd kicking in and > killing the upgrade process. It might be worth looking into before the > final (also beta?) release. > > > The longer story: > I have a couple of hosted VMs, which started their lives as Fedora > Cloud 27 and which I have been upgrading to the next Fedora release > via dnf-plugin-system-upgrade. These VMs run very few things, mainly > some PHP applications on Apache and a good chunk of the PHP stack, > some scheduled network scripts, a private rpm repository, OpenVPN and > Wireguard nodes and when I am far from my main computer and koji or > copr happen to be busy, I test things in mock on them. These VMs are > near clones of each other, so usually only one of them is up at any > moment. > > I've never had an issue with memory use up to now and when the systems > aren't building packages, they seldom require more than 200 MB of RAM. > What's more, there's the default swap-on-zram, so they should be more > than happy memory-wise with 2 GB of RAM. > > I tried to upgrade one VM to F38, to see what problems I might face > and after the packages had been downloaded, I lost the ssh connection > as DNF was running the transaction test, right after a successful > transaction check. I repeated the whole thing a couple more times, > until I remembered that oomd is there and sure enough, DNF had been > consuming 1-1.3 GB until oomd decided to kill it. Well, actually it > killed my entire session and not just DNF. I tried running the upgrade > inside a screen (the GNU one) session and I came back to an empty > session. The transaction involved the upgrade of 1620 packages, with > a total size of 1.2 GB. I think the difference between that and the > previous upgrade F36 to F37) is around 100 MB and 30 or 40 packages. > In each of the failed attempts, the memory pressure was 55 to 78% for > more than 20 seconds, according to systemd-oomd. The exact same issues > appeared when upgrading the second VM. > > Eventually I did the upgrade by doubling the available RAM to 4 GB > (and then set it back to 2), but perhaps this could be a problem for > others, especially those who pay for cloud-based resources. Is DNF > supposed to be a legitimate target for systemd-oomd? Conversely, is > DNF supposed to use up so much memory? I know I'm late to the party, but I think this is still worth taking a look at. I have two small Fedora servers that used to run just fine with 1 GB of RAM, but I needed to up that to 2 GB for the upgrade to Fedora 36 (thanks to DNF running out of memory otherwise). Reading the original post, it sounds like I'll need to up that again soon, either for the upgrade to Fedora 37 or 38? I don't really want to throw money out the window just because DNF eats up all the memory it can :( Additionally, the smallest offerings of popular VPS providers have just 1 or 2 GB of RAM, is Fedora Server no longer supported on systems like these? Do we need to update the documentation for system requirements? Ping cloud hosting providers that they shouldn't offer installing Fedora on configurations with too little RAM? There's also lots of SBCs with limited RAM - are Fedora users on these devices supposed to reinstall Fedora every six months, since in-place upgrades might no longer be possible? Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
DNF Sytem Upgrade requirements for an F37 → F38 upgrade
Hello, TL;DR: DNF memory usage during upgrades from F37 to F38 on a couple of Fedora Cloud images (with 2 GB of RAM each) led to oomd kicking in and killing the upgrade process. It might be worth looking into before the final (also beta?) release. The longer story: I have a couple of hosted VMs, which started their lives as Fedora Cloud 27 and which I have been upgrading to the next Fedora release via dnf-plugin-system-upgrade. These VMs run very few things, mainly some PHP applications on Apache and a good chunk of the PHP stack, some scheduled network scripts, a private rpm repository, OpenVPN and Wireguard nodes and when I am far from my main computer and koji or copr happen to be busy, I test things in mock on them. These VMs are near clones of each other, so usually only one of them is up at any moment. I've never had an issue with memory use up to now and when the systems aren't building packages, they seldom require more than 200 MB of RAM. What's more, there's the default swap-on-zram, so they should be more than happy memory-wise with 2 GB of RAM. I tried to upgrade one VM to F38, to see what problems I might face and after the packages had been downloaded, I lost the ssh connection as DNF was running the transaction test, right after a successful transaction check. I repeated the whole thing a couple more times, until I remembered that oomd is there and sure enough, DNF had been consuming 1-1.3 GB until oomd decided to kill it. Well, actually it killed my entire session and not just DNF. I tried running the upgrade inside a screen (the GNU one) session and I came back to an empty session. The transaction involved the upgrade of 1620 packages, with a total size of 1.2 GB. I think the difference between that and the previous upgrade F36 to F37) is around 100 MB and 30 or 40 packages. In each of the failed attempts, the memory pressure was 55 to 78% for more than 20 seconds, according to systemd-oomd. The exact same issues appeared when upgrading the second VM. Eventually I did the upgrade by doubling the available RAM to 4 GB (and then set it back to 2), but perhaps this could be a problem for others, especially those who pay for cloud-based resources. Is DNF supposed to be a legitimate target for systemd-oomd? Conversely, is DNF supposed to use up so much memory? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
