Good to know! In any case, memory consumption should improve further soon. On the big system w/ 300k RPMs and a bunch of errata, etc, we were using something like 20gb by the end of the migration, and it dropped to <1gb after applying these fixes.
On Thu, Nov 5, 2020 at 9:34 AM Winberg Adam <[email protected]> wrote: > both! After fixing that script the memory usage is very manageable. > > > //Adam > > > ------------------------------ > *From:* Daniel Alley <[email protected]> > *Sent:* 05 November 2020 15:30 > *To:* Winberg Adam > *Cc:* [email protected] > *Subject:* Re: [Pulp-list] 2to3 migration: memory usage and open file > handles > > Do you mean that the bash script was the cause of the file descriptor > issue, or the memory issue, or both? > > On Thu, Nov 5, 2020 at 3:25 AM Winberg Adam <[email protected]> wrote: > >> sorry, ignore my problem. It was totally unrelated to pulp, it was >> another process going haywire at the same time as I was running the >> migration (bug in bash while loop in an unrelated housekeeping script). I >> was so focused on the migration that I didn't even notice that the issue >> was elsewhere. >> >> >> //Adam >> >> >> ------------------------------ >> *From:* [email protected] <[email protected]> on >> behalf of Winberg Adam <[email protected]> >> *Sent:* 05 November 2020 07:42 >> *To:* Daniel Alley >> *Cc:* [email protected] >> *Subject:* Re: [Pulp-list] 2to3 migration: memory usage and open file >> handles >> >> >> thats weird. I had a lot of memory usage while I was running on pulpcore >> 3.4, but after upgrading to 3.7 there was hardly any memory usage. >> >> >> I will reboot and run with 2 workers instead of 4 (don't know if that is >> even relevant in migration) and run a new migration from scratch before >> filing an issue. >> >> >> //Adam >> >> >> ------------------------------ >> *From:* Daniel Alley <[email protected]> >> *Sent:* 05 November 2020 01:58 >> *To:* Winberg Adam >> *Cc:* [email protected] >> *Subject:* Re: [Pulp-list] 2to3 migration: memory usage and open file >> handles >> >> Hi Adam, >> >> We discovered (and fixed) some memory leaks in a library that we are >> using [0] [1], which happens to be a Python extension written in C. Right >> now we're still waiting on the maintainers of that library to review the >> changes before we package them and ship the RPM, but we'll definitely let >> you know what that happens. >> >> However, these issues would have affected every previous version equally, >> so it's a little strange that you're only running into it now. Nothing >> else about your setup has changed I assume? >> >> re: file descriptors, we've been testing migrating much larger systems >> (300k RPMs, 600k errata) without running into problems, so I'm perplexed >> about what could be causing that. File an issue and list which >> repositories you're attempting to migrate and we'll see if we can reproduce. >> >> >> [0] https://github.com/rpm-software-management/createrepo_c/pull/231 >> [1] https://github.com/rpm-software-management/createrepo_c/pull/233 >> >> On Wed, Nov 4, 2020 at 12:40 PM Winberg Adam <[email protected]> >> wrote: >> >>> Hi, >>> >>> >>> running a 2to3 migration with 2to3-migration-0.5.1 seems to consume a >>> whole lot more memory than previous versions. My 12G RAM was quickly spent, >>> i increased to 16G which wasnt enough either. Earlier migrations with 0.5.0 >>> didnt spend anywhere near that amount. >>> >>> >>> Also - the migration fails with >>> >>> OSError: [Errno 23] Too many open files in system: .. >>> >>> >>> The memory usage increases while running the 'Migrating rpm content to >>> Pulp 3 rpm' subtask. With 16G RAM I only get to about 114000/152000 pkgs in >>> that task before the memory is more or less all consumed and the OSError >>> appears. >>> >>> >>> So it seems to me that there is some type of regression here. Any >>> pointers on how I can further debug or work around this? >>> >>> >>> This is on RHEL8 with >>> >>> python3-pulp-rpm-3.7.0-1.el8.noarch >>> python3-pulpcore-3.7.3-1.el8.noarch >>> python3-pulp-2to3-migration-0.5.1-1.el8.noarch >>> >>> //Adam >>> >>> _______________________________________________ >>> Pulp-list mailing list >>> [email protected] >>> https://www.redhat.com/mailman/listinfo/pulp-list >> >>
_______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
