Hi Fabian, Just eyeballing htop's RES column while emerge is running, the max value I see during "emerge -uDvpU --with-bdeps=y @world" increases from 272 MB to 380 MB, so it looks to be around 110 MB of extra memory usage on my system (with 1,094 total packages installed).
Chun-Yu On Sat, Jun 27, 2020, 12:35 AM Fabian Groffen <grob...@gentoo.org> wrote: > > Hi Chun-Yu, > > On 26-06-2020 23:34:12 -0700, Chun-Yu Shei wrote: > > Hi, > > > > I was recently interested in whether portage could be speed up, since > > dependency resolution can sometimes take a while on slower machines. > > After generating some flame graphs with cProfile and vmprof, I found 3 > > functions which seem to be called extremely frequently with the same > > arguments: catpkgsplit, use_reduce, and match_from_list. In the first > > two cases, it was simple to cache the results in dicts, while > > match_from_list was a bit trickier, since it seems to be a requirement > > that it return actual entries from the input "candidate_list". I also > > ran into some test failures if I did the caching after the > > mydep.unevaluated_atom.use and mydep.repo checks towards the end of the > > function, so the caching is only done up to just before that point. > > > > The catpkgsplit change seems to definitely be safe, and I'm pretty sure > > the use_reduce one is too, since anything that could possibly change the > > result is hashed. I'm a bit less certain about the match_from_list one, > > although all tests are passing. > > > > With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world" > > speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup. "emerge > > -ep @world" is just a tiny bit faster, going from 18.69 to 18.22 sec > > (2.5% improvement). Since the upgrade case is far more common, this > > would really help in daily use, and it shaves about 30 seconds off > > the time you have to wait to get to the [Yes/No] prompt (from ~90s to > > 60s) on my old Sandy Bridge laptop when performing normal upgrades. > > > > Hopefully, at least some of these patches can be incorporated, and please > > let me know if any changes are necessary. > > This sounds like a good job to me! Do you have any idea what the added > memory pressure for these changes are? > > Thanks, > Fabian > > > > Thanks, > > Chun-Yu > > > > > > > > -- > Fabian Groffen > Gentoo on a different level
