Dnia June 28, 2020 3:42:33 AM UTC, Zac Medico <zmed...@gentoo.org> napisał(a):
>On 6/27/20 8:12 PM, Michał Górny wrote:
>> Dnia June 28, 2020 3:00:00 AM UTC, Zac Medico <zmed...@gentoo.org>
>>> On 6/26/20 11:34 PM, Chun-Yu Shei wrote:
>>>> Hi,
>>>> I was recently interested in whether portage could be speed up,
>>>> dependency resolution can sometimes take a while on slower
>>>> After generating some flame graphs with cProfile and vmprof, I
>>> 3
>>>> functions which seem to be called extremely frequently with the
>>>> arguments: catpkgsplit, use_reduce, and match_from_list.  In the
>>> first
>>>> two cases, it was simple to cache the results in dicts, while
>>>> match_from_list was a bit trickier, since it seems to be a
>>> requirement
>>>> that it return actual entries from the input "candidate_list".  I
>>> also
>>>> ran into some test failures if I did the caching after the
>>>> mydep.unevaluated_atom.use and mydep.repo checks towards the end of
>>> the
>>>> function, so the caching is only done up to just before that point.
>>>> The catpkgsplit change seems to definitely be safe, and I'm pretty
>>> sure
>>>> the use_reduce one is too, since anything that could possibly
>>> the
>>>> result is hashed.  I'm a bit less certain about the match_from_list
>>> one,
>>>> although all tests are passing.
>>>> With all 3 patches together, "emerge -uDvpU --with-bdeps=y @world"
>>>> speeds up from 43.53 seconds to 30.96 sec -- a 40.6% speedup. 
>>> "emerge
>>>> -ep @world" is just a tiny bit faster, going from 18.69 to 18.22
>>>> (2.5% improvement).  Since the upgrade case is far more common,
>>>> would really help in daily use, and it shaves about 30 seconds off
>>>> the time you have to wait to get to the [Yes/No] prompt (from ~90s
>>>> 60s) on my old Sandy Bridge laptop when performing normal upgrades.
>>>> Hopefully, at least some of these patches can be incorporated, and
>>> please
>>>> let me know if any changes are necessary.
>>>> Thanks,
>>>> Chun-Yu
>>> Using global variables for caches like these causes a form of memory
>>> leak for use cases involving long-running processes that need to
>>> with many different repositories (and perhaps multiple versions of
>>> those
>>> repositories).
>>> There are at least a couple of different strategies that we can use
>>> avoid this form of memory leak:
>>> 1) Limit the scope of the caches so that they have some sort of
>>> collection life cycle. For example, it would be natural for the
>>> depgraph
>>> class to have a local cache of use_reduce results, so that the cache
>>> can
>>> be garbage collected along with the depgraph.
>>> 2) Eliminate redundant calls. For example, redundant calls to
>>> catpkgslit
>>> can be avoided by constructing more _pkg_str instances, since
>>> catpkgsplit is able to return early when its argument happens to be
>>> _pkg_str instance.
>> I think the weak stuff from the standard library might also be
>> --
>> Best regards, 
>> Michał Górny
>Hmm, maybe weak global caches are an option?

It would probably be necessary to add hit/miss counter and compare results 
before and after.

Best regards, 
Michał Górny

Reply via email to