Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Alec Warner
On Thu, Jul 9, 2020 at 2:06 PM Chun-Yu Shei wrote: > Hmm, that's strange... it seems to have made it to the list archives: > https://archives.gentoo.org/gentoo-portage-dev/message/a4db905a64e3c1f6d88c4876e8291a65 > > (but it is entirely possible that I used "git send-email" incorrectly) > Ahhh

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Chun-Yu Shei
Hmm, that's strange... it seems to have made it to the list archives: https://archives.gentoo.org/gentoo-portage-dev/message/a4db905a64e3c1f6d88c4876e8291a65 (but it is entirely possible that I used "git send-email" incorrectly) On Thu, Jul 9, 2020 at 2:04 PM Alec Warner wrote: > > > On Thu,

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Alec Warner
On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei wrote: > Awesome! Here's a patch that adds @lru_cache to use_reduce, vercmp, and > catpkgsplit. use_reduce was split into 2 functions, with the outer one > converting lists/sets to tuples so they can be hashed and creating a > copy of the returned

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-09 Thread Chun-Yu Shei
Awesome! Here's a patch that adds @lru_cache to use_reduce, vercmp, and catpkgsplit. use_reduce was split into 2 functions, with the outer one converting lists/sets to tuples so they can be hashed and creating a copy of the returned list (since the caller seems to modify it sometimes). I tried

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-06 Thread Zac Medico
On 7/6/20 11:03 AM, Zac Medico wrote: > On 7/6/20 10:30 AM, Chun-Yu Shei wrote: >> I finally got a chance to try Sid's lru_cache suggestion, and the >> results were really good. Simply adding it on catpkgsplit and moving >> the body of use_reduce into a separate function (that accepts tuples >>

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-06 Thread Zac Medico
On 7/6/20 10:30 AM, Chun-Yu Shei wrote: > I finally got a chance to try Sid's lru_cache suggestion, and the > results were really good. Simply adding it on catpkgsplit and moving > the body of use_reduce into a separate function (that accepts tuples > instead of unhashable lists/sets) and

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-06 Thread Chun-Yu Shei
I finally got a chance to try Sid's lru_cache suggestion, and the results were really good. Simply adding it on catpkgsplit and moving the body of use_reduce into a separate function (that accepts tuples instead of unhashable lists/sets) and decorating it with lru_cache gets a similar 40% overall

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-06 Thread Francesco Riosa
Il 06/07/20 17:50, Michael 'veremitz' Everitt ha scritto: On 06/07/20 16:26, Francesco Riosa wrote: Il 29/06/20 03:58, Sid Spry ha scritto: There are libraries that provide decorators, etc, for caching and memoization. Have you evaluated any of those? One is available in the standard library:

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-07-06 Thread Francesco Riosa
Il 29/06/20 03:58, Sid Spry ha scritto: There are libraries that provide decorators, etc, for caching and memoization. Have you evaluated any of those? One is available in the standard library: https://docs.python.org/dev/library/functools.html#functools.lru_cache I comment as this would

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-06-28 Thread Sid Spry
On Sat, Jun 27, 2020, at 1:34 AM, Chun-Yu Shei wrote: > According to cProfile, catpkgsplit is called up to 1-5.5 million times > during "emerge -uDvpU --with-bdeps=y @world". Adding a dict to cache its > results reduces the time for this command from 43.53 -> 41.53 seconds -- > a 4.8% speedup. >

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-06-27 Thread Michał Górny
Dnia June 27, 2020 6:34:13 AM UTC, Chun-Yu Shei napisał(a): >According to cProfile, catpkgsplit is called up to 1-5.5 million times >during "emerge -uDvpU --with-bdeps=y @world". Adding a dict to cache >its >results reduces the time for this command from 43.53 -> 41.53 seconds >-- >a 4.8%

[gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-06-27 Thread Chun-Yu Shei
According to cProfile, catpkgsplit is called up to 1-5.5 million times during "emerge -uDvpU --with-bdeps=y @world". Adding a dict to cache its results reduces the time for this command from 43.53 -> 41.53 seconds -- a 4.8% speedup. --- lib/portage/versions.py | 7 +++ 1 file changed, 7