Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function
On Thu, Jul 9, 2020 at 2:06 PM Chun-Yu Shei wrote: > Hmm, that's strange... it seems to have made it to the list archives: > https://archives.gentoo.org/gentoo-portage-dev/message/a4db905a64e3c1f6d88c4876e8291a65 > > (but it is entirely possible that I used "git send-email" incorrectly) > Ahhh it's visible there; I'll blame gMail ;) -A > > On Thu, Jul 9, 2020 at 2:04 PM Alec Warner wrote: > >> >> >> On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei wrote: >> >>> Awesome! Here's a patch that adds @lru_cache to use_reduce, vercmp, and >>> catpkgsplit. use_reduce was split into 2 functions, with the outer one >>> converting lists/sets to tuples so they can be hashed and creating a >>> copy of the returned list (since the caller seems to modify it >>> sometimes). I tried to select cache sizes that minimized memory use >>> increase, >>> while still providing about the same speedup compared to a cache with >>> unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases >>> from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the >>> RES column in htop increases from 280 MB -> 290 MB. >>> >>> "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while >>> max observed RES value actually decreases from 228 MB -> 214 MB (similar >>> values observed across a few before/after runs). >>> >>> Here are the cache hit stats, max observed RES memory, and runtime in >>> seconds for various sizes in the update case. Caching for each >>> function was tested independently (only 1 function with caching enabled >>> at a time): >>> >>> catpkgsplit: >>> CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419) >>> 270 MB >>> 39.217 >>> >>> CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1) >>> 271 MB >>> 39.112 >>> >>> CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000) >>> 271 MB >>> 39.217 >>> >>> CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500) >>> 269 MB >>> 39.438 >>> >>> CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000) >>> 271 MB >>> 39.348 >>> >>> CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100) >>> 271 MB >>> 39.487 >>> >>> >>> use_reduce: >>> CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561) >>> 407 MB >>> 35.77 >>> >>> CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1) >>> 353 MB >>> 35.52 >>> >>> CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000) >>> 335 MB >>> 35.31 >>> >>> CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500) >>> 318 MB >>> 35.85 >>> >>> CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000) >>> 301 MB >>> 36.39 >>> >>> CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100) >>> 299 MB >>> 37.175 >>> >>> >>> I didn't bother collecting detailed stats for vercmp, since the >>> inputs/outputs are quite small and don't cause much memory increase. >>> Please let me know if there are any other suggestions/improvements (and >>> thanks Sid for the lru_cache suggestion!). >>> >> >> I don't see a patch attached; can you link to it? >> >> -A >> >> >>> >>> Thanks, >>> Chun-Yu >>> >>> >>> >>>
Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function
Hmm, that's strange... it seems to have made it to the list archives: https://archives.gentoo.org/gentoo-portage-dev/message/a4db905a64e3c1f6d88c4876e8291a65 (but it is entirely possible that I used "git send-email" incorrectly) On Thu, Jul 9, 2020 at 2:04 PM Alec Warner wrote: > > > On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei wrote: > >> Awesome! Here's a patch that adds @lru_cache to use_reduce, vercmp, and >> catpkgsplit. use_reduce was split into 2 functions, with the outer one >> converting lists/sets to tuples so they can be hashed and creating a >> copy of the returned list (since the caller seems to modify it >> sometimes). I tried to select cache sizes that minimized memory use >> increase, >> while still providing about the same speedup compared to a cache with >> unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases >> from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the >> RES column in htop increases from 280 MB -> 290 MB. >> >> "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while >> max observed RES value actually decreases from 228 MB -> 214 MB (similar >> values observed across a few before/after runs). >> >> Here are the cache hit stats, max observed RES memory, and runtime in >> seconds for various sizes in the update case. Caching for each >> function was tested independently (only 1 function with caching enabled >> at a time): >> >> catpkgsplit: >> CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419) >> 270 MB >> 39.217 >> >> CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1) >> 271 MB >> 39.112 >> >> CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000) >> 271 MB >> 39.217 >> >> CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500) >> 269 MB >> 39.438 >> >> CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000) >> 271 MB >> 39.348 >> >> CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100) >> 271 MB >> 39.487 >> >> >> use_reduce: >> CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561) >> 407 MB >> 35.77 >> >> CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1) >> 353 MB >> 35.52 >> >> CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000) >> 335 MB >> 35.31 >> >> CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500) >> 318 MB >> 35.85 >> >> CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000) >> 301 MB >> 36.39 >> >> CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100) >> 299 MB >> 37.175 >> >> >> I didn't bother collecting detailed stats for vercmp, since the >> inputs/outputs are quite small and don't cause much memory increase. >> Please let me know if there are any other suggestions/improvements (and >> thanks Sid for the lru_cache suggestion!). >> > > I don't see a patch attached; can you link to it? > > -A > > >> >> Thanks, >> Chun-Yu >> >> >> >>
Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function
On Thu, Jul 9, 2020 at 12:03 AM Chun-Yu Shei wrote: > Awesome! Here's a patch that adds @lru_cache to use_reduce, vercmp, and > catpkgsplit. use_reduce was split into 2 functions, with the outer one > converting lists/sets to tuples so they can be hashed and creating a > copy of the returned list (since the caller seems to modify it > sometimes). I tried to select cache sizes that minimized memory use > increase, > while still providing about the same speedup compared to a cache with > unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases > from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the > RES column in htop increases from 280 MB -> 290 MB. > > "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while > max observed RES value actually decreases from 228 MB -> 214 MB (similar > values observed across a few before/after runs). > > Here are the cache hit stats, max observed RES memory, and runtime in > seconds for various sizes in the update case. Caching for each > function was tested independently (only 1 function with caching enabled > at a time): > > catpkgsplit: > CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419) > 270 MB > 39.217 > > CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1) > 271 MB > 39.112 > > CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000) > 271 MB > 39.217 > > CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500) > 269 MB > 39.438 > > CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000) > 271 MB > 39.348 > > CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100) > 271 MB > 39.487 > > > use_reduce: > CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561) > 407 MB > 35.77 > > CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1) > 353 MB > 35.52 > > CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000) > 335 MB > 35.31 > > CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500) > 318 MB > 35.85 > > CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000) > 301 MB > 36.39 > > CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100) > 299 MB > 37.175 > > > I didn't bother collecting detailed stats for vercmp, since the > inputs/outputs are quite small and don't cause much memory increase. > Please let me know if there are any other suggestions/improvements (and > thanks Sid for the lru_cache suggestion!). > I don't see a patch attached; can you link to it? -A > > Thanks, > Chun-Yu > > > >
[gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit
Each of these functions is called repeatedly with the same arguments
many times. Cache sizes were selected to minimize memory use increase,
while still providing about the same speedup compared to a cache with
unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases
from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the
RES column in htop increases from 280 MB -> 290 MB.
"emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while
max observed RES value actually decreases from 228 MB -> 214 MB (similar
values observed across a few before/after runs).
---
lib/portage/dep/__init__.py | 106 +---
lib/portage/versions.py | 3 +
2 files changed, 66 insertions(+), 43 deletions(-)
diff --git a/lib/portage/dep/__init__.py b/lib/portage/dep/__init__.py
index 72988357a..4d91a411a 100644
--- a/lib/portage/dep/__init__.py
+++ b/lib/portage/dep/__init__.py
@@ -23,6 +23,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util:cmp_sort_key,writemsg',
)
+from functools import lru_cache
from portage import _encodings, _unicode_decode, _unicode_encode
from portage.eapi import _get_eapi_attrs
from portage.exception import InvalidAtom, InvalidData, InvalidDependString
@@ -404,49 +405,9 @@ def paren_enclose(mylist, unevaluated_atom=False,
opconvert=False):
mystrparts.append(x)
return " ".join(mystrparts)
-def use_reduce(depstr, uselist=(), masklist=(), matchall=False, excludeall=(),
is_src_uri=False, \
- eapi=None, opconvert=False, flat=False, is_valid_flag=None,
token_class=None, matchnone=False,
- subset=None):
- """
- Takes a dep string and reduces the use? conditionals out, leaving an
array
- with subarrays. All redundant brackets are removed.
-
- @param depstr: depstring
- @type depstr: String
- @param uselist: Sequence of use enabled flags
- @type uselist: Sequence
- @param masklist: Sequence of masked flags (always treated as disabled)
- @type masklist: Sequence
- @param matchall: Treat all conditionals as active. Used by repoman.
- @type matchall: Bool
- @param excludeall: Sequence of flags for which negated conditionals are
always treated as inactive.
- @type excludeall: Sequence
- @param is_src_uri: Indicates if depstr represents a SRC_URI
- @type is_src_uri: Bool
- @param eapi: Indicates the EAPI the dep string has to comply to
- @type eapi: String
- @param opconvert: Put every operator as first element into it's
argument list
- @type opconvert: Bool
- @param flat: Create a flat list of all tokens
- @type flat: Bool
- @param is_valid_flag: Function that decides if a given use flag might
be used in use conditionals
- @type is_valid_flag: Function
- @param token_class: Convert all non operator tokens into this class
- @type token_class: Class
- @param matchnone: Treat all conditionals as inactive. Used by
digestgen().
- @type matchnone: Bool
- @param subset: Select a subset of dependencies conditional on the given
flags
- @type subset: Sequence
- @rtype: List
- @return: The use reduced depend array
- """
- if isinstance(depstr, list):
- if portage._internal_caller:
- warnings.warn(_("Passing paren_reduced dep arrays to %s
is deprecated. " + \
- "Pass the original dep string instead.") % \
- ('portage.dep.use_reduce',),
DeprecationWarning, stacklevel=2)
- depstr = paren_enclose(depstr)
-
+@lru_cache(1024)
+def use_reduce_cached(depstr, uselist, masklist, matchall, excludeall,
is_src_uri, eapi, \
+ opconvert, flat, is_valid_flag, token_class, matchnone, subset):
if opconvert and flat:
raise ValueError("portage.dep.use_reduce: 'opconvert' and
'flat' are mutually exclusive")
@@ -769,6 +730,65 @@ def use_reduce(depstr, uselist=(), masklist=(),
matchall=False, excludeall=(), i
return stack[0]
+def use_reduce(depstr, uselist=(), masklist=(), matchall=False, excludeall=(),
is_src_uri=False, \
+ eapi=None, opconvert=False, flat=False, is_valid_flag=None,
token_class=None, matchnone=False,
+ subset=None):
+ """
+ Takes a dep string and reduces the use? conditionals out, leaving an
array
+ with subarrays. All redundant brackets are removed.
+
+ @param depstr: depstring
+ @type depstr: String
+ @param uselist: Sequence of use enabled flags
+ @type uselist: Sequence
+ @param masklist: Sequence of masked flags (always treated as disabled)
+ @type masklist: Sequence
+ @param matchall: Treat all conditionals as active. Used by repoman.
+ @type matchall: Bool
+ @param excludeall: Sequence of flags for which negated conditionals are
Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function
Awesome! Here's a patch that adds @lru_cache to use_reduce, vercmp, and catpkgsplit. use_reduce was split into 2 functions, with the outer one converting lists/sets to tuples so they can be hashed and creating a copy of the returned list (since the caller seems to modify it sometimes). I tried to select cache sizes that minimized memory use increase, while still providing about the same speedup compared to a cache with unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the RES column in htop increases from 280 MB -> 290 MB. "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while max observed RES value actually decreases from 228 MB -> 214 MB (similar values observed across a few before/after runs). Here are the cache hit stats, max observed RES memory, and runtime in seconds for various sizes in the update case. Caching for each function was tested independently (only 1 function with caching enabled at a time): catpkgsplit: CacheInfo(hits=133, misses=21419, maxsize=None, currsize=21419) 270 MB 39.217 CacheInfo(hits=1218900, misses=24905, maxsize=1, currsize=1) 271 MB 39.112 CacheInfo(hits=1212675, misses=31022, maxsize=5000, currsize=5000) 271 MB 39.217 CacheInfo(hits=1207879, misses=35878, maxsize=2500, currsize=2500) 269 MB 39.438 CacheInfo(hits=1199402, misses=44250, maxsize=1000, currsize=1000) 271 MB 39.348 CacheInfo(hits=1149150, misses=94610, maxsize=100, currsize=100) 271 MB 39.487 use_reduce: CacheInfo(hits=45326, misses=18660, maxsize=None, currsize=18561) 407 MB 35.77 CacheInfo(hits=45186, misses=18800, maxsize=1, currsize=1) 353 MB 35.52 CacheInfo(hits=44977, misses=19009, maxsize=5000, currsize=5000) 335 MB 35.31 CacheInfo(hits=44691, misses=19295, maxsize=2500, currsize=2500) 318 MB 35.85 CacheInfo(hits=44178, misses=19808, maxsize=1000, currsize=1000) 301 MB 36.39 CacheInfo(hits=41211, misses=22775, maxsize=100, currsize=100) 299 MB 37.175 I didn't bother collecting detailed stats for vercmp, since the inputs/outputs are quite small and don't cause much memory increase. Please let me know if there are any other suggestions/improvements (and thanks Sid for the lru_cache suggestion!). Thanks, Chun-Yu
