Re: [gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit
> On Mon, 13 Jul 2020, Chun-Yu Shei wrote: > Ah, I wasn't aware that I should have added that... I'm happy to say > "Signed-off-by: Chun-Yu Shei " somewhere if > necessary. Should be enough to say it here, because this mailing list is archived. We could of course add an empty commit with "Fixes" and "Signed-off-by", so we'd have the paper trail in the repo. > The patch has gone through Google's open source patching approval > process and I'm able to agree to any CLA required. We don't require a CLA. As long as you can signoff the copyright, things are fine. See https://www.gentoo.org/glep/glep-0076.html for all the details. Ulrich signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit
Ah, I wasn't aware that I should have added that... I'm happy to say "Signed-off-by: Chun-Yu Shei " somewhere if necessary. The patch has gone through Google's open source patching approval process and I'm able to agree to any CLA required. On Mon, Jul 13, 2020 at 11:54 AM Ulrich Mueller wrote: > > On Mon, 13 Jul 2020, Zac Medico wrote: > > > Merged: > > > > https://gitweb.gentoo.org/proj/portage.git/commit/?id=d9ee5b09664ab2255b62c1d52d554721ef8b716a > > Looks like the author's copyright signoff is missing? >
Re: [gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit
> On Mon, 13 Jul 2020, Zac Medico wrote: > Merged: > https://gitweb.gentoo.org/proj/portage.git/commit/?id=d9ee5b09664ab2255b62c1d52d554721ef8b716a Looks like the author's copyright signoff is missing? signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit
On 7/12/20 11:30 PM, Chun-Yu Shei wrote: > Each of these functions is called repeatedly with the same arguments > many times. Cache sizes were selected to minimize memory use increase, > while still providing about the same speedup compared to a cache with > unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases > from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the > RES column in htop increases from 280 MB -> 290 MB. > > "emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while > max observed RES value actually decreases from 228 MB -> 214 MB (similar > values observed across a few before/after runs). > --- > lib/portage/dep/__init__.py | 107 +--- > lib/portage/versions.py | 3 + > 2 files changed, 67 insertions(+), 43 deletions(-) Merged: https://gitweb.gentoo.org/proj/portage.git/commit/?id=d9ee5b09664ab2255b62c1d52d554721ef8b716a -- Thanks, Zac signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit
Each of these functions is called repeatedly with the same arguments
many times. Cache sizes were selected to minimize memory use increase,
while still providing about the same speedup compared to a cache with
unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases
from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the
RES column in htop increases from 280 MB -> 290 MB.
"emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while
max observed RES value actually decreases from 228 MB -> 214 MB (similar
values observed across a few before/after runs).
---
lib/portage/dep/__init__.py | 107 +---
lib/portage/versions.py | 3 +
2 files changed, 67 insertions(+), 43 deletions(-)
diff --git a/lib/portage/dep/__init__.py b/lib/portage/dep/__init__.py
index 72988357a..314338f7c 100644
--- a/lib/portage/dep/__init__.py
+++ b/lib/portage/dep/__init__.py
@@ -17,6 +17,7 @@ __all__ = [
import re, sys
import warnings
+from functools import lru_cache
import portage
portage.proxy.lazyimport.lazyimport(globals(),
@@ -404,49 +405,10 @@ def paren_enclose(mylist, unevaluated_atom=False,
opconvert=False):
mystrparts.append(x)
return " ".join(mystrparts)
-def use_reduce(depstr, uselist=(), masklist=(), matchall=False, excludeall=(),
is_src_uri=False, \
- eapi=None, opconvert=False, flat=False, is_valid_flag=None,
token_class=None, matchnone=False,
- subset=None):
- """
- Takes a dep string and reduces the use? conditionals out, leaving an
array
- with subarrays. All redundant brackets are removed.
-
- @param depstr: depstring
- @type depstr: String
- @param uselist: Sequence of use enabled flags
- @type uselist: Sequence
- @param masklist: Sequence of masked flags (always treated as disabled)
- @type masklist: Sequence
- @param matchall: Treat all conditionals as active. Used by repoman.
- @type matchall: Bool
- @param excludeall: Sequence of flags for which negated conditionals are
always treated as inactive.
- @type excludeall: Sequence
- @param is_src_uri: Indicates if depstr represents a SRC_URI
- @type is_src_uri: Bool
- @param eapi: Indicates the EAPI the dep string has to comply to
- @type eapi: String
- @param opconvert: Put every operator as first element into it's
argument list
- @type opconvert: Bool
- @param flat: Create a flat list of all tokens
- @type flat: Bool
- @param is_valid_flag: Function that decides if a given use flag might
be used in use conditionals
- @type is_valid_flag: Function
- @param token_class: Convert all non operator tokens into this class
- @type token_class: Class
- @param matchnone: Treat all conditionals as inactive. Used by
digestgen().
- @type matchnone: Bool
- @param subset: Select a subset of dependencies conditional on the given
flags
- @type subset: Sequence
- @rtype: List
- @return: The use reduced depend array
- """
- if isinstance(depstr, list):
- if portage._internal_caller:
- warnings.warn(_("Passing paren_reduced dep arrays to %s
is deprecated. " + \
- "Pass the original dep string instead.") % \
- ('portage.dep.use_reduce',),
DeprecationWarning, stacklevel=2)
- depstr = paren_enclose(depstr)
-
+@lru_cache(1024)
+def _use_reduce_cached(depstr, uselist, masklist, matchall, excludeall, \
+ is_src_uri, eapi, opconvert, flat, is_valid_flag, token_class, \
+ matchnone,subset):
if opconvert and flat:
raise ValueError("portage.dep.use_reduce: 'opconvert' and
'flat' are mutually exclusive")
@@ -769,6 +731,65 @@ def use_reduce(depstr, uselist=(), masklist=(),
matchall=False, excludeall=(), i
return stack[0]
+def use_reduce(depstr, uselist=(), masklist=(), matchall=False, excludeall=(),
is_src_uri=False, \
+ eapi=None, opconvert=False, flat=False, is_valid_flag=None,
token_class=None, matchnone=False,
+ subset=None):
+ """
+ Takes a dep string and reduces the use? conditionals out, leaving an
array
+ with subarrays. All redundant brackets are removed.
+
+ @param depstr: depstring
+ @type depstr: String
+ @param uselist: Sequence of use enabled flags
+ @type uselist: Sequence
+ @param masklist: Sequence of masked flags (always treated as disabled)
+ @type masklist: Sequence
+ @param matchall: Treat all conditionals as active. Used by repoman.
+ @type matchall: Bool
+ @param excludeall: Sequence of flags for which negated conditionals are
always treated as inactive.
+ @type excludeall: Sequence
+ @param is_src_uri: Indicates if depstr represents a SRC_URI
+ @type is_src_uri: Bool
+
Re: [gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit
On 7/9/20 12:03 AM, Chun-Yu Shei wrote:
> +def use_reduce(depstr, uselist=(), masklist=(), matchall=False,
> excludeall=(), is_src_uri=False, \
> + eapi=None, opconvert=False, flat=False, is_valid_flag=None,
> token_class=None, matchnone=False,
> + subset=None):
> + """
> + Takes a dep string and reduces the use? conditionals out, leaving an
> array
> + with subarrays. All redundant brackets are removed.
> +
> + @param depstr: depstring
> + @type depstr: String
> + @param uselist: Sequence of use enabled flags
> + @type uselist: Sequence
> + @param masklist: Sequence of masked flags (always treated as disabled)
> + @type masklist: Sequence
> + @param matchall: Treat all conditionals as active. Used by repoman.
> + @type matchall: Bool
> + @param excludeall: Sequence of flags for which negated conditionals are
> always treated as inactive.
> + @type excludeall: Sequence
> + @param is_src_uri: Indicates if depstr represents a SRC_URI
> + @type is_src_uri: Bool
> + @param eapi: Indicates the EAPI the dep string has to comply to
> + @type eapi: String
> + @param opconvert: Put every operator as first element into it's
> argument list
> + @type opconvert: Bool
> + @param flat: Create a flat list of all tokens
> + @type flat: Bool
> + @param is_valid_flag: Function that decides if a given use flag might
> be used in use conditionals
> + @type is_valid_flag: Function
> + @param token_class: Convert all non operator tokens into this class
> + @type token_class: Class
> + @param matchnone: Treat all conditionals as inactive. Used by
> digestgen().
> + @type matchnone: Bool
> + @param subset: Select a subset of dependencies conditional on the given
> flags
> + @type subset: Sequence
> + @rtype: List
> + @return: The use reduced depend array
> + """
> + if isinstance(depstr, list):
> + if portage._internal_caller:
> + warnings.warn(_("Passing paren_reduced dep arrays to %s
> is deprecated. " + \
> + "Pass the original dep string instead.") % \
> + ('portage.dep.use_reduce',),
> DeprecationWarning, stacklevel=2)
> + depstr = paren_enclose(depstr)
> +
> + if uselist is not None:
> + uselist = tuple(uselist)
> + if masklist is not None:
> + masklist = tuple(masklist)
> + if excludeall is not None:
> + excludeall = tuple(excludeall)
> + if subset is not None:
> + subset = tuple(subset)
The patch looks great, but maybe it's better if we use frozenset instead
of tuple for these.
--
Thanks,
Zac
signature.asc
Description: OpenPGP digital signature
[gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit
Each of these functions is called repeatedly with the same arguments
many times. Cache sizes were selected to minimize memory use increase,
while still providing about the same speedup compared to a cache with
unbounded size. "emerge -uDvpU --with-bdeps=y @world" runtime decreases
from 44.32s -> 29.94s -- a 48% speedup, while the maximum value of the
RES column in htop increases from 280 MB -> 290 MB.
"emerge -ep @world" time slightly decreases from 18.77s -> 17.93, while
max observed RES value actually decreases from 228 MB -> 214 MB (similar
values observed across a few before/after runs).
---
lib/portage/dep/__init__.py | 106 +---
lib/portage/versions.py | 3 +
2 files changed, 66 insertions(+), 43 deletions(-)
diff --git a/lib/portage/dep/__init__.py b/lib/portage/dep/__init__.py
index 72988357a..4d91a411a 100644
--- a/lib/portage/dep/__init__.py
+++ b/lib/portage/dep/__init__.py
@@ -23,6 +23,7 @@ portage.proxy.lazyimport.lazyimport(globals(),
'portage.util:cmp_sort_key,writemsg',
)
+from functools import lru_cache
from portage import _encodings, _unicode_decode, _unicode_encode
from portage.eapi import _get_eapi_attrs
from portage.exception import InvalidAtom, InvalidData, InvalidDependString
@@ -404,49 +405,9 @@ def paren_enclose(mylist, unevaluated_atom=False,
opconvert=False):
mystrparts.append(x)
return " ".join(mystrparts)
-def use_reduce(depstr, uselist=(), masklist=(), matchall=False, excludeall=(),
is_src_uri=False, \
- eapi=None, opconvert=False, flat=False, is_valid_flag=None,
token_class=None, matchnone=False,
- subset=None):
- """
- Takes a dep string and reduces the use? conditionals out, leaving an
array
- with subarrays. All redundant brackets are removed.
-
- @param depstr: depstring
- @type depstr: String
- @param uselist: Sequence of use enabled flags
- @type uselist: Sequence
- @param masklist: Sequence of masked flags (always treated as disabled)
- @type masklist: Sequence
- @param matchall: Treat all conditionals as active. Used by repoman.
- @type matchall: Bool
- @param excludeall: Sequence of flags for which negated conditionals are
always treated as inactive.
- @type excludeall: Sequence
- @param is_src_uri: Indicates if depstr represents a SRC_URI
- @type is_src_uri: Bool
- @param eapi: Indicates the EAPI the dep string has to comply to
- @type eapi: String
- @param opconvert: Put every operator as first element into it's
argument list
- @type opconvert: Bool
- @param flat: Create a flat list of all tokens
- @type flat: Bool
- @param is_valid_flag: Function that decides if a given use flag might
be used in use conditionals
- @type is_valid_flag: Function
- @param token_class: Convert all non operator tokens into this class
- @type token_class: Class
- @param matchnone: Treat all conditionals as inactive. Used by
digestgen().
- @type matchnone: Bool
- @param subset: Select a subset of dependencies conditional on the given
flags
- @type subset: Sequence
- @rtype: List
- @return: The use reduced depend array
- """
- if isinstance(depstr, list):
- if portage._internal_caller:
- warnings.warn(_("Passing paren_reduced dep arrays to %s
is deprecated. " + \
- "Pass the original dep string instead.") % \
- ('portage.dep.use_reduce',),
DeprecationWarning, stacklevel=2)
- depstr = paren_enclose(depstr)
-
+@lru_cache(1024)
+def use_reduce_cached(depstr, uselist, masklist, matchall, excludeall,
is_src_uri, eapi, \
+ opconvert, flat, is_valid_flag, token_class, matchnone, subset):
if opconvert and flat:
raise ValueError("portage.dep.use_reduce: 'opconvert' and
'flat' are mutually exclusive")
@@ -769,6 +730,65 @@ def use_reduce(depstr, uselist=(), masklist=(),
matchall=False, excludeall=(), i
return stack[0]
+def use_reduce(depstr, uselist=(), masklist=(), matchall=False, excludeall=(),
is_src_uri=False, \
+ eapi=None, opconvert=False, flat=False, is_valid_flag=None,
token_class=None, matchnone=False,
+ subset=None):
+ """
+ Takes a dep string and reduces the use? conditionals out, leaving an
array
+ with subarrays. All redundant brackets are removed.
+
+ @param depstr: depstring
+ @type depstr: String
+ @param uselist: Sequence of use enabled flags
+ @type uselist: Sequence
+ @param masklist: Sequence of masked flags (always treated as disabled)
+ @type masklist: Sequence
+ @param matchall: Treat all conditionals as active. Used by repoman.
+ @type matchall: Bool
+ @param excludeall: Sequence of flags for which negated conditionals are
