Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Kent Fredric
On Sat, 27 Jun 2020 at 19:35, Fabian Groffen wrote: > > Hi Chun-Yu, > > arguments: catpkgsplit, use_reduce, and match_from_list. In the first > > two cases, it was simple to cache the results in dicts, while > > match_from_list was a bit trickier, since it seems to be a requirement > > that it

Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Fabian Groffen
Hi Chun-Yu, On 26-06-2020 23:34:12 -0700, Chun-Yu Shei wrote: > Hi, > > I was recently interested in whether portage could be speed up, since > dependency resolution can sometimes take a while on slower machines. > After generating some flame graphs with cProfile and vmprof, I found 3 >

[gentoo-dev] Last rites: media-sound/guimup

2020-06-27 Thread Mart Raudsepp
# Mart Raudsepp (2020-06-27) # Disappeared upstream and download locations. Potential replacements: # media-sound/quimup, xfce-extra/xfce4-mpc-plugin, media-sound/xfmpc # Bug 729822. Removal in 30 days. media-sound/guimup signature.asc Description: This is a digitally signed message part

Re: [gentoo-dev] */*: Mask Py2 only packages

2020-06-27 Thread Joonas Niilola
On 6/27/20 2:28 AM, Robin H. Johnson wrote: > > TL;DR: Please make it easier to search on the QA reports site for > issues, and only show things directly relevant to the search. > > A long time ago, there was blizzy's site that listed packages that were > stabilization candidates, and you could

Re: [gentoo-dev] */*: Mask Py2 only packages

2020-06-27 Thread Robin H. Johnson
On Sat, Jun 27, 2020 at 10:08:26AM +0300, Joonas Niilola wrote: > > On 6/27/20 2:28 AM, Robin H. Johnson wrote: > > > > TL;DR: Please make it easier to search on the QA reports site for > > issues, and only show things directly relevant to the search. > > > > A long time ago, there was blizzy's

Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Chun-Yu Shei
Hi Fabian, Just eyeballing htop's RES column while emerge is running, the max value I see during "emerge -uDvpU --with-bdeps=y @world" increases from 272 MB to 380 MB, so it looks to be around 110 MB of extra memory usage on my system (with 1,094 total packages installed). Chun-Yu On Sat, Jun

[gentoo-portage-dev] [PATCH 2/3] Add caching to use_reduce function

2020-06-27 Thread Chun-Yu Shei
This function is called extremely frequently with similar arguments, so this optimization reduces "emerge -uDvpU --with-bdeps=y @world" runtime from 43.5 -> 34.5s -- a 25.8% speedup. --- lib/portage/dep/__init__.py | 26 ++ 1 file changed, 26 insertions(+) diff --git

[gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Chun-Yu Shei
Hi, I was recently interested in whether portage could be speed up, since dependency resolution can sometimes take a while on slower machines. After generating some flame graphs with cProfile and vmprof, I found 3 functions which seem to be called extremely frequently with the same arguments:

[gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-06-27 Thread Chun-Yu Shei
According to cProfile, catpkgsplit is called up to 1-5.5 million times during "emerge -uDvpU --with-bdeps=y @world". Adding a dict to cache its results reduces the time for this command from 43.53 -> 41.53 seconds -- a 4.8% speedup. --- lib/portage/versions.py | 7 +++ 1 file changed, 7

[gentoo-portage-dev] [PATCH 3/3] Add partial caching to match_from_list

2020-06-27 Thread Chun-Yu Shei
This function is called frequently with similar arguments, so cache as much of the partial results as possible. It seems that "match_from_list" must return a list containing actual entries from the "candidate_list" argument, so we store frozensets in "_match_from_list_cache" and test items from

Re: [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function

2020-06-27 Thread Michał Górny
Dnia June 27, 2020 6:34:13 AM UTC, Chun-Yu Shei napisał(a): >According to cProfile, catpkgsplit is called up to 1-5.5 million times >during "emerge -uDvpU --with-bdeps=y @world". Adding a dict to cache >its >results reduces the time for this command from 43.53 -> 41.53 seconds >-- >a 4.8%

[gentoo-portage-dev] Passing CFLAGS="-fdebug-prefix-map=..=$(readlink -f ..)"

2020-06-27 Thread Joakim Tjernlund
I am trying to add -fdebug-prefix-map to my CFLAGS but cannot get past "$(readlink -f ..)" Portage will not expand $(anything) Any way to make portage expand "$(readlink -f ..)" ? Jocke

Re: [gentoo-portage-dev] Passing CFLAGS="-fdebug-prefix-map=..=$(readlink -f ..)"

2020-06-27 Thread Joakim Tjernlund
On Sat, 2020-06-27 at 14:00 +, Joakim Tjernlund wrote: > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. > > > I am trying to add -fdebug-prefix-map to my CFLAGS but

Re: [gentoo-dev] Re: News item: xorg-server dropping default suid

2020-06-27 Thread Adam Feldman
On 6/27/20 1:36 PM, William Hubbs wrote: > On Sun, Jun 21, 2020 at 10:02:25PM +0200, Andreas Sturmlechner wrote: >> On Sunday, 21 June 2020 21:27:02 CEST Joonas Niilola wrote: >>> What's the current trend of attaching news items? It >>> makes hard to point out enhancements. >> >> Indeed, I didn't

Re: [gentoo-dev] Re: News item: xorg-server dropping default suid

2020-06-27 Thread William Hubbs
On Sun, Jun 21, 2020 at 10:02:25PM +0200, Andreas Sturmlechner wrote: > On Sunday, 21 June 2020 21:27:02 CEST Joonas Niilola wrote: > > What's the current trend of attaching news items? It > > makes hard to point out enhancements. > > Indeed, I didn't even look at the previous mail that was sent

Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Zac Medico
On 6/26/20 11:34 PM, Chun-Yu Shei wrote: > Hi, > > I was recently interested in whether portage could be speed up, since > dependency resolution can sometimes take a while on slower machines. > After generating some flame graphs with cProfile and vmprof, I found 3 > functions which seem to be

Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Michał Górny
Dnia June 28, 2020 3:00:00 AM UTC, Zac Medico napisał(a): >On 6/26/20 11:34 PM, Chun-Yu Shei wrote: >> Hi, >> >> I was recently interested in whether portage could be speed up, since >> dependency resolution can sometimes take a while on slower machines. >> After generating some flame graphs

Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Michał Górny
Dnia June 28, 2020 3:42:33 AM UTC, Zac Medico napisał(a): >On 6/27/20 8:12 PM, Michał Górny wrote: >> Dnia June 28, 2020 3:00:00 AM UTC, Zac Medico >napisał(a): >>> On 6/26/20 11:34 PM, Chun-Yu Shei wrote: Hi, I was recently interested in whether portage could be speed up, >since

Re: [gentoo-portage-dev] Add caching to a few commonly used functions

2020-06-27 Thread Zac Medico
On 6/27/20 8:12 PM, Michał Górny wrote: > Dnia June 28, 2020 3:00:00 AM UTC, Zac Medico napisał(a): >> On 6/26/20 11:34 PM, Chun-Yu Shei wrote: >>> Hi, >>> >>> I was recently interested in whether portage could be speed up, since >>> dependency resolution can sometimes take a while on slower