On 26/08/2021 17:38, Marco Sirabella wrote:
> -
> Hi Ed,
> I’ve taken a dabble at trying to track down portage’s bottlenecks (and have 
> stopped for the time
> being at solving them :/ )
>     Can anyone give me a leg up on how I could benchmark this further and 
> look for the hotspot?
>     Perhaps someone understand the architecture of this point more intimately 
> and could point at
>     whether there are opportunities to do some of the processing on mass, 
> rather than per file?
> From my notes at the timem, it looks like yappi 
> <https://pypi.org/project/yappi/> worked a bit
> better than python’s built in cProfile for me because it properly dove into 
> async calls. I used
> snakeviz <https://jiffyclub.github.io/snakeviz/> for visualizing the profile 
> results.
> I was taking a look at depclean, but I found similarly that a lot of 
> duplicate process was being
> done due to encapsulated abstractions not being able to communicate that the 
> same thing was being
> done multiple times eg removing each package processes a massive json 
> structure for each package
> removed, although I opted to work on the more-understandable unicode 
> conversions.
> My stalled progress can be found here: #700 
> <https://github.com/gentoo/portage/pull/700>. Lost the
> drive to continue for now unfortunately :<
> Good luck! Looking forward to your optimizations
> – Marco Sirabella

Hi All, thanks for the replies. Wow, Marco, that patch touches a lot of 

OK, I will start by trying to get the profilers going and work from there...

(Alec, to avoid replying separately: Thanks for your notes. Yes, I am not clear 
which of the
install/merge phases specifically is the culprit, but it feels like something 
in that area is
"slow", especially when run under qemu user mode. I think using qmerge won't 
work for my build
script, but great idea to use it for benchmarking to narrow things down - 

Thanks all

Ed W

Reply via email to