On 26/08/2021 17:38, Marco Sirabella wrote: > - > > Hi Ed, > > I’ve taken a dabble at trying to track down portage’s bottlenecks (and have > stopped for the time > being at solving them :/ ) > > Can anyone give me a leg up on how I could benchmark this further and > look for the hotspot? > Perhaps someone understand the architecture of this point more intimately > and could point at > whether there are opportunities to do some of the processing on mass, > rather than per file? > > From my notes at the timem, it looks like yappi > <https://pypi.org/project/yappi/> worked a bit > better than python’s built in cProfile for me because it properly dove into > async calls. I used > snakeviz <https://jiffyclub.github.io/snakeviz/> for visualizing the profile > results. > > I was taking a look at depclean, but I found similarly that a lot of > duplicate process was being > done due to encapsulated abstractions not being able to communicate that the > same thing was being > done multiple times eg removing each package processes a massive json > structure for each package > removed, although I opted to work on the more-understandable unicode > conversions. > > My stalled progress can be found here: #700 > <https://github.com/gentoo/portage/pull/700>. Lost the > drive to continue for now unfortunately :< > > Good luck! Looking forward to your optimizations > > – Marco Sirabella >
Hi All, thanks for the replies. Wow, Marco, that patch touches a lot of stuff...! OK, I will start by trying to get the profilers going and work from there... (Alec, to avoid replying separately: Thanks for your notes. Yes, I am not clear which of the install/merge phases specifically is the culprit, but it feels like something in that area is "slow", especially when run under qemu user mode. I think using qmerge won't work for my build script, but great idea to use it for benchmarking to narrow things down - thanks) Thanks all Ed W
