So... like I said, using a generator there, need to ensure it works with all code. Beyond that, actually a generator can't be used there- the returned list needs to be unique, so you'd have to resort to the same list checking approach.
So... for those who just want a faster portage, give cookies to J?rgen H?tzel for pointing this deficient code out. On my system fixing that func (cp_all) resulted in an emerge -s diffball going from 21s, to around 5s. Big difference in speed, and my sanity in waiting on searches ;) cp_list (several of 'em), p_list, cp_list, and a horrid alg in the eclass cache verification code have been fixed in cvs, and are available via a patch at http://dev.gentoo.org/~ferringb/51-speedups-part-deux.patch (modifies portage.py) If you're still using 2.0.51-r15, http://dev.gentoo.org/~ferringb/flat-non-locking.patch (modifies portage_db_flat.py) also would be worth trying- 2.0.51.16 (hard masked atm for testing) has the same code. Basically, it yanks the locking out of the cache backend, and implements a saner approach, that returns performance back to the previous non-locking performance- sans updates. A bit extra IO then when locking didn't exist, but it's still a helluva lot faster then the -r15 locking approach. Either way, all patches should be safe... so try it out kids. Eat the pudding. ~harring On Wed, Feb 16, 2005 at 06:36:38PM +0100, J?rgen H?tzel wrote: > Hi, > > Because portage searches took so long on my notebook i started > profiling and found the bottleneck in > > class portdbapi::cp_all (/usr/lib/portage/pym/portage.py) > > a huge list is generated and returned. Its better to yield each single > entry. Generators are part of python since 2.3.0. I wonder why they > are not used in portage. Look at the performance improvement of my patch: > > bash-2.05b# time emerge search python-mode > Searching... > [ Results for search key : python-mode ] > [ Applications found : 2 ] > .. > real 0m6.845s > user 0m6.536s > sys 0m0.137s > > After applying the patch: > > bash-2.05b# time emerge search python-mode > Searching... > [ Results for search key : python-mode ] > [ Applications found : 2 ] > .. > real 0m2.323s > user 0m1.914s > sys 0m0.142s > > Should i submit my patch too bugs.gentoo.org? > > Juergen > --- portage_old.py 2005-02-16 17:15:39.731629944 +0100 > +++ portage.py 2005-02-16 16:57:28.755483528 +0100 > @@ -5367,15 +5367,15 @@ > return 0 > > def cp_all(self): > - "returns a list of all keys in our tree" > - biglist=[] > + "yield all keys in our tree" > for x in self.mysettings.categories: > + biglist=[] > for oroot in self.porttrees: > for y in > listdir(oroot+"/"+x,EmptyOnError=1,ignorecvs=1): > mykey=x+"/"+y > if not mykey in biglist: > biglist.append(mykey) > - return biglist > + yield(mykey) > > def p_list(self,mycp): > returnme=[] > > -- > [email protected] mailing list -- [email protected] mailing list
