Works good for me! Nice improvement :-)
thanks to whoever pointed this out. On Fri, 18 Feb 2005 02:49:48 -0600, Brian Harring <[EMAIL PROTECTED]> wrote: > So... like I said, using a generator there, need to ensure it works with all > code. Beyond that, actually a generator > can't be used there- the returned list needs to be unique, so you'd have to > resort to the same list checking approach. > > So... for those who just want a faster portage, give cookies to J?rgen H?tzel > for pointing this deficient code out. > On my system fixing that func (cp_all) resulted in an emerge -s diffball > going from 21s, to around 5s. > > Big difference in speed, and my sanity in waiting on searches ;) > > cp_list (several of 'em), p_list, cp_list, and a horrid alg in the eclass > cache verification code have been fixed in > cvs, and are available via a patch at > http://dev.gentoo.org/~ferringb/51-speedups-part-deux.patch > (modifies portage.py) > > If you're still using 2.0.51-r15, > http://dev.gentoo.org/~ferringb/flat-non-locking.patch > (modifies portage_db_flat.py) > also would be worth trying- 2.0.51.16 (hard masked atm for testing) has the > same code. Basically, it yanks the > locking out of the cache backend, and implements a saner approach, that > returns performance back to the previous > non-locking performance- sans updates. A bit extra IO then when locking > didn't exist, but it's still a helluva lot > faster then the -r15 locking approach. > > Either way, all patches should be safe... so try it out kids. > > Eat the pudding. > ~harring > > On Wed, Feb 16, 2005 at 06:36:38PM +0100, J?rgen H?tzel wrote: > > Hi, > > > > Because portage searches took so long on my notebook i started > > profiling and found the bottleneck in > > > > class portdbapi::cp_all (/usr/lib/portage/pym/portage.py) > > > > a huge list is generated and returned. Its better to yield each single > > entry. Generators are part of python since 2.3.0. I wonder why they > > are not used in portage. Look at the performance improvement of my patch: > > > > bash-2.05b# time emerge search python-mode > > Searching... > > [ Results for search key : python-mode ] > > [ Applications found : 2 ] > > .. > > real 0m6.845s > > user 0m6.536s > > sys 0m0.137s > > > > After applying the patch: > > > > bash-2.05b# time emerge search python-mode > > Searching... > > [ Results for search key : python-mode ] > > [ Applications found : 2 ] > > .. > > real 0m2.323s > > user 0m1.914s > > sys 0m0.142s > > > > Should i submit my patch too bugs.gentoo.org? > > > > Juergen > > --- portage_old.py 2005-02-16 17:15:39.731629944 +0100 > > +++ portage.py 2005-02-16 16:57:28.755483528 +0100 > > @@ -5367,15 +5367,15 @@ > > return 0 > > > > def cp_all(self): > > - "returns a list of all keys in our tree" > > - biglist=[] > > + "yield all keys in our tree" > > for x in self.mysettings.categories: > > + biglist=[] > > for oroot in self.porttrees: > > for y in > > listdir(oroot+"/"+x,EmptyOnError=1,ignorecvs=1): > > mykey=x+"/"+y > > if not mykey in biglist: > > biglist.append(mykey) > > - return biglist > > + yield(mykey) > > > > def p_list(self,mycp): > > returnme=[] > > > > > -- > > [email protected] mailing list > > -- > [email protected] mailing list > > -- [email protected] mailing list
