On Sat, Jul 3, 2010 at 4:27 PM, Andres Vargas <zod...@foresightlinux.org>wrote:
> > The high memory usage its cause on the update-system. this process not use > the XMLRepo. > Not true, you instantiate it (which _opens and parses the XML repos when you do this): jobPath = Cache().checkCachedUpdateJob(applyList) However, that only happens once during update-system. > > ¿smerp where you seen the high memory usages its caused by xmlcache ? > > Well, I saw it before the changes you mentioned below were made to gnome-backend to use update-system. Every package that was updated instantiated a instance of Cache(). I tracked this down with guppy; making it a singleton basically "fixed" PK's excessive memory utilization. IIRC, it's much, much worse during the get-updates phase. * Changing the object model. The object model of using dicts of dicts and > > tuple was a mess. I tried to simpify this by using objects with _slots_ > for > > the specific things, and using dicts for things that couldn't be > predicted. > > > I think about it > http://wiki.foresightlinux.org/display/DEV/Redesign+Conary+Backend and > write > objectives > Ok, a quick review of that page is a good start; however, I seriously doubt we need to do an ORM (Django or otherwise). First-class objects would be far better than just dicts, though. > * What I considered a *major* showstopper (and why I think that the Conary > > PK backend is currently *fundamentally* broken is this: > > > > 1. When asking for updates, PK essentially gets the output from conary > > updateall --items. This is fine, and is what the command line does when > you > > run updateall. The output of this updateall is frozen to disk. This is > what > > is presented to the user as the list of packages that are in need of > > updating. > > > > the method for asking updates is get-updates and its executed by > gpk-icon-viewer > > This "frezze" its contains a Jobs this jobs are package:components what > will > updated. and showed... > > Ok, we're on the same page here. > > > > 2. When *applying* the updates, it then queries the repository *again* > > using the package names in the list! This results in a new updatejob, > which > > is then applied. This updatejob is (a) not the same as ther result from > > conary updateall --items, and (b) is not in optimal order for applying to > > the system. Therefore, most of the time, on large updates, the system is > > hosed after using PK to apply a set of updates. > > > > This its fixed! on gnome-packagekit. > the gnome-packagekit execute a get-updates and for each package do and > update on the past. NOW()! this phase replaced by update-system... > ref: http://bit.ly/amDZpj > > Alright, I have to admit I have not checked your changes to gnome-packagekit. That means that I guess a user is forced to take all changes. I suppose that's OK (and probably preferable since a Conary update is kind of an all or nothing proposition; if you change it by removing things, you need to re-resolve your entire update job, which was a problem when you could individually deselect packages when they were presented to the user after a get-updates action). > > > I had rectified this by freezing the updatejob *and* a normalized list of > > package objects (sorted) which was hashed. This hash -- not the > updatejob's > > trovelist -- was checked against a list built from what the GUI > persisted. > > If it matched, then the cached updatejob stored with that list was > applyed. > > This was faster, and was more accurate > > Unfortanately, I lost all of this work. :( It was *almost* a complete > > rewrite of 60% of the backend. That'll teach me. > > > > > I have to say the update-system as the same methods and operations what > updateall do. > > what as: > > client = ConaryClient(cfg) > updateItems = client.fullUpdateItemList() > applyList = [ (x[0], (None, None), x[1:], True) for x in updateItems ] > updJob = client.newUpdateJob() > sugMap = client.prepareUpdateJob(updJob,applyList) > dir = client.applyUpdateJob(updJob) > updJob.close() > > > this basically as conary updateall do... but with verbose callbacks > http://hg.rpath.com/conary/file/411f4e71df45/conary/callbacks.py#22 > > the unique difference is the fronzen dir for getting the updJob > > So the idea of conary backend do a update-system on a single Job its wrong. > Because internally the backends download jobs, commit,rollback and install > to system. This its showed on /tmp/conarybackend.log when packagekit > running.(this not show now because i remove the logs as embee explain ( > http://lists.rpath.org/pipermail/foresight-devel/2010-July/001801.html)) > > So if the packagekit backend do the same/equal of conary. i ask to my self > ¿what cause the memory usage? other think of conary backend haves and > conary > updateall not. It is a alot of loggers in code for debug on callbacks. > > So the on the last hack i remove all logs on Callbacks and seen a change on > my machine .. on other machine (eMbee not work) so i take care the memory > usage its not on logs. > > Yeah, logs are probably not the issue. > next i try to use profiler like python-gup...@fl:2-devel what seen what > cause the memory usage ... > > Good luck. Let me know what you find. --sgp _______________________________________________ Foresight-devel mailing list Foresight-devel@lists.rpath.org http://lists.rpath.org/mailman/listinfo/foresight-devel
