Re: [Evolution-hackers] massive e-d-s memory leak persuit ...
Meeks,
On Mon, 2008-01-28 at 11:27 +, Michael Meeks wrote:
> Hi Srini,
>
> On Fri, 2008-01-25 at 00:00 +0530, Srinivasa Ragavan wrote:
> > Just check
> > .evolution/addressbook//.summary -- Summary which is
> > in memory for fast autocompletion
> > and
> > .evolution/cache/addressbook///cache.db --Acutal
> > cache of contacts
>
> I have:
>
> [EMAIL PROTECTED]:~/.evolution> find -name 'cache.db' -exec ls -lh "{}" \;
> -rw-r--r-- 1 michael users 25M 2007-12-07 06:14 ./cache/addressbook/[EMAIL
> PROTECTED]/Novell GroupWise Address Book/cache.db
> -rw-r--r-- 1 michael users 12K 2007-04-19 16:32 ./cache/addressbook/[EMAIL
> PROTECTED]/Frequent Contacts/cache.db
> -rw-r--r-- 1 michael users 12K 2007-04-19 16:32 ./cache/addressbook/[EMAIL
> PROTECTED]/Michael Meeks/cache.db
> -rw-r--r-- 1 michael users 21M 2007-04-23 09:44 ./cache/addressbook/[EMAIL
> PROTECTED]/Novell GroupWise Address Book/cache.db
> -rw-r--r-- 1 michael users 12K 2008-01-28 11:19 ./cache/addressbook/[EMAIL
> PROTECTED]/Frequent Contacts/cache.db
> -rw-r--r-- 1 michael users 22M 2008-01-28 11:19 ./cache/addressbook/[EMAIL
> PROTECTED]/Novell GroupWise Address Book/cache.db
> -rw-r--r-- 1 michael users 12K 2008-01-28 11:19 ./cache/addressbook/[EMAIL
> PROTECTED]/Michael Meeks/cache.db
> [EMAIL PROTECTED]:~/.evolution> find -name '*.summary' -exec ls -lh "{}" \;
> -rw-r--r-- 1 michael users 2.2M 2007-09-06 22:31 ./addressbook/[EMAIL
> PROTECTED]/Novell GroupWise Address Book.summary
> -rw-r--r-- 1 michael users 39K 2007-04-19 16:32 ./addressbook/[EMAIL
> PROTECTED]/Frequent Contacts.summary
> -rw-r--r-- 1 michael users 1.7M 2007-04-23 10:03 ./addressbook/[EMAIL
> PROTECTED]/Novell GroupWise Address Book.summary
> -rw-r--r-- 1 michael users 1.9K 2008-01-28 10:55 ./addressbook/local/[EMAIL
> PROTECTED]/addressbook.db.summary
> -rw-r--r-- 1 michael users 120K 2008-01-28 11:04 ./addressbook/local/[EMAIL
> PROTECTED]/addressbook.db.summary
> -rw-r--r-- 1 michael users 289K 2008-01-28 11:07
> ./addressbook/local/system/addressbook.db.summary
> -rw-r--r-- 1 michael users 39K 2008-01-28 11:19 ./addressbook/[EMAIL
> PROTECTED]/Frequent Contacts.summary
> -rw-r--r-- 1 michael users 1.7M 2008-01-28 11:17 ./addressbook/[EMAIL
> PROTECTED]/Novell GroupWise Address Book.summary
> -rw--- 1 michael users 477 2006-07-10 12:29 ./mail/groupwise/[EMAIL
> PROTECTED]/.summary
> -rw--- 1 michael users 481 2007-01-11 10:03 ./mail/groupwise/[EMAIL
> PROTECTED]/.summary
>
> > If you see these with groupwise, at times the summary will be more than
> > the cache. Which is what the revision fixes. Just check and lemme know.
>
> All my summaries are smaller than the caches it seems.
So I think it is all right for you. I think you must have used
10.3/later, which must have cleared the summary's old contacts.
>
> OTOH, several 20+Mb cache file is quite large, what lurks in there ?
> [ I guess if it's attachments etc. then fair enough, best to use the
> space ].
I guess that it is 6000+ Contacts and their complete addresses/details
like address/phone/etc. It may not have attachments, as the GW didn't
support it iirc.
Novell GroupWise Address Book:
total 24M
-rw-r--r-- 1 sragavan users 24M 2008-01-28 19:16 cache.db
Mine it says 24MB. I doubt, how is that 4MB difference for two people
from the same company. I think mine may have some stale/old contacts.
-Srini
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] massive e-d-s memory leak persuit ...
Hi Srini,
On Fri, 2008-01-25 at 00:00 +0530, Srinivasa Ragavan wrote:
> Just check
> .evolution/addressbook//.summary -- Summary which is
> in memory for fast autocompletion
> and
> .evolution/cache/addressbook///cache.db --Acutal
> cache of contacts
I have:
[EMAIL PROTECTED]:~/.evolution> find -name 'cache.db' -exec ls -lh "{}" \;
-rw-r--r-- 1 michael users 25M 2007-12-07 06:14 ./cache/addressbook/[EMAIL
PROTECTED]/Novell GroupWise Address Book/cache.db
-rw-r--r-- 1 michael users 12K 2007-04-19 16:32 ./cache/addressbook/[EMAIL
PROTECTED]/Frequent Contacts/cache.db
-rw-r--r-- 1 michael users 12K 2007-04-19 16:32 ./cache/addressbook/[EMAIL
PROTECTED]/Michael Meeks/cache.db
-rw-r--r-- 1 michael users 21M 2007-04-23 09:44 ./cache/addressbook/[EMAIL
PROTECTED]/Novell GroupWise Address Book/cache.db
-rw-r--r-- 1 michael users 12K 2008-01-28 11:19 ./cache/addressbook/[EMAIL
PROTECTED]/Frequent Contacts/cache.db
-rw-r--r-- 1 michael users 22M 2008-01-28 11:19 ./cache/addressbook/[EMAIL
PROTECTED]/Novell GroupWise Address Book/cache.db
-rw-r--r-- 1 michael users 12K 2008-01-28 11:19 ./cache/addressbook/[EMAIL
PROTECTED]/Michael Meeks/cache.db
[EMAIL PROTECTED]:~/.evolution> find -name '*.summary' -exec ls -lh "{}" \;
-rw-r--r-- 1 michael users 2.2M 2007-09-06 22:31 ./addressbook/[EMAIL
PROTECTED]/Novell GroupWise Address Book.summary
-rw-r--r-- 1 michael users 39K 2007-04-19 16:32 ./addressbook/[EMAIL
PROTECTED]/Frequent Contacts.summary
-rw-r--r-- 1 michael users 1.7M 2007-04-23 10:03 ./addressbook/[EMAIL
PROTECTED]/Novell GroupWise Address Book.summary
-rw-r--r-- 1 michael users 1.9K 2008-01-28 10:55 ./addressbook/local/[EMAIL
PROTECTED]/addressbook.db.summary
-rw-r--r-- 1 michael users 120K 2008-01-28 11:04 ./addressbook/local/[EMAIL
PROTECTED]/addressbook.db.summary
-rw-r--r-- 1 michael users 289K 2008-01-28 11:07
./addressbook/local/system/addressbook.db.summary
-rw-r--r-- 1 michael users 39K 2008-01-28 11:19 ./addressbook/[EMAIL
PROTECTED]/Frequent Contacts.summary
-rw-r--r-- 1 michael users 1.7M 2008-01-28 11:17 ./addressbook/[EMAIL
PROTECTED]/Novell GroupWise Address Book.summary
-rw--- 1 michael users 477 2006-07-10 12:29 ./mail/groupwise/[EMAIL
PROTECTED]/.summary
-rw--- 1 michael users 481 2007-01-11 10:03 ./mail/groupwise/[EMAIL
PROTECTED]/.summary
> If you see these with groupwise, at times the summary will be more than
> the cache. Which is what the revision fixes. Just check and lemme know.
All my summaries are smaller than the caches it seems.
OTOH, several 20+Mb cache file is quite large, what lurks in there ?
[ I guess if it's attachments etc. then fair enough, best to use the
space ].
Thanks,
Michael.
--
[EMAIL PROTECTED] <><, Pseudo Engineer, itinerant idiot
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] massive e-d-s memory leak persuit ...
Hey Michael, We are aware of some situations (I did that in addressbook) where the memory built up can happen. They were fixed later to SLED/SP1 release during late 2.12 and aren't in SLED but are part of trunk already. Candidate for the next support pack. Chen is doing a extensive job of profiling EDS currently for SLED/2.22 releases. Once thing is that the Groupwise Addressbook cache, on a update, wouldn't clear the previous one and just appends everything. Which means if you have 100 and you have a update of 10. Next time your cache would have 210 where as ideally it should have just 110. Just to see, if this is a culprit for you. Just move the ".evolution/cache/addressbook//Novell GroupWise Address Book" some where else and make it resync (showdown evo/eds and start evo and go to addressbook). If the sizes vary a lot, they are affected by that syndrome. There were quite a few bits like these they were fixed post SP1 and are in trunk/2.12 already. GQuarks seems to be a nice idea. I will look at it in some time. Thanks Meeks. -Srini. On Thu, 2008-01-24 at 17:46 +, Michael Meeks wrote: > Hi dudie, > > So - I started to look at the e-d-s memory explosion situation quickly, > took a nice dump from gdb, ran strings on it and the heap has a ton of > strings around the place (as you would expect) - [ currently running at > only ~60Mb > > strings /tmp/eds-heap | sort | uniq -c | sort -n > > gives me: > >1666 -CONTACT-UID >1666 -NAME >1736 ION-DEST-NAME >1894 OLUTION-BOOK-URI >2100 -EMAIL >2184 ION-DEST-EMAIL >2318 OLUTION-FILE-AS >2506 OLUTION-LIST >2992 ION-LIST >3058 comp >3321 OLUTION-DEST-EMAIL >3329 OLUTION-DEST-CONTACT-UID >3993 OLUTION-DEST-NAME >4534 pwise://[EMAIL PROTECTED]/;Novell GroupWise Address Book >5343 BEGIN:VCARD >5372 ION-DEST-EMAIL >5504 END:VCARD >5505 VERSION:3.0 >6786 ION-DEST-NAME >8606 para > 12739 ION-DEST-CONTACT-UID > 13642 OLUTION-DEST-CONTACT-UID > 18082 OLUTION-DEST-NAME > 19252 OLUTION-DEST-EMAIL > 21991 prop > 32508 ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION; > 40253 pA, > > > Where the first column is the count ... 32508 copies of that ATTENDEE > string seems a little excessive, as do the (apparently mangled?) > OLUTION-DEST-... strings. > > Does that provide any insight wrt. code to audit for this huge leak ? > apparently it afflicts everything from SLED10-SP1 onwards. Also - in > general to reduce the (high) e-d-s memory usage, should we be using > GQuarks for some of these field names as we store them ? > > Thanks, > > Michael. > ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] massive e-d-s memory leak persuit ...
Michael, On Thu, 2008-01-24 at 17:46 +, Michael Meeks wrote: >1666 -CONTACT-UID >1666 -NAME >1736 ION-DEST-NAME >1894 OLUTION-BOOK-URI >2100 -EMAIL >2184 ION-DEST-EMAIL >2318 OLUTION-FILE-AS >2506 OLUTION-LIST >2992 ION-LIST >3058 comp >3321 OLUTION-DEST-EMAIL >3329 OLUTION-DEST-CONTACT-UID >3993 OLUTION-DEST-NAME >4534 pwise://[EMAIL PROTECTED]/;Novell GroupWise Address Book >5343 BEGIN:VCARD >5372 ION-DEST-EMAIL >5504 END:VCARD >5505 VERSION:3.0 >6786 ION-DEST-NAME >8606 para > 12739 ION-DEST-CONTACT-UID > 13642 OLUTION-DEST-CONTACT-UID > 18082 OLUTION-DEST-NAME > 19252 OLUTION-DEST-EMAIL > 21991 prop The majority of those are from the contacts component, but because they are truncated in some form I think they are left overs from previous allocations. I thought I'd fixed all of the major leaks in the addressbook libraries, but maybe some new ones got introduced. That said the ATTENDEE field is from the calendar part, which I have never really looked at. If you can replicate this massive memory usage, could you run e-d-s in massif? Ross -- Ross Burton mail: [EMAIL PROTECTED] jabber: [EMAIL PROTECTED] www: http://www.burtonini.com./ PGP Fingerprint: 1A21 F5B0 D8D0 CFE3 81D4 E25A 2D09 E447 D0B4 33DF signature.asc Description: This is a digitally signed message part ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] massive e-d-s memory leak persuit ...
Michael, http://svn.gnome.org/viewvc/evolution-data-server?view=revision&revision=7798 and http://svn.gnome.org/viewvc/evolution-data-server?view=revision&revision=7797 were some of the issues I mentioned. My previous statement was wrong. It wasn't the cache, that is appended, but the summary of the contacts. Just check .evolution/addressbook//.summary -- Summary which is in memory for fast autocompletion and .evolution/cache/addressbook///cache.db --Acutal cache of contacts If you see these with groupwise, at times the summary will be more than the cache. Which is what the revision fixes. Just check and lemme know. -Srini. On Thu, 2008-01-24 at 17:46 +, Michael Meeks wrote: > Hi dudie, > > So - I started to look at the e-d-s memory explosion situation quickly, > took a nice dump from gdb, ran strings on it and the heap has a ton of > strings around the place (as you would expect) - [ currently running at > only ~60Mb > > strings /tmp/eds-heap | sort | uniq -c | sort -n > > gives me: > >1666 -CONTACT-UID >1666 -NAME >1736 ION-DEST-NAME >1894 OLUTION-BOOK-URI >2100 -EMAIL >2184 ION-DEST-EMAIL >2318 OLUTION-FILE-AS >2506 OLUTION-LIST >2992 ION-LIST >3058 comp >3321 OLUTION-DEST-EMAIL >3329 OLUTION-DEST-CONTACT-UID >3993 OLUTION-DEST-NAME >4534 pwise://[EMAIL PROTECTED]/;Novell GroupWise Address Book >5343 BEGIN:VCARD >5372 ION-DEST-EMAIL >5504 END:VCARD >5505 VERSION:3.0 >6786 ION-DEST-NAME >8606 para > 12739 ION-DEST-CONTACT-UID > 13642 OLUTION-DEST-CONTACT-UID > 18082 OLUTION-DEST-NAME > 19252 OLUTION-DEST-EMAIL > 21991 prop > 32508 ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION; > 40253 pA, > > > Where the first column is the count ... 32508 copies of that ATTENDEE > string seems a little excessive, as do the (apparently mangled?) > OLUTION-DEST-... strings. > > Does that provide any insight wrt. code to audit for this huge leak ? > apparently it afflicts everything from SLED10-SP1 onwards. Also - in > general to reduce the (high) e-d-s memory usage, should we be using > GQuarks for some of these field names as we store them ? > > Thanks, > > Michael. > ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] massive e-d-s memory leak persuit ...
Hi there, FYI (although most Evolution hackers already know this) Note that Michael's memory analysis wont include the mailer component's memory usage. The mailer component's memory is mostly consumed by camel-folder-summary.c in Camel. Camel runs in the process of Evolution, not in e-d-s. This is mostly address and contact records, I'm assuming. Their memory might also get copied to the UI's process (depending on the active query). btw, thanks Michael. Cheers, On Thu, 2008-01-24 at 17:46 +, Michael Meeks wrote: > Hi dudie, > > So - I started to look at the e-d-s memory explosion situation quickly, > took a nice dump from gdb, ran strings on it and the heap has a ton of > strings around the place (as you would expect) - [ currently running at > only ~60Mb > > strings /tmp/eds-heap | sort | uniq -c | sort -n > > gives me: > >1666 -CONTACT-UID >1666 -NAME >1736 ION-DEST-NAME >1894 OLUTION-BOOK-URI >2100 -EMAIL >2184 ION-DEST-EMAIL >2318 OLUTION-FILE-AS >2506 OLUTION-LIST >2992 ION-LIST >3058 comp >3321 OLUTION-DEST-EMAIL >3329 OLUTION-DEST-CONTACT-UID >3993 OLUTION-DEST-NAME >4534 pwise://[EMAIL PROTECTED]/;Novell GroupWise Address Book >5343 BEGIN:VCARD >5372 ION-DEST-EMAIL >5504 END:VCARD >5505 VERSION:3.0 >6786 ION-DEST-NAME >8606 para > 12739 ION-DEST-CONTACT-UID > 13642 OLUTION-DEST-CONTACT-UID > 18082 OLUTION-DEST-NAME > 19252 OLUTION-DEST-EMAIL > 21991 prop > 32508 ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION; > 40253 pA, > > > Where the first column is the count ... 32508 copies of that ATTENDEE > string seems a little excessive, as do the (apparently mangled?) > OLUTION-DEST-... strings. > > Does that provide any insight wrt. code to audit for this huge leak ? > apparently it afflicts everything from SLED10-SP1 onwards. Also - in > general to reduce the (high) e-d-s memory usage, should we be using > GQuarks for some of these field names as we store them ? > > Thanks, > > Michael. > -- Philip Van Hoof, freelance software developer home: me at pvanhoof dot be gnome: pvanhoof at gnome dot org http://pvanhoof.be/blog http://codeminded.be ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
