Re: [Evolution-hackers] massive e-d-s memory leak persuit ...

2008-01-28 Thread Srinivasa Ragavan
Meeks,

On Mon, 2008-01-28 at 11:27 +, Michael Meeks wrote:
> Hi Srini,
> 
> On Fri, 2008-01-25 at 00:00 +0530, Srinivasa Ragavan wrote:
> > Just check 
> > .evolution/addressbook//.summary -- Summary which is
> > in memory for fast autocompletion
> > and
> > .evolution/cache/addressbook///cache.db --Acutal
> > cache of contacts
> 
>   I have:
> 
> [EMAIL PROTECTED]:~/.evolution> find -name 'cache.db' -exec ls -lh "{}" \;
> -rw-r--r-- 1 michael users 25M 2007-12-07 06:14 ./cache/addressbook/[EMAIL 
> PROTECTED]/Novell GroupWise Address Book/cache.db
> -rw-r--r-- 1 michael users 12K 2007-04-19 16:32 ./cache/addressbook/[EMAIL 
> PROTECTED]/Frequent Contacts/cache.db
> -rw-r--r-- 1 michael users 12K 2007-04-19 16:32 ./cache/addressbook/[EMAIL 
> PROTECTED]/Michael Meeks/cache.db
> -rw-r--r-- 1 michael users 21M 2007-04-23 09:44 ./cache/addressbook/[EMAIL 
> PROTECTED]/Novell GroupWise Address Book/cache.db
> -rw-r--r-- 1 michael users 12K 2008-01-28 11:19 ./cache/addressbook/[EMAIL 
> PROTECTED]/Frequent Contacts/cache.db
> -rw-r--r-- 1 michael users 22M 2008-01-28 11:19 ./cache/addressbook/[EMAIL 
> PROTECTED]/Novell GroupWise Address Book/cache.db
> -rw-r--r-- 1 michael users 12K 2008-01-28 11:19 ./cache/addressbook/[EMAIL 
> PROTECTED]/Michael Meeks/cache.db
> [EMAIL PROTECTED]:~/.evolution> find -name '*.summary' -exec ls -lh "{}" \;
> -rw-r--r-- 1 michael users 2.2M 2007-09-06 22:31 ./addressbook/[EMAIL 
> PROTECTED]/Novell GroupWise Address Book.summary
> -rw-r--r-- 1 michael users 39K 2007-04-19 16:32 ./addressbook/[EMAIL 
> PROTECTED]/Frequent Contacts.summary
> -rw-r--r-- 1 michael users 1.7M 2007-04-23 10:03 ./addressbook/[EMAIL 
> PROTECTED]/Novell GroupWise Address Book.summary
> -rw-r--r-- 1 michael users 1.9K 2008-01-28 10:55 ./addressbook/local/[EMAIL 
> PROTECTED]/addressbook.db.summary
> -rw-r--r-- 1 michael users 120K 2008-01-28 11:04 ./addressbook/local/[EMAIL 
> PROTECTED]/addressbook.db.summary
> -rw-r--r-- 1 michael users 289K 2008-01-28 11:07 
> ./addressbook/local/system/addressbook.db.summary
> -rw-r--r-- 1 michael users 39K 2008-01-28 11:19 ./addressbook/[EMAIL 
> PROTECTED]/Frequent Contacts.summary
> -rw-r--r-- 1 michael users 1.7M 2008-01-28 11:17 ./addressbook/[EMAIL 
> PROTECTED]/Novell GroupWise Address Book.summary
> -rw--- 1 michael users 477 2006-07-10 12:29 ./mail/groupwise/[EMAIL 
> PROTECTED]/.summary
> -rw--- 1 michael users 481 2007-01-11 10:03 ./mail/groupwise/[EMAIL 
> PROTECTED]/.summary
> 
> > If you see these with groupwise, at times the summary will be more than
> > the cache. Which is what the revision fixes. Just check and lemme know.
> 
>   All my summaries are smaller than the caches it seems.
So I think it is all right for you. I think you must have used
10.3/later, which must have cleared the summary's old contacts.
> 
>   OTOH, several 20+Mb cache file is quite large, what lurks in there ?
> [ I guess if it's attachments etc. then fair enough, best to use the
> space ].
I guess that it is 6000+ Contacts and their complete addresses/details
like address/phone/etc. It may not have attachments, as the GW didn't
support it iirc.

Novell GroupWise Address Book:
total 24M
-rw-r--r-- 1 sragavan users 24M 2008-01-28 19:16 cache.db

Mine it says 24MB. I doubt, how is that 4MB difference for two people
from the same company. I think mine may have some stale/old contacts.

-Srini
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] massive e-d-s memory leak persuit ...

2008-01-28 Thread Michael Meeks
Hi Srini,

On Fri, 2008-01-25 at 00:00 +0530, Srinivasa Ragavan wrote:
> Just check 
> .evolution/addressbook//.summary -- Summary which is
> in memory for fast autocompletion
> and
> .evolution/cache/addressbook///cache.db --Acutal
> cache of contacts

I have:

[EMAIL PROTECTED]:~/.evolution> find -name 'cache.db' -exec ls -lh "{}" \;
-rw-r--r-- 1 michael users 25M 2007-12-07 06:14 ./cache/addressbook/[EMAIL 
PROTECTED]/Novell GroupWise Address Book/cache.db
-rw-r--r-- 1 michael users 12K 2007-04-19 16:32 ./cache/addressbook/[EMAIL 
PROTECTED]/Frequent Contacts/cache.db
-rw-r--r-- 1 michael users 12K 2007-04-19 16:32 ./cache/addressbook/[EMAIL 
PROTECTED]/Michael Meeks/cache.db
-rw-r--r-- 1 michael users 21M 2007-04-23 09:44 ./cache/addressbook/[EMAIL 
PROTECTED]/Novell GroupWise Address Book/cache.db
-rw-r--r-- 1 michael users 12K 2008-01-28 11:19 ./cache/addressbook/[EMAIL 
PROTECTED]/Frequent Contacts/cache.db
-rw-r--r-- 1 michael users 22M 2008-01-28 11:19 ./cache/addressbook/[EMAIL 
PROTECTED]/Novell GroupWise Address Book/cache.db
-rw-r--r-- 1 michael users 12K 2008-01-28 11:19 ./cache/addressbook/[EMAIL 
PROTECTED]/Michael Meeks/cache.db
[EMAIL PROTECTED]:~/.evolution> find -name '*.summary' -exec ls -lh "{}" \;
-rw-r--r-- 1 michael users 2.2M 2007-09-06 22:31 ./addressbook/[EMAIL 
PROTECTED]/Novell GroupWise Address Book.summary
-rw-r--r-- 1 michael users 39K 2007-04-19 16:32 ./addressbook/[EMAIL 
PROTECTED]/Frequent Contacts.summary
-rw-r--r-- 1 michael users 1.7M 2007-04-23 10:03 ./addressbook/[EMAIL 
PROTECTED]/Novell GroupWise Address Book.summary
-rw-r--r-- 1 michael users 1.9K 2008-01-28 10:55 ./addressbook/local/[EMAIL 
PROTECTED]/addressbook.db.summary
-rw-r--r-- 1 michael users 120K 2008-01-28 11:04 ./addressbook/local/[EMAIL 
PROTECTED]/addressbook.db.summary
-rw-r--r-- 1 michael users 289K 2008-01-28 11:07 
./addressbook/local/system/addressbook.db.summary
-rw-r--r-- 1 michael users 39K 2008-01-28 11:19 ./addressbook/[EMAIL 
PROTECTED]/Frequent Contacts.summary
-rw-r--r-- 1 michael users 1.7M 2008-01-28 11:17 ./addressbook/[EMAIL 
PROTECTED]/Novell GroupWise Address Book.summary
-rw--- 1 michael users 477 2006-07-10 12:29 ./mail/groupwise/[EMAIL 
PROTECTED]/.summary
-rw--- 1 michael users 481 2007-01-11 10:03 ./mail/groupwise/[EMAIL 
PROTECTED]/.summary

> If you see these with groupwise, at times the summary will be more than
> the cache. Which is what the revision fixes. Just check and lemme know.

All my summaries are smaller than the caches it seems.

OTOH, several 20+Mb cache file is quite large, what lurks in there ?
[ I guess if it's attachments etc. then fair enough, best to use the
space ].

Thanks,

Michael.

-- 
 [EMAIL PROTECTED]  <><, Pseudo Engineer, itinerant idiot


___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] massive e-d-s memory leak persuit ...

2008-01-24 Thread Srinivasa Ragavan
Hey Michael,

We are aware of some situations (I did that in addressbook) where the
memory built up can happen. They were fixed later to SLED/SP1 release
during late 2.12 and aren't in SLED but are part of trunk already.
Candidate for the next support pack. Chen is doing a extensive job of
profiling EDS currently for SLED/2.22 releases. 

Once thing is that the Groupwise Addressbook cache, on a update,
wouldn't clear the previous one and just appends everything. Which means
if you have 100 and you have a update of 10. Next time your cache would
have 210 where as ideally it should have just 110. Just to see, if this
is a culprit for you. Just move the
".evolution/cache/addressbook//Novell GroupWise Address Book"

some where else and make it resync (showdown evo/eds and start evo and
go to addressbook). If the sizes vary a lot, they are affected by that
syndrome. There were quite a few bits like these they were fixed post
SP1 and are in trunk/2.12 already.

GQuarks seems to be a nice idea. I will look at it in some time. Thanks
Meeks.

-Srini.


On Thu, 2008-01-24 at 17:46 +, Michael Meeks wrote:
> Hi dudie,
> 
>   So - I started to look at the e-d-s memory explosion situation quickly,
> took a nice dump from gdb, ran strings on it and the heap has a ton of
> strings around the place (as you would expect) - [ currently running at
> only ~60Mb
> 
> strings /tmp/eds-heap | sort | uniq -c | sort -n 
> 
> gives me:
> 
>1666 -CONTACT-UID
>1666 -NAME
>1736 ION-DEST-NAME
>1894 OLUTION-BOOK-URI
>2100 -EMAIL
>2184 ION-DEST-EMAIL
>2318 OLUTION-FILE-AS
>2506 OLUTION-LIST
>2992 ION-LIST
>3058 comp
>3321 OLUTION-DEST-EMAIL
>3329 OLUTION-DEST-CONTACT-UID
>3993 OLUTION-DEST-NAME
>4534 pwise://[EMAIL PROTECTED]/;Novell GroupWise Address Book
>5343 BEGIN:VCARD
>5372 ION-DEST-EMAIL
>5504 END:VCARD
>5505 VERSION:3.0
>6786 ION-DEST-NAME
>8606 para
>   12739 ION-DEST-CONTACT-UID
>   13642 OLUTION-DEST-CONTACT-UID
>   18082 OLUTION-DEST-NAME
>   19252 OLUTION-DEST-EMAIL
>   21991 prop
>   32508 ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;
>   40253 pA,
> 
> 
>   Where the first column is the count ... 32508 copies of that ATTENDEE
> string seems a little excessive, as do the (apparently mangled?)
> OLUTION-DEST-... strings.
> 
>   Does that provide any insight wrt. code to audit for this huge leak ?
> apparently it afflicts everything from SLED10-SP1 onwards. Also - in
> general to reduce the (high) e-d-s memory usage, should we be using
> GQuarks for some of these field names as we store them ?
> 
>   Thanks,
> 
>   Michael.
> 
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] massive e-d-s memory leak persuit ...

2008-01-24 Thread Ross Burton
Michael,

On Thu, 2008-01-24 at 17:46 +, Michael Meeks wrote:
>1666 -CONTACT-UID
>1666 -NAME
>1736 ION-DEST-NAME
>1894 OLUTION-BOOK-URI
>2100 -EMAIL
>2184 ION-DEST-EMAIL
>2318 OLUTION-FILE-AS
>2506 OLUTION-LIST
>2992 ION-LIST
>3058 comp
>3321 OLUTION-DEST-EMAIL
>3329 OLUTION-DEST-CONTACT-UID
>3993 OLUTION-DEST-NAME
>4534 pwise://[EMAIL PROTECTED]/;Novell GroupWise Address Book
>5343 BEGIN:VCARD
>5372 ION-DEST-EMAIL
>5504 END:VCARD
>5505 VERSION:3.0
>6786 ION-DEST-NAME
>8606 para
>   12739 ION-DEST-CONTACT-UID
>   13642 OLUTION-DEST-CONTACT-UID
>   18082 OLUTION-DEST-NAME
>   19252 OLUTION-DEST-EMAIL
>   21991 prop

The majority of those are from the contacts component, but because they
are truncated in some form I think they are left overs from previous
allocations.  I thought I'd fixed all of the major leaks in the
addressbook libraries, but maybe some new ones got introduced.  That
said the ATTENDEE field is from the calendar part, which I have never
really looked at.

If you can replicate this massive memory usage, could you run e-d-s in
massif?

Ross
-- 
Ross Burton mail: [EMAIL PROTECTED]
  jabber: [EMAIL PROTECTED]
 www: http://www.burtonini.com./
 PGP Fingerprint: 1A21 F5B0 D8D0 CFE3 81D4 E25A 2D09 E447 D0B4 33DF



signature.asc
Description: This is a digitally signed message part
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] massive e-d-s memory leak persuit ...

2008-01-24 Thread Srinivasa Ragavan
Michael,

http://svn.gnome.org/viewvc/evolution-data-server?view=revision&revision=7798
and
http://svn.gnome.org/viewvc/evolution-data-server?view=revision&revision=7797
were some of the issues I mentioned.

My previous statement was wrong. It wasn't the cache, that is appended,
but the summary of the contacts. 

Just check 
.evolution/addressbook//.summary -- Summary which is
in memory for fast autocompletion
and
.evolution/cache/addressbook///cache.db --Acutal
cache of contacts

If you see these with groupwise, at times the summary will be more than
the cache. Which is what the revision fixes. Just check and lemme know.

-Srini.

On Thu, 2008-01-24 at 17:46 +, Michael Meeks wrote:
> Hi dudie,
> 
>   So - I started to look at the e-d-s memory explosion situation quickly,
> took a nice dump from gdb, ran strings on it and the heap has a ton of
> strings around the place (as you would expect) - [ currently running at
> only ~60Mb
> 
> strings /tmp/eds-heap | sort | uniq -c | sort -n 
> 
> gives me:
> 
>1666 -CONTACT-UID
>1666 -NAME
>1736 ION-DEST-NAME
>1894 OLUTION-BOOK-URI
>2100 -EMAIL
>2184 ION-DEST-EMAIL
>2318 OLUTION-FILE-AS
>2506 OLUTION-LIST
>2992 ION-LIST
>3058 comp
>3321 OLUTION-DEST-EMAIL
>3329 OLUTION-DEST-CONTACT-UID
>3993 OLUTION-DEST-NAME
>4534 pwise://[EMAIL PROTECTED]/;Novell GroupWise Address Book
>5343 BEGIN:VCARD
>5372 ION-DEST-EMAIL
>5504 END:VCARD
>5505 VERSION:3.0
>6786 ION-DEST-NAME
>8606 para
>   12739 ION-DEST-CONTACT-UID
>   13642 OLUTION-DEST-CONTACT-UID
>   18082 OLUTION-DEST-NAME
>   19252 OLUTION-DEST-EMAIL
>   21991 prop
>   32508 ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;
>   40253 pA,
> 
> 
>   Where the first column is the count ... 32508 copies of that ATTENDEE
> string seems a little excessive, as do the (apparently mangled?)
> OLUTION-DEST-... strings.
> 
>   Does that provide any insight wrt. code to audit for this huge leak ?
> apparently it afflicts everything from SLED10-SP1 onwards. Also - in
> general to reduce the (high) e-d-s memory usage, should we be using
> GQuarks for some of these field names as we store them ?
> 
>   Thanks,
> 
>   Michael.
> 
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] massive e-d-s memory leak persuit ...

2008-01-24 Thread Philip Van Hoof
Hi there,

FYI (although most Evolution hackers already know this)

Note that Michael's memory analysis wont include the mailer component's
memory usage. The mailer component's memory is mostly consumed by
camel-folder-summary.c in Camel. Camel runs in the process of Evolution,
not in e-d-s.

This is mostly address and contact records, I'm assuming. Their memory
might also get copied to the UI's process (depending on the active
query).

btw, thanks Michael.


Cheers,


On Thu, 2008-01-24 at 17:46 +, Michael Meeks wrote:
> Hi dudie,
> 
>   So - I started to look at the e-d-s memory explosion situation quickly,
> took a nice dump from gdb, ran strings on it and the heap has a ton of
> strings around the place (as you would expect) - [ currently running at
> only ~60Mb
> 
> strings /tmp/eds-heap | sort | uniq -c | sort -n 
> 
> gives me:
> 
>1666 -CONTACT-UID
>1666 -NAME
>1736 ION-DEST-NAME
>1894 OLUTION-BOOK-URI
>2100 -EMAIL
>2184 ION-DEST-EMAIL
>2318 OLUTION-FILE-AS
>2506 OLUTION-LIST
>2992 ION-LIST
>3058 comp
>3321 OLUTION-DEST-EMAIL
>3329 OLUTION-DEST-CONTACT-UID
>3993 OLUTION-DEST-NAME
>4534 pwise://[EMAIL PROTECTED]/;Novell GroupWise Address Book
>5343 BEGIN:VCARD
>5372 ION-DEST-EMAIL
>5504 END:VCARD
>5505 VERSION:3.0
>6786 ION-DEST-NAME
>8606 para
>   12739 ION-DEST-CONTACT-UID
>   13642 OLUTION-DEST-CONTACT-UID
>   18082 OLUTION-DEST-NAME
>   19252 OLUTION-DEST-EMAIL
>   21991 prop
>   32508 ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;
>   40253 pA,
> 
> 
>   Where the first column is the count ... 32508 copies of that ATTENDEE
> string seems a little excessive, as do the (apparently mangled?)
> OLUTION-DEST-... strings.
> 
>   Does that provide any insight wrt. code to audit for this huge leak ?
> apparently it afflicts everything from SLED10-SP1 onwards. Also - in
> general to reduce the (high) e-d-s memory usage, should we be using
> GQuarks for some of these field names as we store them ?
> 
>   Thanks,
> 
>   Michael.
> 
-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be




___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers