Re: [Evolution-hackers] libebook scalability
2007/4/2, Srinivasa Ragavan [EMAIL PROTECTED]: On Mon, 2007-04-02 at 01:12 +0200, Øystein Gisnås wrote: I discovered a bottleneck for addressbook performance with large addressbooks. Details at http://n800evolution.blogspot.com/2007/04/libebook-scalability.html Looks fine to commit. A proposed fix is attached. I'm not sure if order matters when returned from the backend? Does anyone know? If not, g_list_reverse can be omitted. Atleast here the order wont matter. Im sure that there are more such bottle necks. It will be nice task to take these up. Øystein? I'll look into this, but I can't fit it in very soon, so it's up to anyone to have a go.. I'd also love to create scripts, code and test data to test performance of some of the most important functions. Then we would be able to track performance over time in a more scientific way. Also, whether we're gonna use GQueues, GSLists, or reversed GLists is an interesting discussion. Does GNOME in general have any recommendations for that? Øystein ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libebook scalability
On Mon, 2007-04-02 at 01:12 +0200, Øystein Gisnås wrote: I discovered a bottleneck for addressbook performance with large addressbooks. Details at http://n800evolution.blogspot.com/2007/04/libebook-scalability.html A proposed fix is attached. I'm not sure if order matters when returned from the backend? Does anyone know? If not, g_list_reverse can be omitted. I also pushed a patch into bugzilla to do this, because eds-dbus has had that patch for a long time. You can omit the reverse as ordering from the backend isn't specified. Let me get dressed and so on and I'll file a bug with a set of similar optimisations in e-vcard (which, IIRC, is the biggest bottleneck once the server side book view optimisations landed). Ross -- Ross Burton mail: [EMAIL PROTECTED] jabber: [EMAIL PROTECTED] www: http://www.burtonini.com./ PGP Fingerprint: 1A21 F5B0 D8D0 CFE3 81D4 E25A 2D09 E447 D0B4 33DF signature.asc Description: This is a digitally signed message part ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libebook scalability
On Mon, 2007-04-02 at 09:03 +0200, Øystein Gisnås wrote: I'd also love to create scripts, code and test data to test performance of some of the most important functions. Then we would be able to track performance over time in a more scientific way. http://burtonini.com/bzr/eds-tests/ Check that out with bzr and you get a few tools: 1) a dummy backend for libedata-book. Ask for a contact and you get the same one back. As for a contact list and you'll always get the same 10. Ask for a book view and (mwhaha) you'll get 10 contacts. This makes profiling the EDS infrastructure easier as the backend has almost zero overhead. I should probably reduce the number of contacts returned in a book view as malloc tends to swamp the profiles now. 2) eds-bookview. A test application that will open and repeatedly request book views for a given number of times and URL. For example: $ eds-bookview --uri dummy:/// --repetition 10 --silent Will visibly do nothing for a few minutes but EDS will be very busy. Attach a profiler and come back 10 minutes later to discover that EVCard parsing is still primary bottle neck in eds-dbus. Ross -- Ross Burton mail: [EMAIL PROTECTED] jabber: [EMAIL PROTECTED] www: http://www.burtonini.com./ PGP Fingerprint: 1A21 F5B0 D8D0 CFE3 81D4 E25A 2D09 E447 D0B4 33DF signature.asc Description: This is a digitally signed message part ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libebook scalability
Ross, On Mon, 2007-04-02 at 09:11 +0100, Ross Burton wrote: On Mon, 2007-04-02 at 09:03 +0200, Øystein Gisnås wrote: I'd also love to create scripts, code and test data to test performance of some of the most important functions. Then we would be able to track performance over time in a more scientific way. http://burtonini.com/bzr/eds-tests/ This sounds really great. If you have the bugs/patches please post them across. -Srini. Check that out with bzr and you get a few tools: 1) a dummy backend for libedata-book. Ask for a contact and you get the same one back. As for a contact list and you'll always get the same 10. Ask for a book view and (mwhaha) you'll get 10 contacts. This makes profiling the EDS infrastructure easier as the backend has almost zero overhead. I should probably reduce the number of contacts returned in a book view as malloc tends to swamp the profiles now. 2) eds-bookview. A test application that will open and repeatedly request book views for a given number of times and URL. For example: $ eds-bookview --uri dummy:/// --repetition 10 --silent Will visibly do nothing for a few minutes but EDS will be very busy. Attach a profiler and come back 10 minutes later to discover that EVCard parsing is still primary bottle neck in eds-dbus. Ross ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libebook scalability
On Mon, 2007-04-02 at 10:18 +, Srinivasa Ragavan wrote: This sounds really great. If you have the bugs/patches please post them across. e-vcard.c | 494 +- 1 file changed, 232 insertions(+), 262 deletions(-) Urgh. Give me a minute or ten... Ross -- Ross Burton mail: [EMAIL PROTECTED] jabber: [EMAIL PROTECTED] www: http://www.burtonini.com./ PGP Fingerprint: 1A21 F5B0 D8D0 CFE3 81D4 E25A 2D09 E447 D0B4 33DF signature.asc Description: This is a digitally signed message part ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libebook scalability
On Mon, 2007-04-02 at 10:18 +, Srinivasa Ragavan wrote: This sounds really great. If you have the bugs/patches please post them across. http://bugzilla.gnome.org/show_bug.cgi?id=425464 Ross -- Ross Burton mail: [EMAIL PROTECTED] jabber: [EMAIL PROTECTED] www: http://www.burtonini.com./ PGP Fingerprint: 1A21 F5B0 D8D0 CFE3 81D4 E25A 2D09 E447 D0B4 33DF ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libebook scalability
2007/4/2, Ross Burton [EMAIL PROTECTED]: On Mon, 2007-04-02 at 09:03 +0200, Øystein Gisnås wrote: I'd also love to create scripts, code and test data to test performance of some of the most important functions. Then we would be able to track performance over time in a more scientific way. http://burtonini.com/bzr/eds-tests/ Check that out with bzr and you get a few tools: 1) a dummy backend for libedata-book. Ask for a contact and you get the same one back. As for a contact list and you'll always get the same 10. Ask for a book view and (mwhaha) you'll get 10 contacts. This makes profiling the EDS infrastructure easier as the backend has almost zero overhead. I should probably reduce the number of contacts returned in a book view as malloc tends to swamp the profiles now. 2) eds-bookview. A test application that will open and repeatedly request book views for a given number of times and URL. For example: $ eds-bookview --uri dummy:/// --repetition 10 --silent Will visibly do nothing for a few minutes but EDS will be very busy. Attach a profiler and come back 10 minutes later to discover that EVCard parsing is still primary bottle neck in eds-dbus. Very useful.. And a first hands-on bzr for me.. It indeed seems like the vCard is a bottleneck here. One thing is the CPU cycles for parsing that add up a bit on small datasets. For larger datasets, the size of the vCard itself adds up a lot. My test with 1*10^6 contacts made a 350MB on the disk, plus a 100MB summary. The same data in MySQL resulted in a 10MB database. Øystein ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] libebook scalability
I discovered a bottleneck for addressbook performance with large
addressbooks. Details at
http://n800evolution.blogspot.com/2007/04/libebook-scalability.html
A proposed fix is attached. I'm not sure if order matters when
returned from the backend? Does anyone know? If not, g_list_reverse
can be omitted.
Øystein
--- addressbook/backends/file/e-book-backend-file.c.orig 2007-04-02 00:42:20.871493559 +0200
+++ addressbook/backends/file/e-book-backend-file.c 2007-04-02 00:43:08.298457559 +0200
@@ -412,7 +412,7 @@
db_error = db-get (db, NULL, id_dbt, vcard_dbt, 0);
if (db_error == 0) {
-contact_list = g_list_append (contact_list, vcard_dbt.data);
+contact_list = g_list_prepend (contact_list, vcard_dbt.data);
} else {
g_warning (G_STRLOC : db-get failed with %d, db_error);
status = GNOME_Evolution_Addressbook_OtherError ;
@@ -451,7 +451,7 @@
|| strcmp (id_dbt.data, E_BOOK_BACKEND_FILE_VERSION_NAME)) {
if ((!search_needed) || (card_sexp != NULL e_book_backend_sexp_match_vcard (card_sexp, vcard_dbt.data))) {
- contact_list = g_list_append (contact_list, vcard_dbt.data);
+ contact_list = g_list_prepend (contact_list, vcard_dbt.data);
}
}
@@ -473,6 +473,7 @@
}
}
+ contact_list = g_list_reverse (contact_list);
*contacts = contact_list;
return status;
}
___
Evolution-hackers mailing list
Evolution-hackers@gnome.org
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libebook scalability
On Mon, 2007-04-02 at 01:12 +0200, Øystein Gisnås wrote: I discovered a bottleneck for addressbook performance with large addressbooks. Details at http://n800evolution.blogspot.com/2007/04/libebook-scalability.html A proposed fix is attached. I'm not sure if order matters when returned from the backend? Does anyone know? If not, g_list_reverse can be omitted. Nice catch! An audit of the code base for similar cases would definitely be helpful. In some cases it might be more convenient to use a GQueue, which provides O(1) appends. Matthew Barnes ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libebook scalability
On Mon, 2007-04-02 at 01:12 +0200, Øystein Gisnås wrote: I discovered a bottleneck for addressbook performance with large addressbooks. Details at http://n800evolution.blogspot.com/2007/04/libebook-scalability.html Looks fine to commit. A proposed fix is attached. I'm not sure if order matters when returned from the backend? Does anyone know? If not, g_list_reverse can be omitted. Atleast here the order wont matter. Im sure that there are more such bottle necks. It will be nice task to take these up. Øystein? -Srini. Øystein ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
