Re: [Evolution-hackers] libdb performance issue? (was: Re: libebook: errors when using asynchronous contact addition/removal functions)
On Mo, 2011-05-16 at 13:05 +0200, Milan Crha wrote: > On Mon, 2011-05-16 at 11:35 +0200, Patrick Ohly wrote: > > And it works beautifully. One word added to the source code and the > > stress test passes reliably and quickly :-) > > > > Patch attached. Okay to submit into master (not tested there, though) > > and gnome-2-32 branches (which is where I have tested it)? > > Yes please, master, gnome-3-0 and if you want then gnome-2-32 too. > Thanks. Okay, done. -- Bye, Patrick Ohly -- patrick.o...@gmx.de http://www.estamos.de/ ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libdb performance issue? (was: Re: libebook: errors when using asynchronous contact addition/removal functions)
On Mon, 2011-05-16 at 11:35 +0200, Patrick Ohly wrote: > And it works beautifully. One word added to the source code and the > stress test passes reliably and quickly :-) > > Patch attached. Okay to submit into master (not tested there, though) > and gnome-2-32 branches (which is where I have tested it)? Yes please, master, gnome-3-0 and if you want then gnome-2-32 too. Thanks. Milan ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libdb performance issue? (was: Re: libebook: errors when using asynchronous contact addition/removal functions)
[Milan, can you please keep other people on CC?
I know that you personally prefer to not be CCed on mailing list emails
and I try to remember that in my own replies, but others are not on the
list and may depend on being CCed. For example, I don't know whether
Chris is subscribed.]
On Mo, 2011-05-16 at 11:12 +0200, Patrick Ohly wrote:
> It might be easier to set DB_INIT_CDB, which enforces multiple
> reads/single writer access without deadlocks. I'll give that a try.
And it works beautifully. One word added to the source code and the
stress test passes reliably and quickly :-)
Patch attached. Okay to submit into master (not tested there, though)
and gnome-2-32 branches (which is where I have tested it)?
--
Bye, Patrick Ohly
--
patrick.o...@gmx.de
http://www.estamos.de/
>From 2b4cff9230a049d5ad7fe86873aba9a7bba35af1 Mon Sep 17 00:00:00 2001
From: Patrick Ohly
Date: Mon, 16 May 2011 11:21:04 +0200
Subject: [PATCH] addressbook file backend: libdb must be initialized for concurrent read/write
Very bad performance (100% CPU load, several minutes run time) were
seen for multiple concurrent writes. gdb shows that libdb is
apparently busy polling while writing.
The libdb API docs for DB_ENV->open() imply that either DB_INIT_CDB or
DB_INIT_LOCK must be used in apps which are not read-only, like
EDS. This patch adds DB_INIT_CDB because it is simple and fixes the
performance problem.
In some rare cases, DB_INIT_LOCK might provide better performance by
allowing concurrent writes of independent data, but that seems too
complicated for not enough gain right now (must check for deadlocks).
---
addressbook/backends/file/e-book-backend-file.c | 13 -
1 files changed, 12 insertions(+), 1 deletions(-)
diff --git a/addressbook/backends/file/e-book-backend-file.c b/addressbook/backends/file/e-book-backend-file.c
index d21c527..372ade5 100644
--- a/addressbook/backends/file/e-book-backend-file.c
+++ b/addressbook/backends/file/e-book-backend-file.c
@@ -1146,7 +1146,18 @@ e_book_backend_file_load_source (EBookBackend *backend,
(gpointer (*)(gpointer , gsize))g_try_realloc,
g_free);
- db_error = (*env->open) (env, NULL, DB_CREATE | DB_INIT_MPOOL | DB_PRIVATE | DB_THREAD, 0);
+ /*
+ * We need either DB_INIT_CDB or DB_INIT_LOCK, because we will have
+ * multiple threads reading and writing concurrently without
+ * any locking above libdb.
+ *
+ * DB_INIT_CDB enforces multiple reader/single writer by locking inside
+ * the database. It is used instead of DB_INIT_LOCK because DB_INIT_LOCK
+ * may deadlock, which would have to be called in a separate thread.
+ * Considered too complicated for not enough gain (= concurrent writes)
+ * at this point.
+ */
+ db_error = (*env->open) (env, NULL, DB_INIT_CDB | DB_CREATE | DB_INIT_MPOOL | DB_PRIVATE | DB_THREAD, 0);
if (db_error != 0) {
env->close(env, 0);
g_warning ("db_env_open failed with %s", db_strerror (db_error));
--
1.7.2.5
___
evolution-hackers mailing list
evolution-hackers@gnome.org
To change your list options or unsubscribe, visit ...
http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libdb performance issue? (was: Re: libebook: errors when using asynchronous contact addition/removal functions)
On Mo, 2011-05-16 at 08:11 +0200, Milan Crha wrote: > On Fri, 2011-05-13 at 15:44 +0200, Patrick Ohly wrote: > > In libebook, I get a "Timeout was reached" because the asynchronous > > operation doesn't complete quickly enough. Same for the attempt to > > delete the contacts. The gError->code is 24, which is indeed > > E_BOOK_ERROR_NOT_SUPPORTED. > > Hi, > as long as it's GError, you cannot compare only GError::code, because > the correct way to identify the error is using the whole pair > GError::domain and GError::code. Your domain for "Timeout was reached" > is most likely G_IO_ERROR with a code G_IO_ERROR_TIMED_OUT. Indeed, that explains it. In principle I know that, I just forgot to check in this case ;-} > The new eclient stuff will not suffer of this (unless some backend will > block main thread for too long). The EClient may not time out, but taking several minutes of 100% CPU time just to safe a handful of contacts just can't be right. > I've no idea about libdb, thus only referring to the error. I have one > notice though, the EBook calls are served and processed mostly as soon > as they come, and they can come "in parallel" too. The backend itself > may make sure that it'll not make any trouble, like by some busy lock. libdb is supposed to be thread-safe, so in theory shouldn't need such a lock. Either libdb is broken or EDS is not using it correctly (wrong environment, for example). Looking at the DB API [1] I see this flag for opening the environment: DB_INIT_LOCK Initialize the locking subsystem. This subsystem should be used when multiple processes or threads are going to be reading and writing a Berkeley DB database, so that they do not interfere with each other. If all threads are accessing the database(s) read-only, locking is unnecessary. When the DB_INIT_LOCK flag is specified, it is usually necessary to run a deadlock detector, as well. See db_deadlock and DB_ENV->lock_detect() for more information. EDS does not pass this flag, although it has concurrent reads *and* writes. The behavior in that case is undefined as far as I can tell. The downside of using DB_INIT_LOCK is that deadlocks are possible and need to be checked by regularly calling lock_detect(). It might be easier to set DB_INIT_CDB, which enforces multiple reads/single writer access without deadlocks. I'll give that a try. [1] http://download.oracle.com/docs/cd/E17076_02/html/api_reference/C/envopen.html -- Bye, Patrick Ohly -- patrick.o...@gmx.de http://www.estamos.de/ ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libdb performance issue? (was: Re: libebook: errors when using asynchronous contact addition/removal functions)
On Fri, 2011-05-13 at 15:44 +0200, Patrick Ohly wrote: > In libebook, I get a "Timeout was reached" because the asynchronous > operation doesn't complete quickly enough. Same for the attempt to > delete the contacts. The gError->code is 24, which is indeed > E_BOOK_ERROR_NOT_SUPPORTED. Hi, as long as it's GError, you cannot compare only GError::code, because the correct way to identify the error is using the whole pair GError::domain and GError::code. Your domain for "Timeout was reached" is most likely G_IO_ERROR with a code G_IO_ERROR_TIMED_OUT. This means, for GDBus calls, that the call didn't finish in a default time, the server side didn't reply to the method invocation on time. (As Patrick suggested too). The new eclient stuff will not suffer of this (unless some backend will block main thread for too long). As a workaround, in 3.0, was added e_data_server_util_set_dbus_call_timeout(), which can be called by clients. I've no idea about libdb, thus only referring to the error. I have one notice though, the EBook calls are served and processed mostly as soon as they come, and they can come "in parallel" too. The backend itself may make sure that it'll not make any trouble, like by some busy lock. The sync calendar backend does this differently, it has a flag for using such internal busy lock, thus no call can overlap with another, if the backend implementation wishes to do so. I guess this kind of functionality might be useful when added to EBookBackendSync too. Bye, Milan ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] libdb performance issue? (was: Re: libebook: errors when using asynchronous contact addition/removal functions)
On Fri, May 13, 2011 at 03:44:08PM +0200, Patrick Ohly wrote: > I tried the LD_PRELOAD=libeatmydata.so workaround suggested in the mail > above and it does avoid the problem. If eatmydata removes the bottleneck, then it's likely that either (a) each operation is corresponding to an fsync/fsyncdata/sync call (i.e. each update corresponds to some kind of commit/flush), or (b) the database file is opened O_SYNC, resulting in something similar. I don't know eds or libdb well enough to know if eds is doing something out of the ordinary with the api and/or whether this is "working as designed" and/or whether there's a way to remove the synchronous behavior. > Is there anyone around who understand libdb well enough to shed some > light on this? What is a proper fix? Again from the armchair here (have not even looked at the code, feel free to LART me as a result), if there's no way to avoid the synchronous writes with the libdb api, i guess you could consider batching the async additions together and submitting them as a group? Alternatively, maybe this is another argument for moving forward with the sqlite backend support? :) sean ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... http://mail.gnome.org/mailman/listinfo/evolution-hackers
