Bug#431862: nss-updatedb: db updates are not atomic
Hi Stephen, On Sat, Aug 11, 2007 at 05:28:28PM +0200, Guido Guenther wrote: On Sat, Aug 11, 2007 at 02:20:22PM +0100, Stephen Gran wrote: works rather well and solves this problem but has one major drawback: we can't specify the DB_RECOVER flag here since this would conflict with the DB_RECOVER flag of nss_updatedb. This could leave us with an inconsistent database in case an applications aborts leaving an unrecovered database for all other apps. Any ideas on howto fix this? Could you please try the attached patch against nss-updatedb on alioth? Renaming the db via dbenv-rename isn't atomic but keeps us out of the db_env trouble and is a huge improvement (and as a bouns we don't have to mess with crappy libnss-db). With this I didn't see any lookup failures. Did you find any time to try this yet? Cheers, -- Guido signature.asc Description: Digital signature
Bug#431862: nss-updatedb: db updates are not atomic
On Mon, Jul 23, 2007 at 04:01:05PM +0100, Stephen Gran wrote: This one time, at band camp, Guido Guenther said: I can cook up a patch during the week if you like, I'd be great having something to test this on though, maybe a chroot or machine that can access alioth's database? I can give you access to a read only sanitized view of the database, if that would be helpful. A patch would be great. Maybe the libnss-db maintainer can should comment on this? Yes, that would probably also be helpful. The problem is actually as suspected. libnss-db doesn't use the database environment so it doesn't actually know that the database is currently being truncated/updated which makes the user/group lookups fail. Actually using the dbenv like rc = dbp-dbenv-open(dbp-dbenv, DB_DIR, DB_CREATE|DB_INIT_LOG|DB_INIT_LOCK|DB_INIT_MPOOL| DB_INIT_TXN, 0644); ... works rather well and solves this problem but has one major drawback: we can't specify the DB_RECOVER flag here since this would conflict with the DB_RECOVER flag of nss_updatedb. This could leave us with an inconsistent database in case an applications aborts leaving an unrecovered database for all other apps. Any ideas on howto fix this? Cheers, -- Guido P.S.: could you please insstall db4.3-utils on alioth, this would ease debugging. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#431862: nss-updatedb: db updates are not atomic
This one time, at band camp, Guido Guenther said: rc = dbp-dbenv-open(dbp-dbenv, DB_DIR, DB_CREATE|DB_INIT_LOG|DB_INIT_LOCK|DB_INIT_MPOOL| DB_INIT_TXN, 0644); ... works rather well and solves this problem but has one major drawback: we can't specify the DB_RECOVER flag here since this would conflict with the DB_RECOVER flag of nss_updatedb. This could leave us with an inconsistent database in case an applications aborts leaving an unrecovered database for all other apps. Any ideas on howto fix this? Sorry, no idea. P.S.: could you please insstall db4.3-utils on alioth, this would ease debugging. Installed. Thanks again, -- - | ,''`.Stephen Gran | | : :' :[EMAIL PROTECTED] | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Bug#431862: nss-updatedb: db updates are not atomic
Hi,
On Sat, Aug 11, 2007 at 02:20:22PM +0100, Stephen Gran wrote:
works rather well and solves this problem but has one major drawback: we
can't specify the DB_RECOVER flag here since this would conflict with
the DB_RECOVER flag of nss_updatedb. This could leave us with an
inconsistent database in case an applications aborts leaving an
unrecovered database for all other apps. Any ideas on howto fix this?
Could you please try the attached patch against nss-updatedb on alioth?
Renaming the db via dbenv-rename isn't atomic but keeps us out of the
db_env trouble and is a huge improvement (and as a bouns we don't have
to mess with crappy libnss-db). With this I didn't see any lookup
failures.
Cheers,
-- Guido
diff --git a/cache.c b/cache.c
index 65f48ff..c8ef3eb 100644
--- a/cache.c
+++ b/cache.c
@@ -32,6 +32,7 @@ struct nss_cache {
DB *db;
int index;
#if DB_VERSION_MAJOR = 4
+ char *tmpname;
DB_ENV *dbenv;
DB_TXN *dbtxn;
#endif
@@ -63,6 +64,13 @@ enum nss_status nss_cache_init(const char *filename,
cache-dbenv = NULL;
cache-dbtxn = NULL;
+ rc = asprintf(cache-tmpname, %s TMP_EXT, cache-filename);
+ if (rc == -1) {
+ nss_cache_close(cache);
+ errno = rc;
+ return NSS_STATUS_TRYAGAIN;
+ }
+
rc = db_env_create(cache-dbenv, 0);
if (rc != 0) {
nss_cache_close(cache);
@@ -97,7 +105,7 @@ enum nss_status nss_cache_init(const char *filename,
}
rc = cache-db-open(cache-db,0,
- cache-filename,NULL,
+ cache-tmpname, NULL,
DB_BTREE,DB_CREATE|DB_AUTO_COMMIT, mode);
if (rc != 0) {
nss_cache_close(cache);
@@ -407,6 +415,18 @@ enum nss_status nss_cache_commit(nss_cache_t *cache)
return NSS_STATUS_UNAVAIL;
}
+#if DB_VERSION_MAJOR = 4
+ cache-db-close(cache-db, 0);
+ cache-db = NULL; /* not usable anymore */
+ unlink(cache-filename);
+ rc = cache-dbenv-dbrename(cache-dbenv, NULL, cache-tmpname,
+ NULL, cache-filename, 0);
+ if (rc != 0) {
+ errno = rc;
+ return NSS_STATUS_UNAVAIL;
+ }
+#endif
+
return NSS_STATUS_SUCCESS;
}
@@ -436,6 +456,8 @@ enum nss_status nss_cache_close(nss_cache_t **cache_p)
cache-db-close(cache-db, 0);
if (cache-dbenv != NULL)
cache-dbenv-close(cache-dbenv, 0);
+ if (cache-tmpname != NULL)
+ free(cache-tmpname);
#endif
if (cache-filename != NULL)
free(cache-filename);
diff --git a/cache.h b/cache.h
index 871db45..9121c0f 100644
--- a/cache.h
+++ b/cache.h
@@ -20,6 +20,8 @@
#include pwd.h
#include grp.h
+#define TMP_EXT .tmp
+
struct nss_cache;
typedef struct nss_cache nss_cache_t;
diff --git a/updatedb.h b/updatedb.h
Bug#431862: nss-updatedb: db updates are not atomic
On Fri, Aug 03, 2007 at 11:54:08AM +0200, Guido Guenther said: Hi Stephen, On Mon, Jul 23, 2007 at 04:01:05PM +0100, Stephen Gran wrote: This one time, at band camp, Guido Guenther said: I can cook up a patch during the week if you like, I'd be great having something to test this on though, maybe a chroot or machine that can access alioth's database? I can give you access to a read only sanitized view of the database, if that would be helpful. A patch would be great. This is basically just a short note that I haven't totally forgotten about this. I hope to finish things up next week. Sorry for the delay. No problem, and thanks again for your help. If you need a sanitized view of the database, ping me on irc (sgran, I'm usually idling in #alioth). Cheers, -- -- | Stephen Gran | BOFH excuse #21: POSIX compliance | | [EMAIL PROTECTED] | problem | | http://www.lobefin.net/~steve | | -- signature.asc Description: Digital signature
Bug#431862: nss-updatedb: db updates are not atomic
This one time, at band camp, Guido Guenther said: I can cook up a patch during the week if you like, I'd be great having something to test this on though, maybe a chroot or machine that can access alioth's database? I can give you access to a read only sanitized view of the database, if that would be helpful. A patch would be great. Maybe the libnss-db maintainer can should comment on this? Yes, that would probably also be helpful. Sorry for always being the pathologic nss case, but thanks for all your help :) -- - | ,''`.Stephen Gran | | : :' :[EMAIL PROTECTED] | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Bug#431862: nss-updatedb: db updates are not atomic
Hi Stephen, On Sat, Jul 21, 2007 at 08:51:30PM +0100, Stephen Gran wrote: This one time, at band camp, Guido Guenther said: Hi Stephen, On Thu, Jul 05, 2007 at 02:40:46PM +, Stephen Gran wrote: We have been using nss-updatedb on alioth (a slightly patched version to temporarily work around a bug in nss-pgsql) and we have noticed that the database updates are not atomic. We frequently get exim panicking that it can't find a user it has just decided to suid to, or svn commits failing, or that sort of thing. Did you check the return codes of the bdb calls in libnss-db when things fail? How do I do that? Pointers to docs would be welcome. I'm happy to write the patch and so on, but I'm just not sure what I'm looking for. Have a look how the database is handled in libnss-db (internal_setent in src/db-open.c). It's simply opened without database environment. Compare this to nss_cache_init for the DB_VERSION_MAJOR =4 case in cache.c of nss-updatedb: it first opens the db_environment, creates a db handle and starts the transaction (txn_begin). So (what I think) we should do is to protect the db-get calls in lookup() of libnss-db by transactions and create a proper database environment in internal_setent(). This would (at least I think so) make sure libnss-db sees the old database content until the commit of the new content is finished. Docs are at: http://www.oracle.com/technology/documentation/berkeley-db/db/index.html Especially the Getting started with transaction processing: http://www.oracle.com/technology/documentation/berkeley-db/db/gsg_txn/C/index.html I can cook up a patch during the week if you like, I'd be great having something to test this on though, maybe a chroot or machine that can access alioth's database? Maybe the libnss-db maintainer can should comment on this? Cheers, -- Guido -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#431862: nss-updatedb: db updates are not atomic
This one time, at band camp, Guido Guenther said: Hi Stephen, On Thu, Jul 05, 2007 at 02:40:46PM +, Stephen Gran wrote: We have been using nss-updatedb on alioth (a slightly patched version to temporarily work around a bug in nss-pgsql) and we have noticed that the database updates are not atomic. We frequently get exim panicking that it can't find a user it has just decided to suid to, or svn commits failing, or that sort of thing. Did you check the return codes of the bdb calls in libnss-db when things fail? How do I do that? Pointers to docs would be welcome. I'm happy to write the patch and so on, but I'm just not sure what I'm looking for. The operation in nssupdate-db is transaction protected so we _should_ be able to make the updates look atomic without having to resort to poor copy over the database methods. Help on this would be welcome since I'm far from a BDB expert. Me either, sadly. Maybe we can help each other limp through this one :) -- - | ,''`.Stephen Gran | | : :' :[EMAIL PROTECTED] | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Bug#431862: nss-updatedb: db updates are not atomic
Hi Stephen, On Thu, Jul 05, 2007 at 02:40:46PM +, Stephen Gran wrote: We have been using nss-updatedb on alioth (a slightly patched version to temporarily work around a bug in nss-pgsql) and we have noticed that the database updates are not atomic. We frequently get exim panicking that it can't find a user it has just decided to suid to, or svn commits failing, or that sort of thing. Did you check the return codes of the bdb calls in libnss-db when things fail? The operation in nssupdate-db is transaction protected so we _should_ be able to make the updates look atomic without having to resort to poor copy over the database methods. Help on this would be welcome since I'm far from a BDB expert. Cheers, -- Guido -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#431862: nss-updatedb: db updates are not atomic
Package: nss-updatedb Version: 7-1.1 Severity: important Hi there, We have been using nss-updatedb on alioth (a slightly patched version to temporarily work around a bug in nss-pgsql) and we have noticed that the database updates are not atomic. We frequently get exim panicking that it can't find a user it has just decided to suid to, or svn commits failing, or that sort of thing. I'd like to work with you to resolve this - my first thought is to switch to writing files in a temporary subdirectory, and then move them across - while that doesn't eliminate the race condition, it should lessen it. Take care, -- System Information: Debian Release: 4.0 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Shell: /bin/sh linked to /bin/bash Kernel: Linux 2.6.18-4-xen-amd64 Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.utf8) Versions of packages nss-updatedb depends on: ii libc6 2.3.6.ds1-13 GNU C Library: Shared libraries ii libdb4.34.3.29-8 Berkeley v4.3 Database Libraries [ Versions of packages nss-updatedb recommends: ii libnss-db2.2.3pre1-2 NSS module for using Berkeley Data -- no debconf information -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
