Re: About REP_TEXT_MUSTBEFREED (ITS#6138)

2021-07-30 Thread Hallvard Breien Furuseth
On 30.07.2021 17:39, Quanah Gibson-Mount wrote: (I tried to add a comment in Github, but that didn't seem to work, so mailing here instead.) We don't use github.  I assume you mean the gitlab instance? Oops, right. What error did you get?  Were you logged in? Logged in, but my

About REP_TEXT_MUSTBEFREED (ITS#6138)

2021-07-30 Thread Hallvard Breien Furuseth
REP_TEXT_MUSTBEFREED looks like a bad idea, at lest for now: Errors can happen anywhere, and then sr_text is usually set to a string constant. If REP_TEXT_MUSTBEFREED is already set, slapd will then later try to free that constant. To add this flag, first add result.c:rs_replace_text() or

Some suggested 2.5 code cleanups/renames

2021-03-23 Thread Hallvard Breien Furuseth
Hello. slap_op_time(time_t *t, int *nop) could previously be called with any 2 arg of those types. But now it modifies nop[1], so old code other than slap_op_time(>o_time, >o_tincr) gets a bad memory write. Source-code compat looks like a misfeature now. I'd rather rename and re-prototype:

Re: Logging TLS connection parameters

2019-01-14 Thread Hallvard Breien Furuseth
On 14.01.2019 18:55, Quanah Gibson-Mount wrote: --On Monday, January 14, 2019 6:44 PM +0100 Hallvard Breien Furuseth wrote: I'd like TLS version/cipher logged with "TLS established", so I've cherry-picked these commits to our RE24 installation:   7d6d694 ITS#7683 log tls prot/c

Logging TLS connection parameters

2019-01-14 Thread Hallvard Breien Furuseth
I'd like TLS version/cipher logged with "TLS established", so I've cherry-picked these commits to our RE24 installation: 7d6d694 ITS#7683 log tls prot/cipher info 0045e56 ITS#7683 more for tls version/cipher info Also this looks like it should go with them, though we use OpenSSL: 7a7d941

Re: Increase default olcLocalSSF to 128

2018-07-26 Thread Hallvard Breien Furuseth
I wrote: (...) any particular value will be wrong for someone. Depends on how safe your filesystem setup is and whether it's easier to break in to get at the ldapi socket than it is to just attack slapd. I forgot: You could forge ldapi: credentials in early OpenLDAP versions, depending on

Re: Increase default olcLocalSSF to 128

2018-07-26 Thread Hallvard Breien Furuseth
On 26. juli 2018 09:04, Dieter Klünter wrote: Am Thu, 26 Jul 2018 08:19:34 +0200 schrieb Michael Ströder : On 07/26/2018 04:47 AM, Ryan Tandy wrote: I propose increasing the default olcLocalSSF to 128. Mentioned initially on IRC, now bringing it to the list for completeness and archival. In

Re: LMDB: issue with mdb_cursor_del

2017-10-17 Thread Hallvard Breien Furuseth
On 16/10/17 14:56, Hallvard Breien Furuseth wrote: On 16. okt. 2017 13:43, Леонид Юрьев wrote: (...) Inside freelist_save() the mt_loose_pages could be merged into the mt_free_pgs. http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=libraries/liblmdb/mdb.c;h

Re: LMDB: issue with mdb_cursor_del

2017-10-16 Thread Hallvard Breien Furuseth
On 16. okt. 2017 12:51, Howard Chu wrote: timur.kris...@gmail.com wrote: I have an app that uses LMDB, and I've experienced an interesting issue: when trying to delete a certain item with mdb_cursor_del, it crashed with the following backtrace: https://pastebin.com/7p9wtkj9 Weird backtrace.

Re: ITS review 9/29/2017

2017-10-06 Thread Hallvard Breien Furuseth
On 06/10/17 23:24, Quanah Gibson-Mount wrote: --On Friday, October 06, 2017 10:16 PM +0100 Howard Chu wrote: its7442 - Add debug statements when index_intlen values are out of range Looks pointless. Well, the man

C90 support (was: Variadic Debug() macro)

2017-07-14 Thread Hallvard Breien Furuseth
On 14. juli 2017 17:23, Ondřej Kuzník wrote: I think it is time to modernise the debug logging a bit, this would make our life easier when logging and running make (with the copious warnings compilers will generate unless one passes -Wno-format-extra-args). The major obstacle to this is the

Re: openldap.git branch master updated. e12ca8b6fed6b8a2526c5c8ee820bf5aa942b59d

2017-03-15 Thread Hallvard Breien Furuseth
On 15. mars 2017 14:27, Howard Chu wrote: Not surprising when it was hashing the address instead of contents of a local variable. Whatever that intended to achieve, it looks to me like rand_r() would have done the same but better. It was hashing the bytes of arg. In the common case of the

Re: openldap.git branch master updated. e12ca8b6fed6b8a2526c5c8ee820bf5aa942b59d

2017-03-15 Thread Hallvard Breien Furuseth
On 15. mars 2017 12:15, openldap-commit2de...@openldap.org wrote: commit e12ca8b6fed6b8a2526c5c8ee820bf5aa942b59d Author: Howard Chu Date: Wed Mar 15 11:13:09 2017 + Fixes for multiple threadpool queues Remove poolq_hash, it wasn't distributing work evenly to

Re: When must slap_callback.sc_writewait be set?

2016-06-09 Thread Hallvard Breien Furuseth
On 08. juni 2016 23:55, Howard Chu wrote: Hallvard Breien Furuseth wrote: I lost this thread (from May 19). Summarized in ITS#8435 plus a new suggestion: Revert sc_writewait and get it done with a field somewhere else, maintained by callbacks. Dunno if that's feasible. There is no "some

Re: When must slap_callback.sc_writewait be set?

2016-05-19 Thread Hallvard Breien Furuseth
I wrote: > Oh, right... New problem, though: If a a third-party module sets > sc_private in such an initializer, that now sets sc_writewait instead. I mean, and old module which predates sc_writewait. -- Hallvard

When must slap_callback.sc_writewait be set?

2016-05-19 Thread Hallvard Breien Furuseth
When is it safe to leave sc_writewait uninitialized? The field seems to have emerged in the middle of RE24, and I don't see any flags which say it should/should not be used. Since not setting it crashed back-relay in ITS#8428, does that mean everything which creates a slap_callback should set

Re: LMDB: mdb_cursor_put() and the rest of page in case MDB_RESERVE

2016-03-31 Thread Hallvard Breien Furuseth
On 30. mars 2016 19:25, Леонид Юрьев wrote: 2016-03-30 19:46 GMT+03:00 Howard Chu : Леонид Юрьев wrote: Why mdb_cursor_put() doesn't copy the rest of page in case MDB_RESERVE? For MDB_RESERVE the caller is going to supply their own data later, so nothing on the page is valid

Re: Slapd startup behavior when unable to bind to an interface

2016-01-11 Thread Hallvard Breien Furuseth
On 10. jan. 2016 00:48, Quanah Gibson-Mount wrote: Currently, slapd will start up even if it can't bind to an interface, if more than one potential interface is given where the bind is successful. (...) This is fairly trivial to reproduce. As a non-privileged user, simply do: -h "ldap://

Re: mm_last_pg+1 != filesize

2016-01-05 Thread Hallvard Breien Furuseth
I forgot this thread... On 22/12/15 16:39, Howard Chu wrote: Not malicious. Omit some commits like slapadd -q does, and it can happen naturally. Maybe slapadd -q can do just that with large IDLs. The change in question was for msync, which is only used for MDB_WRITEMAP. True, I missed

Re: mm_last_pg+1 != filesize

2015-12-21 Thread Hallvard Breien Furuseth
On 22/12/15 08:23, Howard Chu wrote: Hallvard Breien Furuseth wrote: Do env_sync() and commit() survive the test program from ITS#7886? It creates a datafile which ends before mm_last_pg+1. That would probably crash on Windows, I'll check it later. Annoying that Windows doesn't just flush

mm_last_pg+1 != filesize (was: openldap.git branch mdb.master updated. 209b56fead1afe8273db6c714c0a74a9c09b9cf6)

2015-12-21 Thread Hallvard Breien Furuseth
On 21/12/15 03:39, openldap-commit2de...@openldap.org wrote: commit 209b56fead1afe8273db6c714c0a74a9c09b9cf6 Author: Howard Chu Date: Mon Dec 21 02:36:20 2015 + ITS#8324 fix for WRITEMAP We called FlushViewOfFile with (map,mapsize) which worked fine

Big leaf nodes in LMDBv2

2015-09-24 Thread Hallvard Breien Furuseth
LMDBv1 supports 1-node leaf pages so it can split a 2-node leaf to make room for a 3rd node. (ITS#8237). LMDBv2 could extend this, to use fewer pages: Allow oversized leaf nodes. mdb_cursor_put() won't create ovpages directly, but call page_split() which instead of splitting may kick an

Re: lmdb: is read operation permitted inside the write transaction?

2015-05-18 Thread Hallvard Breien Furuseth
On 12/05/15 19:36, Howard Chu wrote: Dmytro Milinevskyy wrote: Basically I need to have an opportunity to call mdb_get while in write transaction. The sequence is smth like: - txn = mdb_txn_begin(flags=0) for i in 0..x: - v = mdb_get(txn, i) - mdb_put(txn, x+i, v) - mdb_put(txn,

www.openldap.org not mobile-friendly

2015-04-14 Thread Hallvard Breien Furuseth
It will drop in Google's ratings next week because it fails the https://www.google.com/webmasters/tools/mobile-friendly/ tests. Maybe someone can fix the web pages up a bit? I'm too busy now. -- Hallvard

Re: slaptest-ldif -l ldif tool or slapadd option

2015-03-25 Thread Hallvard Breien Furuseth
On 25. mars 2015 00:56, Emmanuel Lécharny wrote: Le 24/03/15 23:56, Howard Chu a écrit : Hallvard Breien Furuseth wrote: I'd like a slap tool which verifies an LDIF before I try to ldapadd/slapadd it. slapadd -u -o value-check=yes is fairly close. What does it fail to catch? I can think

Re: Fwd: LMDB and text encoding

2015-02-02 Thread Hallvard Breien Furuseth
On 02. feb. 2015 14:24, Howard Chu wrote: Hallvard Breien Furuseth wrote: I suggest we wait to deal with DB names until we also have a way to deal with filenames. And this time test that it works is practice:-) Hopefully users and programmers will only need one method of handling non-ASCII

Re: Fwd: LMDB and text encoding

2015-02-02 Thread Hallvard Breien Furuseth
On 02. feb. 2015 16:03, Timur Kristóf wrote: A path is always a Unicode string, while a DB name can be an arbitrary binary blob. So I don't think that we can treat them the same way. Not the point. A program which uses LDMB can choose to treat its own DB names in its own LMDB environments as

Re: Fwd: LMDB and text encoding

2015-02-02 Thread Hallvard Breien Furuseth
On 02. feb. 2015 16:25, Timur Kristóf wrote: Okay. What do you suggest? I suggest we wait to deal with DB names until we also have a way to deal with filenames. And this time test that it works is practice. And then I also suggest to try to make this mess simple to deal with for programmers

Re: LMDB and text encoding

2015-02-01 Thread Hallvard Breien Furuseth
On 02/02/15 00:40, Howard Chu wrote: It looks OK to me. No one raises any concerns I'll commit it in a few hours. Some sudden last thoughts: mdb_dump.c also has a check (memchr(key.mv_data, '\0', key.mv_size) to exclude non-databases, which is no longer valid. Database names with \0 in them

Re: LMDB and text encoding

2015-02-01 Thread Hallvard Breien Furuseth
On 02/02/15 02:00, Hallvard Breien Furuseth wrote: Come to think of it, I have no idea if the dump format supports DB names with \0 in them. ...and there will now be database names which cannot be spelled on the command line, like for mdb_stat/mdb_dump -s subdb. I don't think that was quite

Re: Question regarding MDB_NOLOCK

2015-01-30 Thread Hallvard Breien Furuseth
On 30/01/15 22:57, David Barbour wrote: On Mon, Dec 1, 2014 at 6:55 AM, Hallvard Breien Furuseth h.b.furus...@usit.uio.no mailto:h.b.furus...@usit.uio.no wrote: (...) Last snapshot is never overwritten. So readers which did begin/renew after latest commit(write txn) are safe from

Re: syncrepl consumer is slow

2015-01-30 Thread Hallvard Breien Furuseth
On 29. jan. 2015 04:12, Howard Chu wrote: I'm considering adding an option to the consumer to write its entries with dbnosync during the refresh phase. The rationale being, there's nothing to lose anyway if the refresh is interrupted. I.e., the consumer can't update its contextCSN until the very

Re: LMDB and text encoding

2015-01-29 Thread Hallvard Breien Furuseth
My take: On 27. jan. 2015 22:39, Timur Kristóf wrote: 1. Database names MDB_val here sounds nice... 2. Path names Functions like mdb_env_open, mdb_env_get_path, mdb_env_copy and the likes accept a char* for path names. This is fine on most unixes where char* is an UTF-8 string, but

Re: LMDB and text encoding

2015-01-29 Thread Hallvard Breien Furuseth
I wrote: Though I notice Windows #defines CreateFile() co as CreateFileA or CreateFileW depending on whether or not UNICODE is #defined (and some other macros), without even mentioning this in the CreateFile() doc. I suppose ldmb.h could do something similar - but with doc:-) Whoops, is does

Re: ext3/ext4 fsync hack

2015-01-06 Thread Hallvard Breien Furuseth
On 01/06/2015 03:18 PM, Howard Chu wrote: Hallvard Breien Furuseth wrote: () If the new pages below mt_next_pgno were freed instead of written, me_size becomes too big. Huh? mt_next_pgno definitively tells how many pages have ever been used in the DB file. No, see ITS#7886: Allocate

Re: ext3/ext4 fsync hack

2015-01-06 Thread Hallvard Breien Furuseth
Sorry, forgot this one. On 01/06/2015 03:18 PM, Howard Chu wrote: Hallvard Breien Furuseth wrote: Another issue: After an MDB_NOSYNC commit, mdb_env_sync() only fdatasync()s. It does not know when the file grew. I suppose we can change the FORCE flag to also cause fsync() to be used

Re: LMDB crash consistency, again

2015-01-05 Thread Hallvard Breien Furuseth
Another sync issue: mdb_env_sync() syncs the wrong way (msync vs fdatasync) if it runs in a process with a different MDB_WRITEMAP setting than one which committed with MDB_NOSYNC or MDB_NOMETASYNC. I.e. this statement in lmdb.h is too weak: * Processes with and without MDB_WRITEMAP on the

Re: LMDB crash consistency, again

2015-01-05 Thread Hallvard Breien Furuseth
On 01/05/2015 11:25 PM, Hallvard Breien Furuseth wrote: I think I added it after an IRC chat. But it should either say that it can break ACID, or env_sync() called explicitly should sync more aggressively - at least if the MDB_env did not commit all transactions since last known sync. Never

ext3/ext4 fsync hack (was: openldap.git branch mdb.master updated. 0018eeb2c3b2239c30def9d47c9d194a4ebf35fe)

2014-12-24 Thread Hallvard Breien Furuseth
On 18/12/14 05:40, openldap-commit2de...@openldap.org wrote: commit 0018eeb2c3b2239c30def9d47c9d194a4ebf35fe Author: Howard Chu h...@openldap.org Date: Thu Dec 18 04:38:53 2014 + Hack for potential ext3/ext4 corruption issue Use regular fsync() if we think this commit grew the

Re: Freeing unused pages between two revisions...

2014-12-16 Thread Hallvard Breien Furuseth
On 15/12/14 22:26, Howard Chu wrote: Yes, finding that gap is the tricky part now. I'm guessing we'll need a freeDB format change to have enough info to do it, though maybe not. Currently we record that in txn X, some list of pageIDs was freed. In LMDB 1.0 we will also have per-page txnID

Re: Freeing unused pages between two revisions...

2014-12-16 Thread Hallvard Breien Furuseth
I wrote: To F, oldest snapshot S which can see the page = max(W, oldest current snapshot). Sorry, that's the oldest easily computable age. There may not be a current snaphot with that age, which is OK.

Re: [LMDB] inconsistent mutex behavior and problems for M:N threading

2014-12-11 Thread Hallvard Breien Furuseth
On 12/11/2014 02:21 AM, Howard Chu wrote: David Barbour wrote: When two writers operate in the same thread using M:N threading, a bug will occur in Windows whenever two different lightweight 'writer' threads grab the same mutex in the same OS thread. Because both writers will succeed. Yes -

Re: LMDB crash consistency, again

2014-11-15 Thread Hallvard Breien Furuseth
On 15/11/14 02:57, Howard Chu wrote: Ah good point. If you check out their slides, #103 of 106 asks the question; the only failure they found in LMDB occurred on ext3 (and not on XFS) so we may just chalk this up to a flaw in ext3 instead. Given that ext3 has already been superseded by ext4,

Re: LMDB crash consistency, again

2014-11-14 Thread Hallvard Breien Furuseth
Catching up with old mail... On 20/10/14 12:44, Howard Chu wrote: This paper https://www.usenix.org/conference/osdi14/technical-sessions/presentation/zheng_mai describes a potential crash vulnerability in LMDB due to its use of fdatasync instead of fsync when syncing writes to the data file.

MDB_TOOL mode vs. large mdb_env_set_maxreaders()?

2014-08-28 Thread Hallvard Breien Furuseth
If an lmdb application configures a non-default maxreaders, this setting is ignored if a tool like mdb_copy is already running (maybe from crontab). I suggest mdb_copy co should use an MDB_TOOL env flag which requires the lockfile (if any) to be valid, and only partly reinitializes it. The

MIPS cachesync (was: openldap.git branch mdb.master updated. 3630066843b7ca6b2cd12911d3e2fe3314cd4549)

2014-08-03 Thread Hallvard Breien Furuseth
Should have changed the Subject: before. Anyway: On 02/08/14 23:28, Howard Chu wrote: Hallvard Breien Furuseth wrote: On 02/08/14 22:31, Howard Chu wrote: Note that this is invalidating an on-chip data cache which is typically only 32KB or so. It has nothing to do with flushing the buffer

Re: openldap.git branch mdb.master updated. 3630066843b7ca6b2cd12911d3e2fe3314cd4549

2014-08-02 Thread Hallvard Breien Furuseth
On 27/07/14 03:27, openldap-commit2de...@openldap.org wrote: commit 3630066843b7ca6b2cd12911d3e2fe3314cd4549 Author: Howard Chu h...@symas.com Date: Sat Jul 26 18:16:02 2014 -0700 Fix MIPS cache coherency on Linux MIPS chips require manual control of on-chip caches. The cacheflush

Re: openldap.git branch mdb.master updated. 3630066843b7ca6b2cd12911d3e2fe3314cd4549

2014-08-02 Thread Hallvard Breien Furuseth
On 02/08/14 18:57, Howard Chu wrote: Hallvard Breien Furuseth wrote: (...) If so, MDB_NOLOCK may be in trouble since it uses pick_meta() instead of mti_txnid. Should there be a separate CACHEFLUSH after writing the datapages if MDB_NOLOCK, and the current CACHEFLUSH should just flush

Re: openldap.git branch mdb.master updated. 3630066843b7ca6b2cd12911d3e2fe3314cd4549

2014-08-02 Thread Hallvard Breien Furuseth
On 02/08/14 22:31, Howard Chu wrote: Hallvard Breien Furuseth wrote: What matters is that nothing *sees* the metapage before its data pages, nor sees the mti_txnid change before the metapage. I thought that's what cache coherency and memory ordering was about. No, memory ordering is about

Compact log format for faster searching?

2014-03-13 Thread Hallvard Breien Furuseth
Have anyone come up with a searchable, compact slapd-log format, so the logs can be searched quicker than the current verbose format? That is, the search tool would understand the compact format without expanding it (fully) first, and only expand the final output. Our syslog from loglevel stats

Re: LMDB stuff

2014-02-03 Thread Hallvard Breien Furuseth
On 2014-02-03 22:14, Howard Chu wrote: Was chatting with Emmanuel Lecharny (who is currently working on Mavibot for ApacheDS, an MVCC backend similar to LMDB) and had an interesting realization: we can avoid the current issue of long-lived reader txns preventing page reclamation. (...) since

Fwd: Re: (ITS#7713) Segmentation fault if the pagesize of the Operating system is not equal to 4096.

2013-10-01 Thread Hallvard Breien Furuseth
Howard Chu writes: Fixing this will either require adding a bunch of ugly code, or changing the on-disk format again. Opinions? (...) Adjusting this would require only a few minor tweaks to the code, but would require a full dump/reload of existing databases. Then I hope for the format

EC_KEY undefined in master (was: openldap.git branch master updated)

2013-09-08 Thread Hallvard Breien Furuseth
openldap-commit2de...@openldap.org writes: commit e631ce808ed56119e61321463d06db7999ba5a08 Author: Howard Chu h...@openldap.org Date: Sat Sep 7 09:47:19 2013 -0700 ITS#7595 Add Elliptic Curve support for OpenSSL tls_o.c:326: error: 'EC_KEY' undeclared. /usr/include/openssl/ssl.h has

Re: LMDB dead process detection

2013-07-18 Thread Hallvard Breien Furuseth
Howard Chu writes: There's been a long-running discussion about the need to have APIs in liblmdb for displaying the reader table and clearing out stale slots. Quite a few open questions on the topic: (...) 3) What approach should be used for automatic detection of stale slots? Currently

Re: NFS-shared LMDB?

2013-07-10 Thread Hallvard Breien Furuseth
Howard Chu writes: It occurs to me that there is the potential to support an interesting use case with LMDB when the database resides on remote shared storage. In the context of slapd, you could run multiple read-only slapds concurrent with a single read-write slapd on a single database.

Re: openldap.git branch mdb.master updated. 21da623bf40dc21f89c3172c523d094075e0824b

2013-05-02 Thread Hallvard Breien Furuseth
openldap-commit2de...@openldap.org writes: Allow reading freelist while working on it The circular dependency issues appear to have been resolved. Still, need to watch closely, maybe revert this change if problems arise. IIRC this could get page-hungry when I tried that

lmdb.h defining mode_t

2013-03-31 Thread Hallvard Breien Furuseth
openldap-commit2de...@openldap.org writes: commit 0c568117cea6764c3f12a0d69ddf54ee467d2dec Author: Howard Chu h...@symas.com Date: Sun Mar 31 08:10:43 2013 -0800 MSVC doesn't define mode_t Nor should ldmb.h. That can break if it meets a header from another package doing almost the

mdb.master commit ITS#7515 fix mdb_dbi_open/close

2013-02-02 Thread Hallvard Breien Furuseth
openldap-commit2de...@openldap.org writes: commit 48dc782ea612f85e8356a50bfbafe22e5be121cf Author: Howard Chu h...@symas.com Date: Thu Jan 31 20:05:48 2013 -0800 ITS#7515 fix mdb_dbi_open/close If a DBI handle is opened by a txn that aborts, the DBI handle should no

MDB mapsize changes

2013-01-17 Thread Hallvard Breien Furuseth
Current mdb.master can change MDB_meta.mm_mapsize when another process has the database open. That can break when one MDB_env grows the DB file beyond another MDB_env's me_mapsize. Coexisting MDB_env's must use the same mapsize. Any mm_mapsize change must be written while mdb_env_open() has the

Avoiding read() of meta pages (was: Fwd: adding support to mdb)

2013-01-15 Thread Hallvard Breien Furuseth
In openldap-technical, Quanah Gibson-Mount wrote: As per Howard Chu (Author of MDB, Primary OpenLDAP Developer): - Full details are in the paper. http://www.openldap.org/pub/hyc/mdm-paper.pdf MDB assumes a unified buffer cache.

Re: Avoiding read() of meta pages

2013-01-15 Thread Hallvard Breien Furuseth
Howard Chu writes: From: Howard Chu h...@symas.com To: Hallvard Breien Furuseth h.b.furus...@usit.uio.no, openldap-devel@openldap.org Subject: Re: Avoiding read() of meta pages Date: Tue, 15 Jan 2013 20:56:40 + Hallvard Breien Furuseth wrote: I presume this can only be a problem

Re: send_ldap_ber mutex overhead

2013-01-07 Thread Hallvard Breien Furuseth
Howard Chu writes: The attached patch avoids grabbing conn-c_mutex unnecessarily. Testing on my laptop with test008 and SLAPD_DEBUG=0 shows a 33% speedup. Before moving ahead with this patch, I'd like to verify that ITS#5835 doesn't reappear. Anyone able to test and report back? That's

Re: connection management changes

2013-01-07 Thread Hallvard Breien Furuseth
Howard Chu writes: We listen for writable sockets if a write attempt returns incomplete. There's a pair of mutexes and condition variables used to synch up here between the writing threads and the listener thread. It's quite a lot of lock overhead. As far as I can tell the main reason we

Re: openldap.git branch mdb.master updated. 0ce6bb4be0034120c850917bc4f59b4d4efc1432

2012-12-14 Thread Hallvard Breien Furuseth
openldap-commit2de...@openldap.org writes: commit aff2693fc0721df4ccb6ceb357f80501c413ed38 Author: Howard Chu h...@symas.com Date: Mon Dec 10 12:16:50 2012 -0800 ITS#7455 simplify Don't try to reclaim overflow pages while operating on the freelist (for now). The circular

Re: openldap.git branch mdb.master updated. 0ce6bb4be0034120c850917bc4f59b4d4efc1432

2012-12-14 Thread Hallvard Breien Furuseth
I wrote: A simple variant of page ranges, to save space and simplify range handling: /* Page range: (pagecount MDB_PGNO_BITS) | (pageno + pagecount) */ typedef pgno_t mdb_pages_t; Lone pages get pagecount=1. With MDB_PGCOUNT_BITS = (64bit 4 ? 19 : 12) MDB_PGCOUNT_BITS =

mdb_dcmp() = EINVAL?

2012-12-03 Thread Hallvard Breien Furuseth
mdb_dcmp() returns EINVAL if no dupsort function is set. I'm not sure it does the caller any favors in doing so: It claims (AB BA) if it gets called with both (A,B) and (B,A). That can confuse the caller, e.g. a sort function. I expect the sensible results without a dcmp function are 0 or a

Re: MDB library naming conflict

2012-12-02 Thread Hallvard Breien Furuseth
Quanah Gibson-Mount writes: [About name clashes with MDB Tools] The only place I can think this could cause an issue would be Debian (and then Ubuntu). They load all library symbols into a shared address space used by every user, including root. This has caused me endless nightmare in the

Re: MDB v2: Replace meta pages with meta position word

2012-11-16 Thread Hallvard Breien Furuseth
Howard Chu writes: Hallvard Breien Furuseth wrote: I wrote: One thing about ACID - IIRC there are some potential data races which my single-word meta info in the header will fix. E.g. if the user does ^Z at an unfortunate time, and with WRITEMAP updating the meta page while a txn is reading

Re: MDB v2: Replace meta pages with meta position word

2012-11-14 Thread Hallvard Breien Furuseth
I wrote: One thing about ACID - IIRC there are some potential data races which my single-word meta info in the header will fix. E.g. if the user does ^Z at an unfortunate time, and with WRITEMAP updating the meta page while a txn is reading it. Reproducible by sending frequent SIGSTOPs and

MDB v2: Replace meta pages with meta position word

2012-11-11 Thread Hallvard Breien Furuseth
I think MDB v2 should move the variable parts of MDB_meta into the data pages. The datafile header would retain a word with the position of the last *synced* MDB_meta, or of the last meta when MDB_NOSYNC. The lockfile header would hold the position of the *last* MDB_meta. All transactions start

Re: MDB v2: Replace meta pages with meta position word

2012-11-11 Thread Hallvard Breien Furuseth
Howard Chu writes: I don't see this approach reducing seek overhead. It may be able to reduce sync overhead, but only if you accept the possibility of delayed syncs failing. Overall I don't see it as any improvement for ACID compliance. What is the real benefit? I thought I said under

Re: MDB v2: Replace meta pages with meta position word

2012-11-11 Thread Hallvard Breien Furuseth
Howard Chu writes: Setting the latest commit becomes atomic: Just change metapos. (Field MDB_txninfo.mti_txnid goes away.) The latest commit is already atomic. mti_txnid is updated atomically in the current code. No sync issues with copying 'MDB_db's from the meta, since the meta will

Re: MDB v2: Replace meta pages with meta position word

2012-11-11 Thread Hallvard Breien Furuseth
Howard Chu writes: Hallvard Breien Furuseth wrote: Then I'm not sure what's the point of so many speedup options like it has now. MDB_NOSYNC is perfectly safe on some filesystems like ZFS that guarantee write order. Nice. Should tweak the doc a bit, then. Do you mean safe without

Re: MDB v2: Replace meta pages with meta position word

2012-11-11 Thread Hallvard Breien Furuseth
Howard Chu writes: Using variably positioned meta pages seems like something we would try to cut down on seek overhead, but looking closer it doesn't appear that it can do that. Could some commits choose where the next commit's MDB_meta should go? Maybe peel a page off the freelist on behalf

Support readonly MDB slaptools on readonly filesystems

2012-10-27 Thread Hallvard Breien Furuseth
Back-mdb needs this test, copied from libmdb, to make use of the libmdb support for read-only filesystems. I don't know if Windows sets errno=EROFS for fopen(fname, w) when fname is on a read-only FS, if so the #ifndef _WIN32 is unnecessary. --- servers/slapd/back-mdb/config.c +++

Re: openldap.git branch mdb.master updated. ef2092eac9f35b77eae250801c96dfa515c6d4b2

2012-10-25 Thread Hallvard Breien Furuseth
commit ef2092eac9f35b77eae250801c96dfa515c6d4b2 Author: Howard Chu h...@symas.com Date: Thu Oct 25 05:28:55 2012 -0700 MDB_DEBUG tweak for old compilers This changes which page DPRINTF says got popped, since mc_top is now read after the mc-mc_top--; statement. I think it should have

Re: openldap.git branch mdb.master updated. 14fb1f59c7e119a100952890f947377d7e95f135

2012-07-25 Thread Hallvard Breien Furuseth
Howard Chu writes: Append mode now does no key comparisons, input must be in sorted order. page_split was not updating cursor parents correctly. MDB_APPEND mode is meant to be used for bulk loading of pre-sorted keys. (e.g., if someone wanted to write an mdb_load utility and the

Re: Added option (-K) on client tools to redirect errors on stdout instead of stderr.

2012-04-17 Thread Hallvard Breien Furuseth
Hans Deragon wrote: I implemented a simple common option (-K) for the client tools that if enabled, sets the file descriptor used to output errors to stdout instead of the default (and currently used) stderr. This is done because when running a batch of commands with the output sent to a

mdb meta pages

2012-04-12 Thread Hallvard Breien Furuseth
Could writing a word/byte to the current meta page break someting? While I'm asking, why are metas separate pages, instead of simply a fixed 256 or so bytes apart to keep them in separate cachelines? The only reason I can think of is if a write gets garbled, the other meta page is safe - but mdb

slapcat of selected attributes

2012-03-14 Thread Hallvard Breien Furuseth
I'd like slapcat to accept an attribute list to output: slapcat -H ldap:///dc=uio,dc=no?telephoneNumber,cn; slapcat URLs won't mean the same as in the LDAP spec, though: An empty attribute list = all attrs, both user user and operational ones: *,+. Elsewhere in LDAP it means just user

Re: CVS branching - Git branching model

2012-01-31 Thread Hallvard Breien Furuseth
Quanah Gibson-Mount wrote: --On Tuesday, January 31, 2012 12:11 PM +0100 Hallvard Breien Furuseth h.b.furus...@usit.uio.no wrote: I think we should leave the current CVS-style branching model and make better use of Git. 'master' is currently two things: The development branch