Thanks, this is great.. I have now learned that MDB_NOLOCK (with correct locking) allows me to get read-write transactions to cross thread boundaries (since they don't use TLS). I didn't see anywhere in the docs that read-write transactions don't use TLS, so I appreciate your input.
For read-only transactions, MDB_NOTLS and thread-crossing has been clear in the docs for a long time. So that part was easier. >From what now I see, LMDB works great with M:N applications for both reading >**and writing**. (M:N is still out there.. Erlang, Go, etc.) Jun 8, 2023, 16:04 by h...@symas.com: > Sam Dave wrote: > >> Very nice, I missed MDB_NOLOCK completely. This might work for me. >> >> However, one thing got my curious about MDB_NOLOCK docs: "For proper >> operation [of MDB_NOLOCK] the caller must ... ensure that no readers are >> using old >> transactions while a writer is active." >> >> Is this also true without MDB_NOLOCK, i.e. does the built-in locking >> mechanism also "ensure that no readers are using old transactions while a >> writer is >> active."? (I guess not because it sounds like it would break "readers don't >> block writers.") >> > > In the built in mechanism, readers never block writers. When using your own > locking, readers must block writers otherwise you'll corrupt the views that > read txns are seeing. > > Needless to say, with NOLOCK you must know what you're doing. From the level > of this conversation so far, I wouldn't recommend you go this route. > > Your initial problem seems to be that you don't understand the actual > semantics of transactions and threads. LMDB expects them to be 1:1. That's it. > Anything else is a waste, just like M:N threading (multiple user-level > threads on top of a different number of kernel threads) is a waste and has > been abandoned by everyone. > >> >> I was hoping I could do my own concurrency management only for write >> transactions without worrying about read transaction - I thought that's what >> "multiversioning" was all about. >> >> >> >> Jun 7, 2023, 19:17 by h...@symas.com: >> >> Sam Dave wrote: >> >> Read-write txns don't use TLS. TLS is only used for read-only txns. >> >> >> Why then is it important that a read-write transaction stays on the same >> thread (including the final abort or commit)? (I just assumed it was because >> it >> needed >> access to TLS, looks like I was wrong) >> >> >> Write txns are serialized by a mutex. The mutex is taken when a write txn >> begins and is released when the txn ends. >> It is illegal for any thread other than the original locking thread to >> release the mutex. >> >> If you don't want these semantics, just use MDB_NOLOCK and do your own >> concurrency management. >> >> >> Jun 7, 2023, 17:23 by h...@symas.com: >> >> Sam Dave wrote: >> >> Alternatively, could a more stateful API be provided (as an alternative to >> API relying on TLS)? >> >> Roughly what I have in mind is, mdb_txn_create() returns a state pointer, >> which should be passed into the next mdb_put(), which returns a state >> pointer, >> which >> should be passed into the next mdb_put(), and so on, until the final state >> is passed into mdb_commit() or mdb_abort(). >> >> >> A txn pointer is a state pointer. >> >> >> Jun 7, 2023, 10:26 by samd...@tuta.io: >> >> Thanks, I was just hoping abort was a (currently undocumented) exception to >> this :) >> >> Would it be possible/feasible in the future to bring MDB_NOTLS support to >> read-write transactions? (or at least some partial NOTLS support for only >> e.g. >> abort) >> >> >> Read-write txns don't use TLS. TLS is only used for read-only txns. >> >> >> Motivation: Ability to abort "later" (e.g. at garbage collection in some >> languages).. as this "later" stuff does not necessarily happen on the >> original >> thread. >> >> - Samuel >> >> >> >> Jun 6, 2023, 18:44 by h...@symas.com: >> >> Sam Dave wrote: >> >> Hi, >> >> The documentation for mdb_txn_begin() says "A transaction and its cursors >> must only be used by a single thread, and a thread may only have a single >> transaction >> at a time.". >> >> Does this also hold for mdb_txn_abort() for read-write transactions? I.e. >> does the abort also have to happen on the same OS thread as all the >> previous things >> (begin, put, etc.)? >> >> >> Calling mdb_txn_abort() requires passing the txn as a parameter, which >> looks pretty suspiciously to me like "using" it. Is this really a serious >> question? >> > > > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ >
