Emmanuel Lécharny wrote:
Le 30/10/15 08:37, Howard Chu a écrit :
Kiran Ayyagari wrote:
On Thu, Oct 29, 2015 at 6:13 PM, Emmanuel Lécharny <[email protected]
<mailto:[email protected]>> wrote:
Hi guys,
here are a bit of my last night's insomnia thoughts (with the
shower,
and the smallest room of my appartment, this is the only place
and time
I have for such thoughts atm ;-) :
- txn are now going to be explicit, ie you have to begin a txn and
commit it before and after any update, like in :
// Inject some data
recordManager.beginTransaction();
btree.insert( 1L, "1" );
btree.insert( 4L, "4" );
btree.insert( 2L, "2" );
btree.insert( 3L, "3" );
btree.insert( 5L, "5" );
recordManager.commit();
if a user forgets to call commit() and keeps inserting we will run
out of
memory at some point
so wondering what is the better way to prevent that from happening
without
maintaining a
log (a WAL or something similar)
In the original LMDB implementation we simply had a max number of
dirty pages in a txn. Once that was hit, further write requests simply
failed with MDB_TXN_FULL.
In current versions we still have the same max number of dirty pages
at once, but when the limit is hit we spill a portion of pages
immediately to disk so we can reuse their memory. As such, the current
version allows txns of unlimited size, but still with a fixed RAM
footprint.
This is ok if there is no relation between pages. The pb is that when
you use LMDB (or Mavibot) in a LDAP context, you will have to update
many pages in a single (application) transaction. Letting the underlying
DB to arbitrary flush pages based on a size/time limit simply break the
application transaction barriers.
No, it doesn't break transaction barriers. Pages written to the DB are
meaningless (and invisible to other DB users) until the final meta page update
at the end of the transaction.
We can think about another solution though : instead of having to
explicitely start a transaction, and commit it, we can imagine a backend
for which we pass the full list of operations to apply, up to the bakend
to play those operations and commit (or rollback) them all. This remind
me about 'continuation', where we computed the code to be executed,
before passing it to the part in charge of its execution (it was back in
1988, when I was programing in LISP, where code is data).
That doesn't change the problem at all. (In fact this is exactly how slapd
implements LDAP txns. But it all boils down to somebody explicitly starting a
txn, feeding in the sequence of ops, and explicitly committing at the end.)
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
