On Wed, Jun 4, 2014 at 10:22 AM, Howard Chu <[email protected]> wrote:
> Brian G. Merrell wrote:
>
>> Hi all,
>>
>> First, I'm having trouble finding resources to answer a question like this
>> myself, so please forgive me if I've missed something.
>>
>
> http://symas.com/mdb/doc/
Thanks. I did see and skim the API portion of the docs before asking, but
I was just having trouble knowing how the pieces fit together to solve a
problem.
>
>
> I'm considering using LMDB (versus LevelDB) for a project I'm working on
>> where
>> I'll be receiving a high volume (hundreds per second) of high priority
>> requests (over HTTP) and issuing multiple (<10) database queries per
>> request.
>>
>> I'll also have a separate process receiving updates for the data and
>> writing
>> to the database. This will happen often (several times a minute,
>> perhaps),
>> but the priority is much lower than the read requests.
>>
>> LMDB appealed to me because of the read performance and that I could have
>> one
>> processing reading data from LMDB and another process writing data
>> updates to
>> LMDB.
>>
>> For proof of concept, I hacked up the following (I'll use pseudocode
>> since I
>> used the Go bindings for my actual programs, and hopefully my question is
>> sufficiently abstract not to matter):
>>
>
> Process 1, the writer, simply writes a random integer (from 0 to 1000) to
>> a
>> defined set of keys:
>>
>> env = NewEnv()
>> env.Open("/tmp/foo", 0, 0664)
>> txn = env.BeginTxn(nil, 0)
>> dbi = txn.DBIOpen(nil, 0)
>> txn.Commit()
>>
>> txn = env.BeginTxn(nil, 0)
>> n_entries = 5
>> for i = 0; i < n_entries; i++ {
>> key = sprintf("Key-%d", i)
>> val = sprintf("Val-%d", rand.Int(1000))
>> txn.Put(dbi, key, val, 0)
>> }
>> txn.Commit()
>> env.DBIClose(dbi)
>> env.Close()
>>
>> Process 2, the reader, simply loops forever and does random access reads
>> on
>> the data from process 1 (I won't benefit from a cursor for my actual
>> problem),
>> and prints out that data occasionally:
>>
>> env = NewEnv()
>> env.Open("/tmp/foo", 0, 0664)
>> while {
>> txn = BeginTxn(nil, 0)
>> dbi = txn.DBIOpen(nil, 0)
>> txn.Commit()
>> for i = 0; i < n_entries; i++ {
>> key = sprintf("Key-%d", i)
>> val = txn.Get(dbi, key)
>> print("%s: %s", key, value)
>> }
>> env.DBIClose(dbi)
>> sleep(5)
>> }
>>
>> So my high level question is: What am I doing wrong? This seems to work
>> OK,
>> but a lot of it was guesswork, so I'm sure I'm doing some silly things.
>>
>
> Your reader process should be using read transactions.
OK, I interpret this as meaning that I need to pass the MDB_RDONLY flag to
mdb_txn_begin. Is that correct?
>
> For example, first I put the BeginTxn() and DBIOpen() calls in process 2
>> outside of the while loop, but when I did that, I never saw the updates
>> values
>> upon running process 1 simultaneously. In my real-world application, it
>> seems
>> like adding these calls to every request (to be sure the data being read
>> is
>> up-to-date) could be an unnecessary performance penalty.
>>
>
> In the actual LMDB API read transactions can be reused by their creating
> thread, so they are zero-cost after the first time. I don't know if any of
> the other language wrappers leverage this fact.
>
This helps a lot. I will investigate what the case is with gomdb.
>
> Opening a DBI only needs to be done once per process. Opening per
> transaction would be stupid, like reopening a file handle on every request.
>
>
I suspected so. The fact that mdb_dbi_open takes a transaction had me
confused a bit, because I thought I would need to pass in the new
transaction every time I got a transaction from mdb_txn_begin.
I've refactored the reader to look like this:
env = NewEnv()
env.Open("/tmp/foo", 0, 0664)
txn = BeginTxn(nil, mdb.RDONLY) // parent txn is the nil arg
dbi = txn.DBIOpen(nil, 0)
txn.Abort()
while {
txn = BeginTxn(nil, mdb.RDONLY) // parent txn is the nil arg
for i = 0; i < n_entries; i++ {
key = sprintf("Key-%d", i)
val = txn.Get(dbi, key)
print("%s: %s", key, value)
}
txn.Commit()
sleep(5)
}
env.DBIClose(dbi)
Now, I guess the big question that BeginTxn inside the loop is zero-cost.
Thanks for the tips so far Howard; it has been very helpful.
>
> I was suspect there are flags that I can/should be using, but I'm not
>> sure.
>>
>> Thanks for any input.
>>
>> Brian
>>
>
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
Brian
