Hallvard wrote "Currently a moderate number of slots are cheap but a huge number gets expensive: 7-120 words per transaction, and every #mdb_dbi_open() does a linear search of the opened slots."
I haven't seen a performance hit with around 10000 named databases. By the way, I was hoping to only open those dbi's on demand rather than opening all at iniatialization. "With threads 1 and 2 coexisting? When thread 2 called mdb_dbi_open(), thread 1's prospect of using mdb_dbi_open() at all was lost." Yeah with both coexisting. Thats what I thought. @Klaus Yeah. I know there can be only one write transactions. I was talking about 1 write and 1 or more read transactions. It is not as if I am first looking to open dbi in the read transaction. It is because I can't guarantee whether another read transaction will start and will attempt to open the same named dbi when a write is in progress. "And first looking in a read transaction whether a database exists and then creating it in a second write transaction is definitely a bad and risky programming style, as it carries an assumption from one transaction to the next, which is typically not valid." That was not what I tried to do. "you still have the option to combine all your logical databases into a big single database" Its a workaround that I haven't thought about before. Hoping to avoid the extra complexity. Is there any prospect of implementing mdb_dbi_open or mdb_db_open_immediate to put the dbi into the shared environment without waiting for txn commit. I learned earlier from Howard Chu that it is not a wanted phenomenon in ACID. But just in case, because otherwise (without opening all the dbi's in initialization) in a multi-threaded environment, the possibility to open a dbi on demand ending in failure goes up. On Mon, May 22, 2017 at 2:01 PM, Klaus Malorny <[email protected]> wrote: > On 5/21/17 9:43 PM, Muhammed Muneer wrote: > >> Howard Chu wrote >> >> "Just follow the recommendation to open all handles at the beginning of >> the program." >> >> But what if I have lots of named databases like maybe 10000 or more. >> Wouldn't this be expensive. >> >> I am developing a MongoDB like database (similar in query and update >> syntax) around LMDB. >> The thing is I have some enhancements on my own like the ability to >> generate update queries >> from within an ongoing update. >> >> So in a multi threaded environment, if the name of a named dbi is >> generated from within a write >> transaction (thread1) and proceeds to mdb_dbi_open it only to find that >> another read transaction >> (thread 2) just opened the same named dbi after the write-txn of thread 1 >> started, the prospect of >> mdb_dbi_open the same named dbi for thread 1 is lost forever. >> >> > Please remember that you can have only one writing transaction at once. > And first looking in a read transaction whether a database exists and then > creating it in a second write transaction is definitely a bad and risky > programming style, as it carries an assumption from one transaction to the > next, which is typically not valid. > > I have no experience with a large number of databases, but if it is a > performance problem as Hallvard and the docs describe, then you still have > the option to combine all your logical databases into a big single > database. In this case you would maintain a database ID (e.g. four byte > integer) that is prepended to the user provided key for all get and put > operations. Only some care needs to be taken for range searches and cursor > operations, as you might get a key/value pair that belongs to another > logical database, but this is not a big deal. I use that approach for > composite search keys quite a lot. > > The association between database names and their IDs could be maintained > in a separate database. > > Regards, > > Klaus > >
