Hi Howard, Thank you very much for your patient answers, and my apologies for my late reply. The "reserved space" was indeed occupied by the FREE_DBI, thanks again for pointing this out for me. Since I will have hundreds of envs (of unpredictable size) in my application, I am a bit reluctant in allocating too much space for all of them, and that's why I was trying to find a way emit a warning when room is about to lack. In this regard I think I have read something very promising in the docs: the mt_next_pgno field of the txn. It looks perfect. Do you think it would be safe to expose it in a read-only way? In any case I will follow you advice and try to find a safe compromise for the map size.
With best regards, Bruno. ________________________________________ De : openldap-technical [[email protected]] de la part de Howard Chu [[email protected]] Envoyé : jeudi 7 janvier 2016 22:25 À : Bruno Freudensprung; [email protected] Objet : Re: LMBD questions (stat & copy) Bruno Freudensprung wrote: > > Hi, > > I am still experimenting on this area and noticed that I can actually put more > data in LMDB by commiting after each put (for the sake of the test) than by > doing one commit in the middle of the data set. > In the end, in terms of amount of data inserted into the database, the big > picture is: > > 1 commit at the end > 1 commit after each put > 1 commit in the middle of > the data set + 1 commit at the end > > And mdb_stat can't really help anticipating MDB_MAP_FULL (maybe that wasn't > its purpose). The docs tell you to configure a large mapsize and go about your business. Doing anything else is a waste of time. > Anyway, if anybody had a hint about the maximum "reserved pages" (for internal > purpose) that I could subtract from the map size, I would be very grateful to > hear about it. Your sample code gives useless results because you're only counting pages in the Main DB. As plainly stated in the LMDB documentation, there is also a Free DB which is used to track free pages, and it obviously consumes pages itself in order to store the free page info. > > With best regards, > > Bruno (lmdb enthousiast). > > ------------------------------------------------------------------------------ > *De :* openldap-technical [[email protected]] de la part > de Bruno Freudensprung [[email protected]] > *Envoyé :* jeudi 31 décembre 2015 10:02 > *À :* [email protected] > *Objet :* LMBD questions (stat & copy) > > Hi, > > I am trying to detect MDB_MAP_FULL based on mdb_stat(txn, dbi, &stat) results > and I am wondering if I am on the right track or doing things correctly. > I have a small and simple test program (http://pastebin.com/SPCYgWMC) > exhibiting the following behavior (consistent between Windows 7 64-bit and > Ubuntu 15.10 64-bit - latest version of lmdb cloned and recompiled on Linux): > > * Inserting 20990 key value pairs in a database is OK when using one > transaction. Just before commiting the write transaction, mdb_stat(txn, > dbi, &stat) shows that the total number of pages (253) is very close the > the number of pages of the map (256). Indeed, adding another entry leads > to MDB_MAP_FULL. To put it shortly, this scenario looks OK (I am assuming > there might be 3 "internal-purpose" pages not shown by mdb_stat) and can > be tested by setting the stat_test variable to 0 at line 20 of the test > program. > * Inserting same 20990 key value pairs in a database fails (MDB_MAP_FULL > encountered at 19357) when using two consecutive write transactions. Just > before MDB_MAP_FULL, mdb_stat(txn, dbi, &stat) shows that the total number > of pages (232) is "quite far" from the the number of pages of the map > (256), leading my MDB_MAP_FULL detection heuristic to fail. It means that > only 92% of the total amount of data can be sucessfully added to the > database. Other scenarii (with same key/value lengths though) stop at 85% > only. This can be tested by setting the stat_test variable to 1 (line 20). > > By "total number of pages" I mean ms_branch_pages + ms_leaf_pages + > ms_overflow_pages. > > Do you think I am on a wrong track? (maybe mdb_stat does not show 24 > "internal-purpose" pages in the second case? or I am forgetting something?). > > One last remark: it seems that mdb_env_copy(db, copy_dir) is possible during a > write transaction on Windows (like stated in the documentation), but not on > Linux (the test program seems to be stopped on a mutex at line 62). This can > be tested by changing the copy_test variable to 1 at line 19. Is it expected > given my program? > > Best regards, > > Bruno. > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
