Dominik Taborsky wrote:
Hello,
I've tried stress-testing LMDB in our use case and I've discovered something I
consider a bug.
In our use case we need to keep inserting to the DB until it's full, flush it
and repeat. We've designed the insert function to fail if there are not enough
headroom pages for safe delete / DB drop. During the tests the DB works
correctly for some amount of repeats, but then it suddenly refuses to insert
anything. It seems LMDB doesn't clear or reuse all of its pages. I would like
to get anyone's opinion on this.
I have attached the source file containing the test. The DB size is set to
20MB, the inserted values have by default 25kB. In the first run the DB accepts
365 inserts, which decreases over time and stabilizes at 280. After 18 repeats
the DB does not accept any inserts at all:
ok 1 - pass #1 fillup run (365 inserts)
ok 2 - pass #2 fillup run (300 inserts)
ok 3 - pass #3 fillup run (286 inserts)
ok 4 - pass #4 fillup run (283 inserts)
ok 5 - pass #5 fillup run (281 inserts)
ok 6 - pass #6 fillup run (280 inserts)
ok 7 - pass #7 fillup run (280 inserts)
ok 8 - pass #8 fillup run (280 inserts)
ok 9 - pass #9 fillup run (280 inserts)
ok 10 - pass #10 fillup run (280 inserts)
ok 11 - pass #11 fillup run (280 inserts)
ok 12 - pass #12 fillup run (280 inserts)
ok 13 - pass #13 fillup run (280 inserts)
ok 14 - pass #14 fillup run (280 inserts)
ok 15 - pass #15 fillup run (280 inserts)
ok 16 - pass #16 fillup run (280 inserts)
ok 17 - pass #17 fillup run (280 inserts)
ok 18 - pass #18 fillup run (108 inserts)
not ok 19 - pass #19 fillup run (0 inserts)
not ok 20 - pass #20 fillup run (0 inserts)
The test is run 4 times with different approaches that we thought could have
had some impact. These approaches combine opening the DB per insert or per
single fillup, and reopening the DB just for the DB drop. In this case these
options have no effect and the results are the same, but originally a unit test
that this originates from had some differences so I kept them in.
Note that I also tried calling mdb_drop with "1" as the last parameter, but it
had no effect on the result.
I am linking against the latest LMDB from sources.
I also tried adding a constant number of pages to the reserve before the insert
function fails. This only changes the amount of inserted items but doesn't
change the functionality.
If you compile the program you can supply it with an argument to change the
default size of the item value size. Lower values break the DB sooner (20kB
requires only 12 repeated fillups and drops).
If anyone has any idea what might be the issue, please let me know. I know
deleting the DB from filesystem and creating a new one would work, but that's a
hack, not a fix.
Your calculation of max_pages vs used_pages is totally bogus. There's nothing
wrong with LMDB.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
