Windl, Ulrich wrote: > Hi! > > If it's like that (I had asked a similar question some time ago), we would > need better MDB statistics. > For example some extent-based filesystem did show a histogram of free > extents, ordered by the power of two sizes. > However MDB is not extent-based (AFAIK), but block based. > Maybe in addition to "unused blocks" (olmMDBPagesMax - olmMDBPagesUsed) and > "free blocks" (olmMDBPagesFree) (I was assuming those are "100% free") there > should be a number "fragmented free space in blocks" (or so).
All of that information is currently available from the mdb_stat command. There is currently no library function to return that info, but one is being added in LMDB 1.0. > > Kind regards, > Ulrich Windl > >> -----Original Message----- >> From: hubou...@microsoft.com <hubou...@microsoft.com> >> Sent: Thursday, July 3, 2025 3:29 AM >> To: openldap-technical@openldap.org >> Subject: [EXT] LMDB and Disk fragmentation >> >> >> Hi everyone, >> >> We’ve been stress‑testing an LMDB database by continually inserting and >> deleting records ranging from 4 KB to 5 MB. To track utilization, we rely on >> two independent metrics: >> Application index – we track the sum of the payload bytes of keys currently >> present. >> mdb_stat output – pages in use /overlay etc.... >> The two figures agree closely. >> >> Despite this, we repeatedly hit MDB_MAP_FULL when both metrics say the >> map is only ≈ 50 % occupied (on both metrics). >> Our tentative conclusion is heavy page fragmentation: LMDB can’t find a >> sufficiently large contiguous run of free pages even though half of the map >> is >> nominally free. >> Questions >> - Is it normal to lose that much effective capacity with an insert/delete >> workload like ours? >> - How far can we realistically fill the map before hitting fragmentation >> limits? >> - Are there recommended ways to mitigate or “defragment”? >> >> Any advice would be greatly appreciated! >> >> Thanks! >> Hugues -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
