On 2015-11-10 14:10, Tinker wrote:
...
A "safe" approach to file access would be to read data using mmap()
but write data using fwrite() only. Mmap does have a read-only mode.
This does NOT work in OpenBSD currently though because of the absence
of unified caching.
The nice thing about reading files from memory using mmap instead of
using fread(), is that you offload Lots of work to the OS kernel.
Suddenly file reading is free of mallocs for instance.
And the program doesn't need internal file caching, so the extent of the
OS' disk caching is increased. And I guess maybe the OS disk cache can
prioritize better what to keep in RAM.
In a way, mmap() is a way to "zero-copy file access", which is just
awesome.
A database that uses this technique is LMDB (OpenLDAP's default DB
backend).
A key feature of LMDB is that it's only 9600 locs,
https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/mdb.c :
LMDB is interesting in how lowlevel it is, as it's written in C and the
"loaded" database entries are simply pointers into the mmap():ed space.
I saw some fantastic-looking benchmarks, I think at
http://symas.com/mdb/#bench , where LMDB goes light-speed where others
remain on the ground.
(LMDB has a limit in usability in that it never shrinks a DB file,
however, that is in no way because of its use of mmap() and could really
be overcome by working more on it.
Also, LMDB serializes its DB writes, which also is an architecture
decision specific to it and which has severe performance implications -
and that is unspecific to mmap() also, and could be overcome.)
