Thanks! This is very helpful. To avoid issues with overwriting the file, would it work if I write a command-line utility that gets exclusive access to the db with mdb_env_excl_lock and then overwrite/move/rename the main mdb file before releasing the lock?
Sela On Thu, Sep 15, 2022 at 10:00 AM Howard Chu <[email protected]> wrote: > [email protected] wrote: > > I am working on developing a new document-oriented (XML+JSON) database, > using LMDB as an engine, and I have two questions. > > > > 1. So far, it have been working really smoothly for me. But my one > customer so far for the DB is really concerned about running LMDB in a > virtual environment such as Docker, when performing reads and writes. > Especially when mounting volumes. Their concern is because of the following > caveat: > > "Do not use LMDB databases on remote filesystems, even between processes > on the same host. This breaks flock() on some OSes, possibly memory map > sync, and certainly sync between programs on different hosts" > > I think it shouldn't be an issue with Docker, but I want to be certain. > > That caveat applies to actual remote filesystems like NFS or SMB. > > If you're just using a local filesystem as persistent storage it should be > fine. But, LMDB records process IDs in its reader table, > and containers all number their processes starting from 1. So if you have > multiple containers accessing the same DB at once, you're > likely to get PID collisions and break reader consistency. > > > 2. We have one server that updates the database, and another server with > a read-only copy of the same database. Our plan was to simply copy the mdb > files from the update machine to the read only machine, but we noticed that > if we copy the file immediately after writing, the copy may end up being > corrupted. My solution was to suspend all writing and wait few minutes > before writing, to make sure everything back from memory, and I'm also > using the "sync" command (not sure if it does anything here). It seem to be > working, but I wonder if there is a more robust way of doing that? And > also, is it safe to overwrite to the read-only server while it performs > read transactions to the current file (or maybe rename it and copy to a new > file with the same name)? > > > That's what the mdb_copy command is for... > > And no, it is not safe to overwrite the file while any process is reading > it. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ >
