Dear Dr. Hipp, I suggest that your conclusion about mmap support on OpenBSD 2014-02-10:
I suspect that adding msync() calls *would wipe out any speed advantage* > for using memory-mapped I/O. And since speed is the only advantage to > memory mapped I/O and because there are many disadvantages, I don't see a > use-case for trying to make mmap work on OpenBSD. ( http://sqlite.1065341.n5.nabble.com/SQLite-3-7-17-preview-2x-faster-td68021i40.html ) is not correct. For instance, the whole file can be mmap:ed, with benefit. The only relevant considerations I see here are: 1) If sqlite's database open code would use read() before switching over to mmapped IO there could be a serious problem there i.e. getting outdated data during initialization. If so indeed some engineering or some kind of trick would be needed to make the database open switch over to mmapped IO early enough. 2) If by some reason sqlite uses write() for all writes instead of direct memory access presently, it would be a minor tweak to make it write directly to memory instead. You who know SQLite all the way through, would either 1) or 2) or something else still make a problem on OpenBSD? If not, how do I use SQLite with all-file memory mapping on OpenBSD: I just * Somehow force SQLite to compile with mmapping support turned on (what's the best way to do this, ./configure --force-enable-mmap ?), * Open the SQLite file via the API the ordinary way, and then * As first query after opening make "PRAGMA mmap_size = 1099511627776;" (=1TB or any other size that's guaranteedly above the size that the DB file will ever have)? (I understand that having mmapping forced off by default is the best option on this OS, at least for 32bit architectures, however mmapping on OpenBSD is still a very valid usecase and as per above I guess the tweaks to make it go are very simple, so would very much appreciate your thoughts and guidance on how to make this work!) Thank you very much! Mikael 2015-05-22 14:18 GMT+05:30 Mikael <mikael.trash at gmail.com>: > What do you say? > > Also I'm curious about Dr. Hipp meant in 2013 when talking about all the > BSD:s and OpenBSD in particular at > http://sqlite.1065341.n5.nabble.com/SQLite-3-7-17-preview-2x-faster-td68021.html > by > > "We have found that it does not work on OpenBSD, for reasons we > have not yet been able to uncover; but as a precaution, memory mapped I/O > is > disabled by default on all of the *BSDs until we understand the problem. " > > > Also I wonder if mmapping was switched on on the other BSD:s eventually. > > So basically: What's the problem/What was the problem. > > Thanks! > > > 2015-05-22 13:47 GMT+05:30 Mikael <mikael.trash at gmail.com>: > >> >> >> 2015-05-22 13:28 GMT+05:30 Dan Kennedy <danielk1977 at gmail.com>: >> >>> On 05/22/2015 02:37 PM, Mikael wrote: >>> >>>> Hi, >>>> >>>> If mmap_size is big enough, read() will never be needed, so what about >>>> enabling mmap on platforms that "lack of a unified buffer cache"? >>>> >>>> I mean, on a 64bit platform, setting mmap_size to say 1TB should be >>>> fine - >>>> >>>> So what about enabling mmap_size on platforms that "lack of a unified >>>> buffer cache"? >>>> >>>> >>>> Also, if read() by some reason would be needed, there's msync() which is >>>> like a flush(), so here the "lack of a unified buffer cache" shouldn't >>>> be >>>> an issue anyhow? >>>> >>> >>> I think part of the concern is two processes accessing the same SQLite >>> database - one using mmap() and one using regular read() and write() calls. >>> >> >> >> You mean, the issue would be if two processes had the same SQLite >> database opened and their mmap_size setting would be different? >> >> So SQLite already does msync() so the mmap+read()/write() is not actually >> an issue within the scope of one OS process, that the mmap and readwrite >> would go "out of sync" so you'd get data corruption that way? >> >> >> Well if that's the only problem then I guess on OBSD mmap_size can safely >> be turned on, just with that as precaution to the users i.e. all different >> OS processes that open one and the same SQLite database *must* have the >> *exactly* same mmap_size setting, or your file will be fried!? >> >> Sounds good to me! In particular that you can just put the mmap_size to >> 1TB on all instances all over? >> >> (as at least on a 64bit machine you have 16.7 million 1TB address spaces >> available so it's not like a scarce resource) >> >> >>> Just out of interest, is it a specific platform that you're concerned >>> about? >>> >> >> OBSD just as specifically mentioned on https://www.sqlite.org/mmap.html ! >> >> >>> Dan. >>> >> >
