Re: [sqlite] Memory-mapped I/O [was: SQLite 3.7.17 preview - 2x faster?]
On Sat, Feb 8, 2014 at 7:26 AM, Richard Hipp wrote: > OpenBSD lacks a coherent filesystem cache. That is to say, changes to a > file made using write() are not necessarily reflected in mmap-ed memory > right away. And change to a mmap-ed segment are not necessarily reflected > in subsequent read() operations. Availability of unified caches are not entirely a function of the OS; the OS may support it but the filesystem might not. The right thing to do is to msync() with MS_INVALIDATE in this case. It should be a no-op when the OS+filesystem implement a unified cache. It will be expensive when they don't. I don't recall if there's a portable way to find out if the OS+filesystem provide a unified cache. Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Memory-mapped I/O [was: SQLite 3.7.17 preview - 2x faster?]
On Sun, Feb 9, 2014 at 5:03 PM, Richard Hipp wrote: > On Sun, Feb 9, 2014 at 5:49 PM, James K. Lowden > wrote: > I suspect that adding msync() calls would wipe out any speed advantage for > using memory-mapped I/O. And since speed is the only advantage to memory > mapped I/O and because there are many disadvantages, I don't see a use-case > for trying to make mmap work on OpenBSD. msync() with MS_SYNC will definitely destroy performance, so don't do that. msync() with MS_ASYNC and/or MS_INVALIDATE are generally no-ops when the OS and *filesystem* implement a shared cache. The extra cost here, then, is the cost of a system call (plus whatever work has to be done to determine that the call is a no-op). Nico -- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Memory-mapped I/O [was: SQLite 3.7.17 preview - 2x faster?]
On Sun, Feb 9, 2014 at 7:19 PM, Drake Wilson wrote: > Quoth "James K. Lowden" , on 2014-02-09 > 17:49:15 -0500: > > That's true, although it's not quite fair to say the filesystem cache > > isn't "coherent". It's just not the Linux implementation. > > FYI, the term "coherent" I interpret as being used in the specific > technical sense involved (data is reflected automatically between > mmap and read/write), by relation to "cache coherent". So in that > sense it is both correct and not a disparagement of the OpenBSD > behavior. > > Correct. I should have been clearer in my original statement. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Memory-mapped I/O [was: SQLite 3.7.17 preview - 2x faster?]
Quoth "James K. Lowden" , on 2014-02-09 17:49:15 -0500: > That's true, although it's not quite fair to say the filesystem cache > isn't "coherent". It's just not the Linux implementation. FYI, the term "coherent" I interpret as being used in the specific technical sense involved (data is reflected automatically between mmap and read/write), by relation to "cache coherent". So in that sense it is both correct and not a disparagement of the OpenBSD behavior. ---> Drake Wilson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Memory-mapped I/O [was: SQLite 3.7.17 preview - 2x faster?]
On Sun, Feb 9, 2014 at 5:49 PM, James K. Lowden wrote: > > I know portability is important to the SQLite project. Is this > particular issue a matter of manpower, know-how, or policy? > > The mmap functionality in SQLite is completely portable to OpenBSD now, since as currently implemented, requests to activate mmap are silently ignored in OpenBSD. So you always get the correct answer. I suspect that adding msync() calls would wipe out any speed advantage for using memory-mapped I/O. And since speed is the only advantage to memory mapped I/O and because there are many disadvantages, I don't see a use-case for trying to make mmap work on OpenBSD. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Memory-mapped I/O [was: SQLite 3.7.17 preview - 2x faster?]
On Sat, 8 Feb 2014 08:26:43 -0500 Richard Hipp wrote: > > > The memory-mapped I/O is only enabled for windows, linux, mac > > > OS-X, and solaris. We have found that it does not work on > > > OpenBSD, for reasons we have not yet been able to uncove; but as > > > a precaution, memory mapped I/O > > is > > > disabled by default on all of the *BSDs until we understand the > > > problem. > > > > Was the problem ever identified? > > > > OpenBSD lacks a coherent filesystem cache. That is to say, changes > to a file made using write() are not necessarily reflected in mmap-ed > memory right away. And change to a mmap-ed segment are not > necessarily reflected in subsequent read() operations. That's true, although it's not quite fair to say the filesystem cache isn't "coherent". It's just not the Linux implementation. It looks like a sprinkling of msync(2) would do the trick. My copy of the Linux Programming Interface (chapter 49) says, "However, a unified virtual memory system is not required by SUSv3 and is not employed on all UNIX implementations. On such systems, a call to msync() is required to make changes to the contents of a mapping visible to other processes that read() the file, and the MS_INVALIDATE flag is required to perform the converse action of making writes to the file by another process visible in the mapped region." That appears to be the situation you describe. It continues, "Multiprocess applications that employ both mmap() and I/O system calls to operate on the same file should be designed to make appropriate use of msync() if they are to be portable to systems that don?t have a unified virtual memory system." I know portability is important to the SQLite project. Is this particular issue a matter of manpower, know-how, or policy? --jkl ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Memory-mapped I/O [was: SQLite 3.7.17 preview - 2x faster?]
On Fri, Feb 7, 2014 at 1:01 PM, varro wrote: > Regarding the following old post: > > Richard Hipp wrote: > > By making use of memory-mapped I/O, the current trunk of SQLite (which > will > > eventually become version 3.7.17 after much more refinement and testing) > > can be as much as twice as fast, on some platforms and under some > > workloads. We would like to encourage people to try out the new code and > > report both success and failure. Snapshots of the amalgamation can be > > found at > > > >http://www.sqlite.org/draft/download.html > > > > Links to the relevant documentation can bee seen at > > > >http://www.sqlite.org/draft/releaselog/3_7_17.html > > > > The memory-mapped I/O is only enabled for windows, linux, mac OS-X, and > > solaris. We have found that it does not work on OpenBSD, for reasons we > > have not yet been able to uncove; but as a precaution, memory mapped I/O > is > > disabled by default on all of the *BSDs until we understand the problem. > > Was the problem ever identified? > OpenBSD lacks a coherent filesystem cache. That is to say, changes to a file made using write() are not necessarily reflected in mmap-ed memory right away. And change to a mmap-ed segment are not necessarily reflected in subsequent read() operations. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Memory-mapped I/O [was: SQLite 3.7.17 preview - 2x faster?]
Regarding the following old post: Richard Hipp wrote: > By making use of memory-mapped I/O, the current trunk of SQLite (which will > eventually become version 3.7.17 after much more refinement and testing) > can be as much as twice as fast, on some platforms and under some > workloads. We would like to encourage people to try out the new code and > report both success and failure. Snapshots of the amalgamation can be > found at > >http://www.sqlite.org/draft/download.html > > Links to the relevant documentation can bee seen at > >http://www.sqlite.org/draft/releaselog/3_7_17.html > > The memory-mapped I/O is only enabled for windows, linux, mac OS-X, and > solaris. We have found that it does not work on OpenBSD, for reasons we > have not yet been able to uncove; but as a precaution, memory mapped I/O is > disabled by default on all of the *BSDs until we understand the problem. Was the problem ever identified? The answer isn't really important to me, since the version I'm using (3.8.0.2) works perfectly fine for me under FreeBSD 9.1, but I'm curious about the resolution (if any). -- Will ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users