Trevor Talbot wrote:
Michael is referring to a direct map from disk pages to memory pages
(OS notion of pages, not sqlite's), using something like mmap() on
unix or MapViewOfFile() on Windows. This way memory is directly
backed by the file it refers to, instead of copying the data to
entirely new pages (possibly backed by swap). It removes one level of
(redundant) cache.
The complications tend to come in when considering I/O control and
locking. OS pages don't necessarily map to sqlite pages, so there can
be some "odd" boundaries there. This would be most noticable when
trying to flush data to disk. (The typical mmap abstraction requires
you force dirty OS pages to disk. Interactions between file maps
(often copy-on-write) and underlying OS caches can be weird.).
You're also bounded by VM space when trying to map large files. Most
mapping abstractions use "windows" intended to map several sequential
OS pages, and since sqlite randomly accesses pages, it would probably
be too much overhead when trying to handle files larger than the VM
space you're willing to sacrifice to the map.
In the general case I don't see it paying off, but in some specific
cases it could be a win. I'd be interested to see experiments.
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
A well reasoned explanation. We use mmap'ing with great success, but
not for very large files or databases, for the reasons described above.
You definitely get a performance lift with mmap'd I/O largely because a
level of buffer shadowing is removed.
On a B-Tree index access I measured a speed improvement of about 40% by
changing from reads and local cacheing to mmap'd access.
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
