Thanks Jens!
On Mon, 22 Apr 2019, Jens Alfke wrote:
But yeah, I agree with you that it seems odd to have a compiled-in
restriction on the maximum memory-map size.
I looked a bit into the history, and it appears the just-under-2GB limit
was specifically put there (by drh) so that the value would fit into a
signed 32-bit integer:
https://www.sqlite.org/src/info/460752b857532016
Maybe this was in order to fit into a ssize_t on 32-bit platforms? (Just
a guess as the size_t type is used for mmap.)
If that's the reason, would drh & the sqlite team consider changing the
default SQLITE_MAX_MMAP_SIZE definition from 0x7fff0000 to SSIZE_MAX
(defined in limits.h) ?
...
Somewhat of an aside:
Most current OSs have a universal buffer cache, wherein filesystem
caches and VM pages use the same RAM buffers. A page-fault and a file
read will incur similar amounts of work. The big benefit is that the
memory-mapped pages can be evicted from RAM when needed for other stuff,
whereas a malloc-ed page cache is considered dirty and has to be swapped
out before the RAM page can be reused.
Right, i think we're agreeing here. I just mean that if a db file is much
larger than the available ram, and all of it is used for a query, then
whether bytes come in via mmap+memcpy() or regular file read(), the kernel
will page-in missing pages into its page cache, and (at its discretion) it
will eventually have to evict those pages to make room for others. (I
didn't mean to make reference to disk swap or sqlite's user-space
pagecache.) The point is that the kernel page loading/evicting part is
the same, but the mmap also saves the overhead for the seek/read system
calls.
And i say this from my personal experience with sqlite -- when doing a
long running join on a db that does not fit into ram, with the timer on, i
observed that in the regular non-mmap mode, the system time for the query
would be about equal to the user time. But when i mmap'ed the whole file,
the system time for the query was < 1% of the user time, and the user time
was also less than it had been in non-mmap mode. I can only guess the
huge difference in the system time was mainly the kernel handling all the
seek/read system calls, vs just memcpy when the whole db file is mmap'ed.
The bottom line in any case is i saw a substantial speedup for long
queries when mmap'ing a large db, even when it did not fit into ram.
Thanks..!
Carl
_______________________________________________
sqlite-users mailing list
[email protected]
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
