> Page faults aren’t necessarily due to swapping. If you read a memory-mapped > file, the first access to any page will fault to disk. Since SQLite supports > memory-mapping, I’d assume this can occur during a query — the column data > returned by sqlite may point into mmap’ed pages. (Is that correct?)
> In that situation, if a blob value is significantly larger than one page, the > difference between scanning everything up to byte offset x, vs. reading a few > bytes at the beginning and then jumping directly to x, can be significant. I don't think SQLITE will return a pointer directly into an mmap page offset, and it definitely can't in the case of a column spilled into an overflow page - the data will be discontiguous. Even something like the blob API's don't help, SQLITE overflow pages link to each other at the beginning of each page. So you can't e.g. load the 5th page without also loading page 1 through 4. Since SQLITE pages are generally smaller than disk pages, it means faulting in all the disk pages on the way. Anyway, by faulting I meant 'paging back out due to memory pressure'. Sorry, should have been more clear - knew that one was going to bite me as soon as I hit send... How do you take a thread offline here? - Deon -----Original Message----- From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Jens Alfke Sent: Saturday, March 25, 2017 10:35 AM To: SQLite mailing list <sqlite-users@mailinglists.sqlite.org> Subject: Re: [sqlite] Memoization in sqlite json1 functions > On Mar 24, 2017, at 4:48 PM, Deon Brewis <de...@outlook.com> wrote: > > Yeah ok, if you take I/O hits then things like memory pre-fetching makes zero > difference. We're more in the business of "You take a page fault" == "You buy > more memory". Different level of performance requirements. (And glad that > SQLITE works well for both of us). Page faults aren’t necessarily due to swapping. If you read a memory-mapped file, the first access to any page will fault to disk. Since SQLite supports memory-mapping, I’d assume this can occur during a query — the column data returned by sqlite may point into mmap’ed pages. (Is that correct?) In that situation, if a blob value is significantly larger than one page, the difference between scanning everything up to byte offset x, vs. reading a few bytes at the beginning and then jumping directly to x, can be significant. >> In Fleece I put a lot of effort into making the C++ API nice to use, so that >> I don’t have to have any other data structure. That's worked well so far. > > Strong typing? That only happens at higher levels in our stack. At the level I mostly work on, this is a document-oriented, schemaless data store. Up above there’s the option to use a data-modeling layer that binds document properties to platform object properties with strong typing (though we don’t support C++.) But even those bindings can be considerably more efficient than the typical JSON object tree — you’re not allocating string objects for dictionary keys, nor number objects for numeric/boolean values. It’s also generally faster to populate instance variables of objects (usually at fixed offsets) vs. adding key/value pairs to a hash table. [I think we’re getting off topic, but I’ll be happy to continue offline.] —Jens _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
