Hi Simon, On Wed, Apr 26, 2017 at 11:46 PM, Simon Slavin <slav...@bigfraud.org> wrote: > > On 26 Apr 2017, at 10:00pm, Kim Gräsman <kim.gras...@gmail.com> wrote: > >> Specifically, I wonder if 4MiB for the general-purpose heap is maybe >> entirely unreasonable? Is there a way to forecast how much memory will >> be necessary for transactions and query processing, or does that >> depend entirely on the workload? > > Depends on a number of factors, including the data present in the > database. In other words, you can forecast memory-usage based on the > data in your test database, but that does not allow you to predict > memory-use for your customer’s databases unless you know how many rows > they have in their tables, whether the chunkiness of their data has > changed since the last ANALYZE and such things. This isn’t the way > you’re meant to use SQL. You’re meant to issue your commands and have > the SQL engine worry about executing them.
https://www.sqlite.org/malloc.html leads me to believe otherwise. Especially the zero-malloc feature-set seems exactly aimed at controlling this. > So do you have good indexes ? Yes. And we force index use explicitly to make sure we get consistent results, instead of relying on the query planner. > With them, SQLite can just iterate through the entries in the table, as > indicated by index order. This dramatically reduces usage of the cache. Cache usage should not be a problem -- we can control that with the soft heap limit to have it auto-adjust. Compared to page cache, lookaside and scratch, general-purpose memory is not as well-described. I think that makes sense -- it's essentially "everything else" -- but I was expecting and hoping SQLite wouldn't indiscriminately use general-purpose memory for transaction/query processing. That said, it's hard to see through all the layers here. Some of this is described in: https://www.sqlite.org/malloc.html#_computing_and_controlling_parameters_m_and_n I don't see anything there that applies directly to our situation. Thanks, - Kim _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
