Re: [sqlite] Multiple sessions, page sizes and cache

Hi Richard, I got side-tracked with other work, but am coming back to this now. On Sun, Jan 15, 2017 at 5:50 PM, Richard Hipp <d...@sqlite.org> wrote: > On 1/15/17, Kim Gräsman <kim.gras...@gmail.com> wrote: >> >> 1) If I configure a global SQLite heap with SQLITE_C

Re: [sqlite] BUG: Illegal initialization in icu.c : sqlite3IcuInit

On Thu, Jan 26, 2017 at 10:08 PM, David Empson wrote: > >> On 26/01/2017, at 8:46 PM, Clemens Ladisch wrote: >> >> Ziemowit Laski wrote: >>> Visual C++ >> >> Which one? >> >>> correctly catches this. >> >> Oh? What exactly is illegal about this? >> >>>

Re: [sqlite] BUG: Illegal initialization in icu.c : sqlite3IcuInit

David, On Fri, Jan 27, 2017 at 12:51 PM, David Empson <demp...@emptech.co.nz> wrote: > >> On 27/01/2017, at 9:09 PM, Kim Gräsman <kim.gras...@gmail.com> wrote: >> >> On Thu, Jan 26, 2017 at 10:08 PM, David Empson <demp...@emptech.co.nz >> <mailto:dem

Re: [sqlite] Smallest reasonable cache size

On Fri, Feb 17, 2017 at 8:37 PM, Richard Hipp <d...@sqlite.org> wrote: > On 2/17/17, Kim Gräsman <kim.gras...@gmail.com> wrote: >> >> Thanks, shrink_memory could come in handy. But this is not the problem >> we're trying to solve -- rather we want to claim as mu

Re: [sqlite] Smallest reasonable cache size

Thanks for your responses! On Thu, Feb 16, 2017 at 8:56 PM, Dominique Pellé wrote: > > I think it's probably best to give a large cache_size to > each connection, and limit the total amount of memory > used by SQLite with sqlite3_soft_heap_limit64(). > This will

Re: [sqlite] Smallest reasonable cache size

Den 17 feb. 2017 5:33 em skrev "Richard Hipp" <d...@sqlite.org>: On 2/17/17, Kim Gräsman <kim.gras...@gmail.com> wrote: > > If we let it use malloc directly and control it > with the soft heap limit, we'll have a steadily rising memory curve > until all caches

Re: [sqlite] Multiple connections, page sizes and cache

mory, increasing overall memory usage. Would something like "PRAGMA cache_allocator=malloc" be a crazy idea? Thanks, - Kim On Sat, Feb 4, 2017 at 3:21 PM, Kim Gräsman <kim.gras...@gmail.com> wrote: > Hi Richard, > > I got side-tracked with other work, but am coming back to this

[sqlite] Smallest reasonable cache size

Hi all, In my battles with lots of connections competing over precious cache memory, I've considered giving some non-critical connections zero cache using `PRAGMA cache_size=0`. Is this a reasonable thing to do? If zero is too extreme, what might a more moderate small cache size be? 32? 64? 100?

Re: [sqlite] Multiple sessions, page sizes and cache

Hi Clemens, Thanks for your help! On Sat, Jan 14, 2017 at 9:54 AM, Clemens Ladisch <clem...@ladisch.de> wrote: > Kim Gräsman wrote: >> I would expect SQLite to allocate a page cache per session. > > There is typically one page cache instance per connection. > (A connect

[sqlite] Multiple sessions, page sizes and cache

Hi all, We have an application with multiple databases of different page sizes. We keep a few long-running sessions, but also open transient sessions occasionally. In an effort to reduce memory usage/fragmentation, we're trying to configure SQLite to allocate as much memory as necessary up-front

Re: [sqlite] Multiple sessions, page sizes and cache

On Sat, Jan 14, 2017 at 1:09 PM, Simon Slavin wrote: > > On 14 Jan 2017, at 8:54am, Clemens Ladisch wrote: > >> If your use case is more complex than the simple mechanism offered by >> SQLITE_CONFIG_PAGECACHE, consider using SQLITE_CONFIG_PCACHE2 (or >>

Re: [sqlite] Multiple sessions, page sizes and cache

Hi Richard, On Sat, Jan 14, 2017 at 2:17 PM, Richard Hipp <d...@sqlite.org> wrote: > On 1/13/17, Kim Gräsman <kim.gras...@gmail.com> wrote: >> >> In an effort to reduce memory usage/fragmentation, we're trying to >> configure SQLite to allocate as much memor

Re: [sqlite] Multiple sessions, page sizes and cache

Hi all, On Sun, Jan 15, 2017 at 5:50 PM, Richard Hipp <d...@sqlite.org> wrote: > On 1/15/17, Kim Gräsman <kim.gras...@gmail.com> wrote: >> >> 1) If I configure a global SQLite heap with SQLITE_CONFIG_HEAP, won't >> I just trade malloc heap fragmentation for SQL

Re: [sqlite] Multiple sessions, page sizes and cache

On Fri, Apr 21, 2017 at 6:05 PM, Simon Slavin <slav...@bigfraud.org> wrote: > > On 21 Apr 2017, at 4:18pm, Kim Gräsman <kim.gras...@gmail.com> wrote: > >> Unfortunately, we're getting occasional out-of-memory errors on >> statement execution. > > Is there a

Re: [sqlite] SQLITE_OMIT_* vs amalgamation

On Wed, Apr 19, 2017 at 1:18 PM, Richard Hipp <d...@sqlite.org> wrote: > On 4/19/17, Kim Gräsman <kim.gras...@gmail.com> wrote: >> I'm building for both Windows and Linux and it's a reproducible build, >> so I'd rather not depend on anything extra. > > The o

Re: [sqlite] Multiple sessions, page sizes and cache

On Fri, Apr 21, 2017 at 6:21 PM, Simon Slavin <slav...@bigfraud.org> wrote: > > On 21 Apr 2017, at 5:16pm, Kim Gräsman <kim.gras...@gmail.com> wrote: > >> Could be. Would that show up in EXPLAIN somehow? > > You would find it easier to spot using EXPLAIN QUERY

Re: [sqlite] Query default lookaside pool size

On Wed, Mar 8, 2017 at 2:41 PM, Simon Slavin <slav...@bigfraud.org> wrote: > > On 8 Mar 2017, at 11:09am, Kim Gräsman <kim.gras...@gmail.com> wrote: > >> Is there a way to query SQLite build parameters at runtime, more >> specifically SQLITE_DEFAULT_LOOKA

[sqlite] Query default lookaside pool size

Hi all, Is there a way to query SQLite build parameters at runtime, more specifically SQLITE_DEFAULT_LOOKASIDE? Thanks, - Kim ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org

Re: [sqlite] Minor doc patches

On Mon, Mar 6, 2017 at 12:36 PM, Richard Hipp <d...@sqlite.org> wrote: > On 3/6/17, Kim Gräsman <kim.gras...@gmail.com> wrote: >> Hi all, >> >> I'm reading the documentation on memory >> (https://www.sqlite.org/malloc.html) again and stumbled on some typos: &

[sqlite] Minor doc patches

Hi all, I'm reading the documentation on memory (https://www.sqlite.org/malloc.html) again and stumbled on some typos: > 3.3 Page cache memory > [... ] It is not unusual to see _the database page cache consumes_ over 10 > times more memory than the rest of SQLite combined. This should read

[sqlite] SQLITE_OMIT_* vs amalgamation

Hi all, The docs at https://sqlite.org/compile.html are a little ominous when it comes to the OMIT flags with the amalgamation: > Important Note: The SQLITE_OMIT_* options may not work with the amalgamation. > SQLITE_OMIT_* compile-time options usually work correctly only when SQLite is > built

Re: [sqlite] SQLITE_OMIT_* vs amalgamation

amalgamation) is straightforward. > You may have to install a few extras, but nothing onerous. > > > > On Wed, Apr 19, 2017 at 5:57 AM, Kim Gräsman <kim.gras...@gmail.com> wrote: >> Hi all, >> >> The docs at https://sqlite.org/compile.html are a little ominous when

[sqlite] -shm grows with large transaction

Hi again, I've been experimenting with limiting memory usage in our SQLite-based app. Ran into an unrelated oddity that I thought I'd ask about: We're running a couple of massive upgrade steps on over 5 million quite large (70+ columns) rows. There are two unrelated steps; 1) DROP

Re: [sqlite] -shm grows with large transaction

Den 26 apr. 2017 3:45 em skrev "Richard Hipp" : > On 4/26/17, Richard Hipp wrote: > > That would imply you are changing about a > > half million pages of your database inside a single transaction. > > Correction: About 5 million pages. Missed a zero. (Time

Re: [sqlite] -shm grows with large transaction

On Wed, Apr 26, 2017 at 5:58 PM, Dominique Devienne <ddevie...@gmail.com> wrote: > On Wed, Apr 26, 2017 at 5:34 PM, Kim Gräsman <kim.gras...@gmail.com> wrote: > >> Great, that means the numbers add up. This is a monster transaction >> updating 5M rows, and page si

[sqlite] SQLite memory calculator (was Multiple sessions, page sizes and cache)

On Fri, Apr 21, 2017 at 5:18 PM, Kim Gräsman <kim.gras...@gmail.com> wrote: > Hi all, > > On Sun, Jan 15, 2017 at 5:50 PM, Richard Hipp <d...@sqlite.org> wrote: >> On 1/15/17, Kim Gräsman <kim.gras...@gmail.com> wrote: >>> >>> 1) If I configure

Re: [sqlite] SQLite memory calculator (was Multiple sessions, page sizes and cache)

On Wed, Apr 26, 2017 at 11:00 PM, Kim Gräsman <kim.gras...@gmail.com> wrote: > > I've used a spreadsheet internally to forecast how much memory SQLite > will need in a given configuration. > > I've cleaned/generalised it a little and posted here: > https://docs.g

Re: [sqlite] SQLite memory calculator (was Multiple sessions, page sizes and cache)

Hi Simon, On Wed, Apr 26, 2017 at 11:46 PM, Simon Slavin <slav...@bigfraud.org> wrote: > > On 26 Apr 2017, at 10:00pm, Kim Gräsman <kim.gras...@gmail.com> wrote: > >> Specifically, I wonder if 4MiB for the general-purpose heap is maybe >> entirely unreasonable? Is

Re: [sqlite] Multiple sessions, page sizes and cache

On Fri, Apr 21, 2017 at 5:18 PM, Kim Gräsman <kim.gras...@gmail.com> wrote: > > I have a nagging suspicion that the tempdb is allocated from the > private heap (we've set pragma temp_store=memory) -- what are the > characteristics of that database? Can I

Re: [sqlite] PRAGMA cache_size=0 increases memory usage

Hi Bob, Den 18 maj 2017 9:15 em skrev "Bob Friesenhahn" < bfrie...@simple.dallas.tx.us>: On Thu, 18 May 2017, Kim Gräsman wrote: > >> The request is issued early on when the connection is first opened so no >> actual queries have been issued at that time. &

Re: [sqlite] PRAGMA cache_size=0 increases memory usage

On Thu, May 18, 2017 at 8:27 PM, Bob Friesenhahn <bfrie...@simple.dallas.tx.us> wrote: > On Thu, 18 May 2017, Kim Gräsman wrote: > >> On Thu, May 18, 2017 at 6:10 PM, Bob Friesenhahn >> <bfrie...@simple.dallas.tx.us> wrote: >>> >>> Does anyone kno

Re: [sqlite] PRAGMA cache_size=0 increases memory usage

On Thu, May 18, 2017 at 6:10 PM, Bob Friesenhahn wrote: > Does anyone know why using 'PRAGMA cache_size=0' (or some other small value) > to attempt to decrease memory usage (and it is reported as immediately > decreased in the shell by .stats) actually significantly

[sqlite] Duplicate documentation

Hi all, I just recently realized that there are two sets of documentation for the SQLite configuration options: https://www.sqlite.org/c3ref/c_config_getmalloc.html (this looks outdated) and https://www.sqlite.org/c3ref/c_config_covering_index_scan.html Similarly for the DB configuration verbs: