I am using the errlog interface, which is why I have line numbers for where its
occurring. The location varies with some clusters around
R-47608-56469, and /* Freeblock off the end of the page */, although there are
other locations as well.
>Try not using memory mapping. I don’t have a specific reason to think it’s
>causing the problem but it’s easy to turn off without modifying much of your
>source code, and > it has caused problems on one platform in the past.
I only recently turned it on in order to relieve memory pressure from other
parts of the system. Given how low the frequency is (something in the
thousandths of a percent chance in a given day for a given device [It was
~0.0073%, its currently ~0.0185%), its hard to know when there is a signal vs
>I forgot to ask whether you’re using simultaneously accessing the database
>from multiple threads or processes or applications. If you’re doing this,
>please give details, including of your disk format, since this can be an easy
>cause of corruption. If not, this simplifies things considerably.
I am using multiple threads and processes to access the databases. The library
is configured as multi-thread, with the connection handles mutexed by the
application. Most statements have their sqlite3_stmt get cached (and these
caches are per-connection handle since the prepared statements only work
against a given handle). The page size is configured at 4k (although I suppose
that is the default now). The database files grow in 1 MiB increments (using
sqlite3_file_control). Most database handles have multiple databases attached
(with main being a database with an empty schema).
Having multiple handles per database is in part why switching to mmap saved
some memory pressure (since page cache isn't shared without shared_cache).
Disk is formatted as ext3 (linux 2.6.31 kernel; relevant mount options:
relatime,errors=continue,barrier=1,data=ordered). Writes are not actually
getting coalesced by the kernel disk scheduling (there are reasons for this,
but none of which are relevant for this).
From: sqlite-users <sqlite-users-boun...@mailinglists.sqlite.org> on behalf of
Simon Slavin <slav...@bigfraud.org>
Sent: Tuesday, November 7, 2017 8:43 AM
To: SQLite mailing list
Subject: Re: [sqlite] Tracking database corruption
On 7 Nov 2017, at 1:11am, Nicholas Lovell <nicholas.lov...@tivo.com> wrote:
> In terms of pragmas during normal usage: user_version, recursive_triggers,
> cache_size, mmap_size (max size is set to 15 MiB; which covers the files in
> question entirely), journal_mode (=wal). The only somewhat recent addition is
> mmap (along with an upgrade from 22.214.171.124 to 3.16.2).
Try not using memory mapping. I don’t have a specific reason to think it’s
causing the problem but it’s easy to turn off without modifying much of your
source code, and it has caused problems on one platform in the past.
> These databases are stored on a local disk (admittedly using an encrypting
> loop-back adapter). I doubt that I'll necessarily be able to track down what
> is causing the corruption given how infrequently it ends up occurring,
> storing the corrupt copies is my attempt at being able to track them down
> (since they are very infrequent and automatically "recovered" from).
Encrypting adapters should not be causing a problem.
I forgot to ask whether you’re using simultaneously accessing the database from
multiple threads or processes or applications. If you’re doing this, please
give details, including of your disk format, since this can be an easy cause of
corruption. If not, this simplifies things considerably.
sqlite-users mailing list
This email and any attachments may contain confidential and privileged material
for the sole use of the intended recipient. Any review, copying, or
distribution of this email (or any attachments) by others is prohibited. If you
are not the intended recipient, please contact the sender immediately and
permanently delete this email and any attachments. No employee or agent of TiVo
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by
email. Binding agreements with TiVo Inc. may only be made by a signed written
sqlite-users mailing list