I am using the errlog interface, which is why I have line numbers for where its 
occurring. The location varies with some clusters around

R-47608-56469, and /* Freeblock off the end of the page */, although there are 
other locations as well.


>Try not using memory mapping.  I don’t have a specific reason to think it’s 
>causing the problem but it’s easy to turn off without modifying much of your 
>source code, and > it has caused problems on one platform in the past.

I only recently turned it on in order to relieve memory pressure from other 
parts of the system. Given how low the frequency is (something in the 
thousandths of a percent chance in a given day for a given device [It was 
~0.0073%, its currently ~0.0185%), its hard to know when there is a signal vs 
just noise.

>I forgot to ask whether you’re using simultaneously accessing the database 
>from multiple threads or processes or applications.  If you’re doing this, 
>please give details, including of your disk format, since this can be an easy 
>cause of corruption.  If not, this simplifies things considerably.

I am using multiple threads and processes to access the databases. The library 
is configured as multi-thread, with the connection handles mutexed by the 
application. Most statements have their sqlite3_stmt get cached (and these 
caches are per-connection handle since the prepared statements only work 
against a given handle).  The page size is configured at 4k (although I suppose 
that is the default now). The database files grow in 1 MiB increments (using 
sqlite3_file_control). Most database handles have multiple databases attached 
(with main being a database with an empty schema).

Having multiple handles per database is in part why switching to mmap saved 
some memory pressure (since page cache isn't shared without shared_cache).

Disk is formatted as ext3 (linux 2.6.31 kernel; relevant mount options: 
relatime,errors=continue,barrier=1,data=ordered).  Writes are not actually 
getting coalesced by the kernel disk scheduling (there are reasons for this, 
but none of which are relevant for this).



________________________________
From: sqlite-users <sqlite-users-boun...@mailinglists.sqlite.org> on behalf of 
Simon Slavin <slav...@bigfraud.org>
Sent: Tuesday, November 7, 2017 8:43 AM
To: SQLite mailing list
Subject: Re: [sqlite] Tracking database corruption



On 7 Nov 2017, at 1:11am, Nicholas Lovell <nicholas.lov...@tivo.com> wrote:

> In terms of pragmas during normal usage: user_version, recursive_triggers, 
> cache_size, mmap_size (max size is set to 15 MiB; which covers the files in 
> question entirely), journal_mode (=wal). The only somewhat recent addition is 
> mmap (along with an upgrade from 3.8.8.3 to 3.16.2).

Try not using memory mapping.  I don’t have a specific reason to think it’s 
causing the problem but it’s easy to turn off without modifying much of your 
source code, and it has caused problems on one platform in the past.


> These databases are stored on a local disk (admittedly using an encrypting 
> loop-back adapter). I doubt that I'll necessarily be able to track down what 
> is causing the corruption given how infrequently it ends up occurring, 
> storing the corrupt copies is my attempt at being able to track them down 
> (since they are very infrequent and automatically "recovered" from).

Encrypting adapters should not be causing a problem.

I forgot to ask whether you’re using simultaneously accessing the database from 
multiple threads or processes or applications.  If you’re doing this, please 
give details, including of your disk format, since this can be an easy cause of 
corruption.  If not, this simplifies things considerably.

Simon.
_______________________________________________
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

________________________________

This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.
_______________________________________________
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Reply via email to