I am using the errlog interface, which is why I have line numbers for where its occurring. The location varies with some clusters around
R-47608-56469, and /* Freeblock off the end of the page */, although there are other locations as well. >Try not using memory mapping. I don’t have a specific reason to think it’s >causing the problem but it’s easy to turn off without modifying much of your >source code, and > it has caused problems on one platform in the past. I only recently turned it on in order to relieve memory pressure from other parts of the system. Given how low the frequency is (something in the thousandths of a percent chance in a given day for a given device [It was ~0.0073%, its currently ~0.0185%), its hard to know when there is a signal vs just noise. >I forgot to ask whether you’re using simultaneously accessing the database >from multiple threads or processes or applications. If you’re doing this, >please give details, including of your disk format, since this can be an easy >cause of corruption. If not, this simplifies things considerably. I am using multiple threads and processes to access the databases. The library is configured as multi-thread, with the connection handles mutexed by the application. Most statements have their sqlite3_stmt get cached (and these caches are per-connection handle since the prepared statements only work against a given handle). The page size is configured at 4k (although I suppose that is the default now). The database files grow in 1 MiB increments (using sqlite3_file_control). Most database handles have multiple databases attached (with main being a database with an empty schema). Having multiple handles per database is in part why switching to mmap saved some memory pressure (since page cache isn't shared without shared_cache). Disk is formatted as ext3 (linux 2.6.31 kernel; relevant mount options: relatime,errors=continue,barrier=1,data=ordered). Writes are not actually getting coalesced by the kernel disk scheduling (there are reasons for this, but none of which are relevant for this). ________________________________ From: sqlite-users <sqlite-users-boun...@mailinglists.sqlite.org> on behalf of Simon Slavin <slav...@bigfraud.org> Sent: Tuesday, November 7, 2017 8:43 AM To: SQLite mailing list Subject: Re: [sqlite] Tracking database corruption On 7 Nov 2017, at 1:11am, Nicholas Lovell <nicholas.lov...@tivo.com> wrote: > In terms of pragmas during normal usage: user_version, recursive_triggers, > cache_size, mmap_size (max size is set to 15 MiB; which covers the files in > question entirely), journal_mode (=wal). The only somewhat recent addition is > mmap (along with an upgrade from 3.8.8.3 to 3.16.2). Try not using memory mapping. I don’t have a specific reason to think it’s causing the problem but it’s easy to turn off without modifying much of your source code, and it has caused problems on one platform in the past. > These databases are stored on a local disk (admittedly using an encrypting > loop-back adapter). I doubt that I'll necessarily be able to track down what > is causing the corruption given how infrequently it ends up occurring, > storing the corrupt copies is my attempt at being able to track them down > (since they are very infrequent and automatically "recovered" from). Encrypting adapters should not be causing a problem. I forgot to ask whether you’re using simultaneously accessing the database from multiple threads or processes or applications. If you’re doing this, please give details, including of your disk format, since this can be an easy cause of corruption. If not, this simplifies things considerably. Simon. _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ________________________________ This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement. _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
