Since we didn't notice any particular regressions or concerns with auxiliary Sqlite files persistence in Nightly and Early Beta, I plan to enable it for Release 117. The enabling will happen in https://bugzilla.mozilla.org/show_bug.cgi?id=1834043.
Cheers, Marco On Wed, May 10, 2023 at 12:15 PM Marco Bonardo <[email protected]> wrote: > *Summary* > > An open Sqlite database connection may use auxiliary files to achieve its > tasks, the most common ones are the shared memory file (*-shm), used to > coordinate concurrent access, and the journal file (*-journal, *-wal) that > contains transactions to be merged. > These files are commonly created when the connection is opened, and > deleted when the connection is closed. > Those creations and deletions have a cost, I indeed could identify async > shutdown crashes where the stack pointed to this file removal operation. > > I intend to test, in Early Beta and Nightly, an option to persist those > auxiliary files. That means even when Firefox is closed, -shm, -journal and > -wal files will remain in the profile folder, alongside the .sqlite files. > Once this change will demonstrate to be stable, I will let it ride to > Release. > This change won't make ESR115, it's more likely to be in 116 or 117. > > To limit the likelihood of users moving around a .sqlite file without its > auxiliary files, or mismatching files (that may either cause dataloss, or > corruption), I am also setting SQLITE_DEFAULT_JOURNAL_SIZE_LIMIT, that will > ensure Sqlite truncates the journal file when closing the connection (plus > ensures Sqlite consumers don't forget about limiting the journal file). > The journal file should thus be a persisted 0-bytes file, unless a crash > or power outage happened. > > If you have questions or concerns about this change, please reach out to > me asap. > > *Notes* > > a. The journal size limit is only enforced on idle and empty journals, a > journal can grow over the set limit, and will be truncated once emptied, > thus when everything has been merged into the main database. > b. Having a journal size limit is not a way to avoid checkpointing your > wal journal, indeed checkpointing is what allows the journal to be emptied, > that is what in the end will enforce the limit and truncate the journal. > c. due to our default large page_size of 32768, the default checkpoint of > 1000 pages is not good, that'd end up generating a 32MiB journal. As a rule > of thumb you can autocheckpoint at about 1/3 of your journal file size > limit (e.g. 1.5MiB size limit => 512KiB checkpoint => 16 pages). > > *References* > https://bugzilla.mozilla.org/show_bug.cgi?id=1820478 > > https://sqlite.org/walformat.html > > https://sqlite.org/c3ref/c_fcntl_begin_atomic_write.html#sqlitefcntlpersistwal > https://sqlite.org/compile.html#default_journal_size_limit > https://sqlite.org/wal.html#ckpt > https://sqlite.org/howtocorrupt.html > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAPDqYT1%2BRrNPL3aC3Xu0NMn3mb1UhnMz5y0pFy4fD11G%2B1b8GQ%40mail.gmail.com.
