I'm aware that what I'm asking for is not be very portable between sql-engines, and that is fine. It does not affect core-product/value, just infrastructure. Also, this is about optimizing infrastructure work, so nice if it works, no biggi if it does not.
But the thing we want to optimize is the size we need to copy to move/copy db to different nodes. Today we must copy the full 20Gb+ to accomplish this, and this will also be required in your text-file solution. One possible solution is to store a read-only "base-image" of the db in a network-mounted folder, and only copy over the changes since base-image was created. This will reduce copy-time to almost zero in our case. Data is accumulated over a long time so day-to-day changes are small (<1Mb?). Also, I know that moving database around is not a very common requirement, but this is something we often do. Anyway, I will look more into the WAL-system when I get some time. If someone know for sure what I'm planning to do will crash and burn, I will appriciate a heads-up on why. Fredrik On Mon, Oct 7, 2019 at 11:42 AM Simon Slavin <slav...@bigfraud.org> wrote: > On 7 Oct 2019, at 10:34am, Fredrik Larsen <frel...@gmail.com> wrote: > > > In my head, checkpointing implies copying back all dirty-pages from the > WAL/COW-log to the main db-file. If we never checkpoint, the writes are > still completed, but lives in the WAL-file. We will offcourse merge back > pages to the main db-file, but this would be an offline process we start > when we want to create a new "base-image" of the db. For this to work, it > is very important that the main db-file is untouched until we actively want > to update this file. > > Your use of SQLite depends on several obscure facts about how SQLite > works. If you ever want to change implementation details you're going to > need someone who understands SQLite very thoroughly. > > Had you considered just making a text file of all the SQL commands > executed, and running that against your main database file ? It would take > less CPU time, less filespace, be easier to debug, and be simpler for > another programmer to understand. > _______________________________________________ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
