Re: [sqlite] Copy-on-write VFS

On Monday, 7 October, 2019 03:16, Simon Slavin wrote: >> Given that we disable ceckpointing, can we assume that the main >> database-file will never be modified, and therefor could potentially be >> mounted read-only? No, and No. >You are correct that the WAL file contains database pages,

Re: [sqlite] Copy-on-write VFS

I'm aware that what I'm asking for is not be very portable between sql-engines, and that is fine. It does not affect core-product/value, just infrastructure. Also, this is about optimizing infrastructure work, so nice if it works, no biggi if it does not. But the thing we want to optimize is the

Re: [sqlite] Copy-on-write VFS

On 7 Oct 2019, at 10:34am, Fredrik Larsen wrote: > In my head, checkpointing implies copying back all dirty-pages from the > WAL/COW-log to the main db-file. If we never checkpoint, the writes are still > completed, but lives in the WAL-file. We will offcourse merge back pages to > the main

Re: [sqlite] Copy-on-write VFS

Hi Simon In my head, checkpointing implies copying back all dirty-pages from the WAL/COW-log to the main db-file. If we never checkpoint, the writes are still completed, but lives in the WAL-file. We will offcourse merge back pages to the main db-file, but this would be an offline process we

Re: [sqlite] Copy-on-write VFS

On 7 Oct 2019, at 9:56am, Fredrik Larsen wrote: > nowhere does it warn about severe regression with > unbounded WAL-size [snip] There are tons of bad stuff the documentation doesn't warn you about. You might want to read

Re: [sqlite] Copy-on-write VFS

sable checkpointing? > > -- > The fact that there's a Highway to Hell but only a Stairway to Heaven says > a lot about anticipated traffic volume. > > >-Original Message- > >From: sqlite-users On > >Behalf Of Fredrik Larsen > >Sent: Friday, 4 October, 2019

Re: [sqlite] Copy-on-write VFS

nticipated traffic volume. >-Original Message- >From: sqlite-users On >Behalf Of Fredrik Larsen >Sent: Friday, 4 October, 2019 14:14 >To: SQLite mailing list >Subject: Re: [sqlite] Copy-on-write VFS > >A file-system with COW support would work, but that is a b

Re: [sqlite] Copy-on-write VFS

Applying the dump from session extension will modify the db, so you need to copy all data to the node that will run sqlite, then apply changes, then start db. A COW system would allow you to to store the bulk of the data in a shared and network-mounted folder, and only copy over the changes since

Re: [sqlite] Copy-on-write VFS

A file-system with COW support would work, but that is a big dependency/constraint to bring into a project, and not always possible/practical. A file based version (snapshot + changes) will be more practical and easier to manage, and also very doable IMHO. Anyway, I was just wondering if anyone

Re: [sqlite] Copy-on-write VFS

lite-users [sqlite-users-boun...@mailinglists.sqlite.org] on behalf of Keith Medcalf [kmedc...@dessus.com] Sent: Friday, October 04, 2019 12:19 PM To: SQLite mailing list Subject: Re: [sqlite] Copy-on-write VFS On Friday, 4 October, 2019 05:18, Fredrik Larsen wrote: >A copy-on-write IO-path where

Re: [sqlite] Copy-on-write VFS

On Friday, 4 October, 2019 05:18, Fredrik Larsen wrote: >A copy-on-write IO-path where data is split into static and dynamic parts >(think snapshots for storage) would be very helpful for our project, . What do you mean? Useful how? >This would simplify backups, testing, moving data around in

Re: [sqlite] Copy-on-write VFS

On 4 Oct 2019, at 12:17pm, Fredrik Larsen wrote: > A copy-on-write IO-path where data is split into static and dynamic parts > (think snapshots for storage) would be very helpful for our project. SQLite abstracts changes-only tracking at the transaction level, and provides it as the session

Re: [sqlite] Copy-on-write VFS

Hello Fredrik, Why does it need to be part of a VFS instead of using a file system with COW like ZFS? On Fri, 4 Oct 2019 at 12:18, Fredrik Larsen wrote: > Hi > > A copy-on-write IO-path where data is split into static and dynamic parts > (think snapshots for storage) would be very helpful for

[sqlite] Copy-on-write VFS

Hi A copy-on-write IO-path where data is split into static and dynamic parts (think snapshots for storage) would be very helpful for our project, . This would simplify backups, testing, moving data around in a multinode environment, etc. Does something like this exist for sqlite? In my head this