Re: [sqlite] Performance of writing blobs

2018-06-12 Thread Eduardo Morras
On Tue, 12 Jun 2018 14:13:33 +0200 Dominique Devienne wrote: > > You're right of course. Thank you Clemens. > > With synchronous = OFF, which suits my use-case here, the commit-time > just vanishes, > and even out-performs HDF5 now (see below). I might still prefer HDF5, > mainly because > the

Re: [sqlite] Performance of writing blobs

2018-06-12 Thread Dominique Devienne
On Tue, Jun 12, 2018 at 12:49 PM Clemens Ladisch wrote: > Dominique Devienne wrote: > > In JOURNAL mode, new data goes to DB file directly, and modified pages > go to the JOURNAL file. > > And since here this is INSERT-only, from empty tables, I assumed pages > copied to the JOURNAL > > file shou

Re: [sqlite] Performance of writing blobs

2018-06-12 Thread Clemens Ladisch
Dominique Devienne wrote: > In JOURNAL mode, new data goes to DB file directly, and modified pages go to > the JOURNAL file. > And since here this is INSERT-only, from empty tables, I assumed pages copied > to the JOURNAL > file should be minimal. Yes. You can check the journal size with PRAGMA

Re: [sqlite] Performance of writing blobs

2018-06-12 Thread Dominique Devienne
On Tue, Jun 12, 2018 at 8:03 AM Clemens Ladisch wrote: > Dominique Devienne wrote: > > On Mon, Jun 11, 2018 at 4:27 PM Clemens Ladisch > wrote: > >> It does write to the same pages, but those pages must be copied to the > >> rollback journal so that they can be restored if the transaction is > >

Re: [sqlite] Performance of writing blobs

2018-06-11 Thread Clemens Ladisch
Dominique Devienne wrote: > On Mon, Jun 11, 2018 at 4:27 PM Clemens Ladisch wrote: >> It does write to the same pages, but those pages must be copied to the >> rollback journal so that they can be restored if the transaction is >> rolled back. (Or are the two passes inside the same transaction?)

Re: [sqlite] Performance of writing blobs

2018-06-11 Thread Dominique Devienne
On Mon, Jun 11, 2018 at 4:27 PM Clemens Ladisch wrote: > Dominique Devienne wrote: > > My assumption > > was that after the zeroblob(N), there was enough room in the main DBs > > pages, such that the subsequent blob open+write+close did not need to > > generate any "page churn" (i.e. journal acti

Re: [sqlite] Performance of writing blobs

2018-06-11 Thread Clemens Ladisch
Dominique Devienne wrote: > My assumption > was that after the zeroblob(N), there was enough room in the main DBs > pages, such that the subsequent blob open+write+close did not need to > generate any "page churn" (i.e. journal activity) and could write directly > to the pages created on initial in