On Tue, Jun 12, 2018 at 12:49 PM Clemens Ladisch <clem...@ladisch.de> wrote:
> Dominique Devienne wrote:
> > In JOURNAL mode, new data goes to DB file directly, and modified pages
> go to the JOURNAL file.
> > And since here this is INSERT-only, from empty tables, I assumed pages
> copied to the JOURNAL
> > file should be minimal.
> Yes. You can check the journal size with PRAGMA journal_mode = PERSIST.
> > ... --blobs-as-hdf5=false --blobs-inline=false
> > Write ROWs = 4.358s (12.6%)
> > Write BLOBs = 3.130s ( 9.0%)
> > Epilog = 10.221s (29.5%)
> > ... --blobs-as-hdf5=true --blobs-inline=false
> > Write ROWs = 4.368s (17.0%)
> > Write BLOBs = 4.179s (16.3%)
> > Epilog = 0.461s ( 1.8%)
> This looks as if HDF5 does not do fsync().
> Compare with PRAGMA synchronous = OFF, or run
> after you've finished with the HDF5 file.
You're right of course. Thank you Clemens.
With synchronous = OFF, which suits my use-case here, the commit-time just
and even out-performs HDF5 now (see below). I might still prefer HDF5,
the N-d typed arrays are self-describing and thus visible using Hdf5View
for example, instead
of completely opaque in SQLite, but now I know SQLite in non-durable mode
is as fast or even
faster for these smallish blob sizes than HDF5. I'll need to double-check
for larger sizes
(individual blobs get upward of 1GB in size), with my large 31GB, 200K
blobs, 1.2M entities project.
So thanks again Clemens for the guidance. Amazing job from SQLite, as
D:\>... --export-sqlite GL-blobs-outofline-sync-OFF.db ...
Total = 23.455s
Write ROWs = 4.110s (17.5%)
Write BLOBs = 2.822s (12.0%)
Epilog = 0.005s ( 0.0%)
D:\>... --export-sqlite GL-blobs-ashdf5-sync-OFF.db ...
Total = 25.238s
Write ROWs = 4.464s (17.7%)
Write BLOBs = 4.358s (17.3%)
Epilog = 0.020s ( 0.1%)
06/12/2018 01:49 PM 64,475,136 GL-blobs-ashdf5-sync-OFF.db
06/12/2018 01:49 PM 1,039,652,416 GL-blobs-ashdf5-sync-OFF.h5
06/12/2018 01:48 PM 1,111,736,320 GL-blobs-outofline-sync-OFF.db
sqlite-users mailing list