On Tue, Jun 12, 2018 at 12:49 PM Clemens Ladisch <clem...@ladisch.de> wrote:
> Dominique Devienne wrote: > > In JOURNAL mode, new data goes to DB file directly, and modified pages > go to the JOURNAL file. > > And since here this is INSERT-only, from empty tables, I assumed pages > copied to the JOURNAL > > file should be minimal. > > Yes. You can check the journal size with PRAGMA journal_mode = PERSIST. > > > ... --blobs-as-hdf5=false --blobs-inline=false > > Write ROWs = 4.358s (12.6%) > > Write BLOBs = 3.130s ( 9.0%) > > Epilog = 10.221s (29.5%) > > > > ... --blobs-as-hdf5=true --blobs-inline=false > > Write ROWs = 4.368s (17.0%) > > Write BLOBs = 4.179s (16.3%) > > Epilog = 0.461s ( 1.8%) > > This looks as if HDF5 does not do fsync(). > > Compare with PRAGMA synchronous = OFF, or run > https://docs.microsoft.com/en-us/sysinternals/downloads/sync > after you've finished with the HDF5 file. > You're right of course. Thank you Clemens. With synchronous = OFF, which suits my use-case here, the commit-time just vanishes, and even out-performs HDF5 now (see below). I might still prefer HDF5, mainly because the N-d typed arrays are self-describing and thus visible using Hdf5View for example, instead of completely opaque in SQLite, but now I know SQLite in non-durable mode is as fast or even faster for these smallish blob sizes than HDF5. I'll need to double-check for larger sizes (individual blobs get upward of 1GB in size), with my large 31GB, 200K blobs, 1.2M entities project. So thanks again Clemens for the guidance. Amazing job from SQLite, as usual. --DD D:\>... --export-sqlite GL-blobs-outofline-sync-OFF.db ... --blobs-as-hdf5=false --blobs-inline=false ... Total = 23.455s ... Write ROWs = 4.110s (17.5%) Write BLOBs = 2.822s (12.0%) Epilog = 0.005s ( 0.0%) D:\>... --export-sqlite GL-blobs-ashdf5-sync-OFF.db ... --blobs-as-hdf5=true --blobs-inline=false ... Total = 25.238s ... Write ROWs = 4.464s (17.7%) Write BLOBs = 4.358s (17.3%) Epilog = 0.020s ( 0.1%) 06/12/2018 01:49 PM 64,475,136 GL-blobs-ashdf5-sync-OFF.db 06/12/2018 01:49 PM 1,039,652,416 GL-blobs-ashdf5-sync-OFF.h5 06/12/2018 01:48 PM 1,111,736,320 GL-blobs-outofline-sync-OFF.db _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
