I would think you're write -- you're writing your data into a log, then writing into the DB, that seems like it'd be slower. But...
WAL docs (https://sqlite.org/wal.html) say "WAL is significantly faster in most scenarios." (... to be fair they're comparing it to the regular journal mode, though, not journal_mode=off.) They say WAL provides more concurrency (readers and writers don't usually block each other... I"m assuming the main thread, and the writeback threads, can block each other's access to the DB if only momentrarily), inserts are put into the WAL sequentially which is fast (and then I assume put into the DB at sqlite's leisure, when the DB is not busy), and WAL uses far fewer fsync() calls. That said, on an SSD it may just mean "more writes" (one into WAL and one into DB) given the typical SSD's near-0 latency and high throughput. Might be a bit (like later in the week) but I can do that -- pick out some test data (probably 1 test with larger files, 1 with smaller), copy it in, copy it out, s3qlcp, and finally remove both the 1st and s3qlcp'ed copy (so I don't have deduplication throw off the later test runs.) Test with and without WAL. I'll make sure to NOT test on my 8TB disk -- it's SMR so you give it like 100GB writes without enough of a breather and it's suddenly dead slow 8-) . Thanks! --Henry On Saturday, September 11, 2021 at 6:20:32 AM UTC-5 [email protected] wrote: > On Sep 10 2021, Henry Wertz <[email protected]> wrote: > > What do you think about enabling WAL (Write Ahead Logging)? > > > [...] > > > > I didn't benchmark anything, but rsync'ing in small files is visibly > > faster, and the file system is better under load (i.e. I can copy stuff > in > > and any simultaneous directory lookups, copying stuff out, etc. is > > noticeably faster and more responsive.) > > As I understand, WAL should not result in any speed-ups, it just > improves reliability in case of a crash. So I'd be very interested to > see actual benchmark data here rather than subjective impressions :-). > > Best, > -Nikolaus > > -- > GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F > > »Time flies like an arrow, fruit flies like a Banana.« > -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/c2fb3dbe-1a5a-4602-a041-06a24f399a66n%40googlegroups.com.
