On Mar 27 2019, Cliff Stanford <[email protected]> wrote: > This is the current output of s3qlstat: > > Directory entries: 20015509 > Inodes: 20015512 > Data blocks: 2101939 > Total data size: 3.25 TB > After de-duplication: 442 GiB (13.28% of total) > After compression: 320 GiB (9.62% of total, 72.47% of de-duplicated) > Database size: 3.02 GiB (uncompressed) > Cache size: 4.00 GiB, 3045 entries > Cache size (dirty): 0 bytes, 0 entries > Queued object removals: 0 > > Is this too big to be sensible?
I wouldn't say that, no. > Both s3qlstat and df take a very long time to respond after s3ql has > been running for a while. By a long time, I mean several minutes. If > I rerun the command, it takes between 1 and 4 seconds. This is very odd. It sounds like it might be a caching issue (both commands have to read the entire DB), but several minutes to read 3 GB sounds like a lot.. Could you try to manually run the relevant SQL commands inside the SQLite CLI? They are (from https://github.com/s3ql/s3ql/blob/master/src/s3ql/fs.py#L911): SELECT COUNT(id) FROM objects SELECT COUNT(id) FROM inodes SELECT SUM(size) FROM block (The database is the *.db file in ~/.s3ql). Do you get the same behavior here? > If it crashes, fsck, working from the local database, takes over 45 > minutes to run. That's possible. There's a number of optimization opportunities here, but since running fsck.s3ql should be the exception, no one is working on them :-). > I'm wondering what the best way to deal with this is. > > 1. Break the directory down into separate s3ql mounts for each backup type? > > 2. Move the .s3ql directory to local SSD? (Less secure than the Ceph > drive)? Aah.. Ceph is some network filesystem, right? In that case that is your problem. Not sure why that'd be "less secure" though - secure against what? Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
