I don't have 10TB in one, but I can tell you on the three I have (default 10MB block size) I have 1 with just over 4,000,000 directory entries, 4.12TB of data using 2.67TB space, it has a 750MB (uncompressed) DB. 2nd has about 166,000 files, 3.31TB using 2.25TB space,, 88.4MB DB. The 3rd, 11,341,879 directory entries (but only 3,812,814 data blocks), 8.27TB data using 3.79TB space, 1.86GB DB. Small files do not use blocks, they are put directly in the DB. The 2nd one mainly has some VirtualBox .OVA files and such and videos (almost all large stuff) while the other 2 are a mix of videos and hard drive backups (not images, rsync, so there's load of tiny files from /etc, /usr/include, and /usr/src that are probably put straight into the DB.)
I'm using it on my USB disks, using local:// backend and putting cache on the same disk (2 of these drives are permanently hooked up, but 1 is portable, keeping cache on same drive is a nice failsafe if I have to move the drive to another computer. I put the s3ql software on the "raw" ext4 filesystem in an s3ql directory (in case I wanted to installed the .deb on a "new" system to access the filesystem), an s3ql-cache directory, s3ql-data, and s3ql-fs mount point, and a shell script that runs s3ql fsck (which returns immediately if there's nothing to fix) and then mount the filesystem (and one to unmount it.). 50GB cache, it runs pretty well! sqlite is nice and robust, I had the cable pop out once or twice and had to repair the DB (but after runming sqlite3 repair fsck fixed things up fine.) And several times I had the drive drop off, I didn't have to do anything with the DB, just run fsck and it came right up.). It does keep 10 backup copies of the database in s3ql-data in case the database really goes sideways. Just like fsck on a conventional filesystem, if you were in the middle of writing stuff when the disk drops off, the fsck will put some stuff in lost+found and delete other blocks that are not pointed to by anything. Speed is actually good, I run the portable (the one with 750MB DB) off a system with 2GB RAM sometimes and it runs comfortably on that. Best solution I've found to have a local filesystem with compression and dedup -- btrfs caused me problems (and was slow), and I found the minimum requirements for ZFS to be too high to look into it. Thanks! --Henry On Wednesday, June 23, 2021 at 5:10:48 AM UTC-5 [email protected] wrote: > > A more serious question, though: what if the machine crashes or loses > > power and I lose the cache contents? Is this handled gracefully in > > fsck.s3ql (any way other than crashing / abandoning the filesystem)? > > fsck.s3ql will just assume that there is no cached data, i.e. that all > dirty data has been flushed and the cache emptied before the crash. > > So the filesystem should continue to work, but you may have stale data > in files or (in case where new data was appended but not yet updated) to > will have holes in the files. > > > Best, > -Nikolaus > > -- > GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F > > »Time flies like an arrow, fruit flies like a Banana.« > -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/7b5c842f-7345-4a93-b423-daaedbf38882n%40googlegroups.com.
