On Monday, September 15, 2025 at 4:05:32 AM UTC-4 Nikolaus Rath wrote:
On Mon, 15 Sep 2025, at 02:36, '[email protected]' via s3ql wrote: > Running fsck.s3ql --force on this filesystem takes about 4 days, is there > a way to avoid doing this? Which steps take the longest? Depending on that, --fast may or may not help. I'm currently mounting the file system with a fairly large cache size, so it tends to have a fairly large number of dirty blocks (about 1 million). I will try mounting with a much smaller cache size in future and see if that helps... That's an interesting datapoint, but doesn't answer the question. If you'd like advice on this, you need to tell us which fsck steps take the longest. Best, -Nikolaus The current fsck seems to be taking about the same time as the previous one, the last one was using s3ql 5.3 with the --fast option, and the longest steps were: Checking DB integrity, approx 1 day Uploading metadata, approx 3 days I've appended an edited version of the fsck log at the end of this email, removing some hopefully unimportant info... Here's the about from s3qlstat /mnt/s3ql form about a month ago, in case that's of any use: Directory entries: 940256836 Inodes: 940256838 Data objects: 13035637 Total data size: 147 TB After de-duplication: 2.61 TB (1.76% of total) After compression: 1.23 TB (0.83% of total, 47.15% of de-duplicated) Database size: 109 GiB (uncompressed) Cache size: 89.8 MiB, 90 entries Cache size (dirty): 0 bytes, 0 entries Queued object removals: 0 Thanks! Joseph 2025-09-06 11:42:12.567 1347769:MainThread s3ql.fsck.main: Starting fast fsck of gs://bucketname.../ 2025-09-06 11:42:12.845 1347769:MainThread s3ql.fsck.main: Using cached metadata. 2025-09-06 11:42:12.852 1347769:MainThread s3ql.database.sqlite3_log: sqlite3: recovered 3 frames from WAL file /mnt/backup/c> 2025-09-06 11:42:12.860 1347769:MainThread s3ql.fsck.main: File system was not unmounted cleanly 2025-09-06 11:42:12.860 1347769:MainThread s3ql.fsck.main: Checking DB integrity... 2025-09-07 10:00:56.583 1347769:MainThread s3ql.fsck.check: Creating temporary extra indices... 2025-09-07 11:49:44.169 1347769:MainThread s3ql.fsck.check_lof: Checking lost+found... 2025-09-07 11:49:45.015 1347769:MainThread s3ql.fsck.check_cache: Checking for dirty cache objects... 2025-09-07 11:49:45.061 1347769:MainThread s3ql.fsck.check_names_refcount: Checking names (refcounts)... 2025-09-07 11:51:42.137 1347769:MainThread s3ql.fsck.check_contents_name: Checking contents (names)... 2025-09-07 11:57:56.590 1347769:MainThread s3ql.fsck.check_contents_inode: Checking contents (inodes)... 2025-09-07 12:19:18.962 1347769:MainThread s3ql.fsck.check_contents_parent_inode: Checking contents (parent inodes)... 2025-09-07 12:26:18.833 1347769:MainThread s3ql.fsck.check_objects_hash: Checking objects.hash... 2025-09-07 12:26:18.898 1347769:MainThread s3ql.fsck.check_objects_refcount: Checking objects (reference counts)... 2025-09-07 12:27:50.174 1347769:MainThread s3ql.fsck.check_objects_id: Checking objects (backend)... 2025-09-07 12:29:07.418 1347769:MainThread s3ql.fsck.check_objects_id: Processed 177475 objects so far 2025-09-07 13:38:40.142 1347769:MainThread s3ql.fsck.check_objects_id: Processed 13003855 objects so far 2025-09-07 13:42:02.701 1347769:MainThread s3ql.fsck.check_objects_phys_size: Checking objects (physical sizes)... 2025-09-07 13:42:03.605 1347769:MainThread s3ql.fsck.check_inode_blocks_obj_id: Checking block-object mapping... 2025-09-07 13:46:06.383 1347769:MainThread s3ql.fsck.check_inode_blocks_inode: Checking inode_blocks.inode... 2025-09-07 13:52:18.184 1347769:MainThread s3ql.fsck.check_inodes_refcount: Checking inodes (refcounts)... 2025-09-07 14:14:28.235 1347769:MainThread s3ql.fsck.check_inodes_size: Checking inodes (sizes)... 2025-09-07 19:48:56.131 1347769:MainThread s3ql.fsck.check_ext_attributes_name: Checking extended attributes (names)... 2025-09-07 19:48:56.161 1347769:MainThread s3ql.fsck.check_ext_attributes_inode: Checking extended attributes (inodes)... 2025-09-07 19:48:56.161 1347769:MainThread s3ql.fsck.check_symlinks_inode: Checking symlinks (inodes)... 2025-09-07 19:53:41.031 1347769:MainThread s3ql.fsck.check_loops: Checking directory reachability... 2025-09-07 21:29:46.610 1347769:MainThread s3ql.fsck.check_unix: Checking unix conventions... 2025-09-07 22:56:11.105 1347769:MainThread s3ql.fsck.check_foreign_keys: Checking referential integrity... 2025-09-07 23:22:00.714 1347769:MainThread s3ql.fsck.check: Dropping temporary indices... 2025-09-08 05:28:28.875 1347769:MainThread s3ql.database.upload_metadata: Uploading metadata... 2025-09-08 05:34:43.290 1347769:MainThread s3ql.database.upload_metadata: Uploaded 2459 out of ~1785367 dirty blocks (0%) 2025-09-11 05:45:36.418 1347769:MainThread s3ql.database.upload_metadata: Uploaded 1785367 out of ~1785367 dirty blocks (100%) 2025-09-11 05:45:36.518 1347769:MainThread s3ql.database.upload_metadata: Calculating metadata checksum... 2025-09-11 06:17:06.301 1347769:MainThread s3ql.fsck.main: Completed fsck of gs://bucketname.../ -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/s3ql/3f3641ae-27a0-4f2d-bdfa-30cef04ffa21n%40googlegroups.com.
