On Tue, 16 Sep 2025, at 01:29, '[email protected]' via s3ql wrote: > > On Monday, September 15, 2025 at 12:49:16 PM UTC-4 Nikolaus Rath wrote: >> >> On Mon, 15 Sep 2025, at 15:53, '[email protected]' via s3ql wrote: >>> On Monday, September 15, 2025 at 4:05:32 AM UTC-4 Nikolaus Rath wrote: >>>> On Mon, 15 Sep 2025, at 02:36, '[email protected]' via s3ql wrote: >>>>> > Running fsck.s3ql --force on this filesystem takes about 4 days, is >>>>> > there >>>>>> > a way to avoid doing this? >>>>>> >>>>>> Which steps take the longest? Depending on that, --fast may or may not >>>>>> help. >>> >>> The current fsck seems to be taking about the same time as the previous >>> one, the last one was using s3ql 5.3 with the --fast option, and the >>> longest steps were: >>> >>> Checking DB integrity, approx 1 day >>> >> >> Nothing that can be done here, unfortunately. This is just SQLite doing its >> thing. Only thing that might help is a faster CPU or putting the DB on a >> faster disk. >> >>> Uploading metadata, approx 3 days >> >> That's only after an unclean unmount, right? It should be much faster if you >> run fsck.s3ql --force on a clean filesystem? >> >> The problem is that fsck.s3ql needs to re-upload the entire database (109 GB >> in your case), because the information about which parts were modified was >> lost when mount.s3ql crashed. >> >> Is the upload time (3 days for 109 GB, maybe 50 GB after compression) >> consistent with your bandwidth? >> >> If bandwidth is the limiting factor, then in theory this could be sped up by >> calculating a checksum of each DB block, comparing it with the checksum of >> the block stored in remote storage, and only re-uploading if the checksum >> changed. That'd require someone to write the code, of course :-). >> > > Bandwidth does not seem to be the limiting factor, I'm getting roughly > 100KB/sec upload to google storage, but I get 5MB/sec or more copying data > to/from other machines. In fact, nothing on the machine seems maxed out, > top says cpu usage less than 20% and memory usage less than 2%, iotop does > not see that much disk usage, so not sure where the bottleneck is...
If you truly uploaded at 100 KB/sec, then it would take you about (50*1024^3) / (100*1024) / (60*60) = 145 hours to upload 50 GB. So either compression is a lot better than I'd expect, or you must be uploading more quickly than that. What's your metadata block size (specified on mkfs.s3ql time)? I suspect it's relatively small, and since metadata is uploaded sequentially, you're limited by network latency rather than bandwidth. This will be addressed by https://github.com/s3ql/s3ql/issues/310, which I plan to work on if ever I find some free time... Best, -Nikolaus -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/s3ql/7547019d-683b-4281-840e-b83d984687b5%40app.fastmail.com.
