Installed S3QL version 2.28. I use an S3 backend for storing data, with hourly backups. This has worked well for several years.
On June 24 this year I started creating snapshots, and the S3 costs seem to have linearly increased since then (see chart below), with the bulk of the new costs being APS2-EarlyDelete-SIA and APS2-Retrieval-SIA. Since the data is relatively static (very little deleted, slow additions), my naive expectation was that with de-duping etc, there would be little or no impact. Yet I am seeing quite remarkable growth. I should note that the total storage in that time has gone from 19GB to 31GB, so not a huge increase. The basic hourly process is: - mount the S3 file system - rsync -avr /my/local/data /my/s3/mountpoint - unmount (this syncs local data with the 'data' dir in the S3 mount point). The basic daily operation is: - mount the S3 file system - s3qlcp /my/s3/mountpoint/data /my/s3/mountpoint/snapshots/<daily-dir> - rsync -avr --delete /my/local/data /my/s3/mountpoint - unmount The additions in this process are the daily rsync with "--delete" and the s3qlcp; I do not expect the --delete is deleting many files based on log files of the operation. Have I been hopelessly naive here? Or is there something I am missing in how this should work? What I had expected is that the snapshots would add little or no overhead. [image: S3 costs.PNG] I have now disabled the daily snapshots and will see if the costs reduce, but any help or insights would be appreciated. -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
