Hi,
I confirm you can setup tiering between two differents S3 pools on the
same ceph cluster with squid.
You have to create 2 differents pools and placement groups relying on
your two type of physical storage drives. You may use the same RGW
client and use the S3 storage class attribute or you may also create a
new RGW client dedicated to each pool.
I use S3 bucket policy for this and you should prefer using standard AWS
names of storage class ("GLACIER" or "STANDARD_IA"...) to do this easily
using the aws cli.
On 4/29/26 14:18, Anthony D'Atri via ceph-users wrote:
Is it possible to use RGW lifecycle policies to automatically move objects
between different storage tiers for example, from a hot tier (NVMe) to a
cold tier (HDD) based on inactivity over a period of time?
Absolutely. This is one key application of S3 storage classes. Note that LC
policies are run once a day by default, so with a heavy load you may wish to
deploy a set of RGWs that are not user-facing, but which only handle LC tasks,
and disable LC processing for the user-facing cohort of daemons.
Additionally, if those objects become active again (i.e., are accessed
frequently after being moved to the cold tier), is there a mechanism to
automatically move them back to the original hot tier (NVMe)?
But of course! Transparent tiering is available from Squid, at least when
offloading cold objects to a different endpoint. I'm not positive that this
works natively within a single cluster, but it may well. You could though most
likely still use LC policies, albeit in a less elegant fashion.
I may be misunderstanding how lifecycle management works in RGW, so please
correct me if that’s the case. If this type of tiering is supported, could
you point me toward the recommended approach or configuration?
The following awesome blog posts from Daniel Parkes will guide you. When
deploying a cold tier, it is ideal to leave the first STANDARD storage class as
the default so that HEAD objects are always on faster storage. They don't take
much space, but this accelerates various operations.
https://ceph.io/en/news/blog/2025/rgw-tiering-enhancements-part1/
Depending on your object sizes and workload, consider replication vs erasure coding.
When using EC for your slower -- or even faster -- storage class, be sure to select a
value of m >= 2. Profiles wider than 8+3 may increase space amp and cost write,
scrub, and recovery performance. Always choose (k+m < the number of CRUSH failure
domains.
Writes to a replicated pool will generally be faster, but EC makes more
efficient use of underlying storage.
This is nuanced, though for the smallest objects. Check out Mark Nelson's most
excellent spreadsheet that shows this visually:
https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToPQjN8bDI/edit?gid=358760253#gid=358760253
Through the Squid release, RADOS always writes full stripes, so if you are
using EC, the very smallest objects will consume more underlying raw storage
than one might think. The wider the EC profile, the more pronounced this
effect. If you have a large proportion of tiny objects, say below 256 - 512
KiB, this space amplification may be significant. If the number of very small
objects is dwarfed by the number * size of larger objects, this is amortized
into the noise floor. If not, you can consider client headers or Lua scripting
to assort objects to appropriate storage classes at ingest. Check out these
presentations by Frederic Nass and Steve U:
https://ceph.io/en/news/blog/2024/auto-tiering-ceph-object-storage-part-1/
https://www.youtube.com/watch?v=qKd2Po2hTe4
Note that with the Tentacle release, bucket data pools with the EC
optimizations flag significantly reduce this space amplification, lowering the
size threshold for special handling and reducing the need to worry about it.
https://ceph.io/en/news/blog/2025/tentacle-fastec-performance-updates/
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
