I would suggest 2GB partitions for WAL partitions and 150GB osds to make an SSD only pool for the fs metadata pool. I know that doesn't use the whole disk, but there's no need or reason to. By under-provisioning the nvme it just adds that much more longevity to the life of the drive.
You cannot change the size of any part of a bluestore osd after creation. On Sat, May 12, 2018, 3:09 PM Oliver Schulz <[email protected]> wrote: > Dear David, > > On 11.05.2018 22:10, David Turner wrote: > > For if you should do WAL only on the NVMe vs use a filestore journal, > > that depends on your write patterns, use case, etc. > > we mostly use CephFS, for scientific data processing. It's > mainly larger files (10 MB to 10 GB, but sometimes also > a bunch of small files), typically written once, and read > several times. We also keep Singularity software container > images on CephFS, here the read patterns are more scattered. > > We currently have about 1PB raw capacity on ca. 150 OSDs, and > I'll add another 45 OSDs with 10 TB disks now. The old > OSDs are all filestore, I'm planning to to switch them > over to bluestore bit by bit once the new OSDs are online. > > > > that ceph will prioritize things such that the WAL won't spill over at > > all and just have the DB going over to the HDD. I didn't want to deal > > with speed differentials between OSDs. > > So would it make sense to just assign 15GB SSD partition > for WAL+DB per 10TB OSD (more I don't have available), > and let the DB spill over? > > Or do you think it would make for more predictable/uniform > OSD performance to a few TB WAL and always keep the DB on > the HDDs (like in your cluster), for our use case? > > Or should I try to use a 2 TB WAL and a 13 TB DB partition > per 10 TB OSD - maybe 13 TB for DB is just way too small to > give any benefit , here? > > One important question - is it possible to change WAL and > DB later on without deleting and re-creating the OSD(s)? > > > > The troubleshooting slow requests of that just sounds awful. > > So true! > > > Thanks again for all the advice, > > Oliver >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
