Hi all, Thanks for the feedback. Though, just to be sure:
1. There is no 30GB limit if I understand correctly for the RocksDB size. If metadata crosses that barrier, the L4 part will spillover to the primary device? Or will it just move the RocksDB completely? Or will it just stop and indicate it's full? 2. Since the WAL will also be written to that device, I assume a few additional GB's is still usefull... With my setup (13x 14TB + 2 NVMe of 1.6TB / host, 10 hosts) I have multiple possible scenario's: - Assigning 35GB of space of the NVMe disk (30GB for DB, 5 spare) would result in only 455GB being used (13 x 35GB). This is a pity, since I have 3.2TB of NVMe disk space... Options line-up: *Option a*: Not using the NVMe for block.db storage, but as RGW metadata pool. Advantages: - Impact of 1 defect NVMe is limited. - Fast storage for the metadata pool. Disadvantage: - RocksDB for each OSD is on the primary disk, resulting in slower performance of each OSD. *Option b: *Hardware mirror of the NVMe drive Advantages: - Impact of 1 defect NVMe is limited - Fast KV lookup for each OSD Disadvantage: - I/O to NVMe is serialized for all OSDs on 1 host. Though the NVMe are fast, I imagine that there still is an impact. - 1 TB of NVMe is not used / host *Option c: *Split the NVMe's accross the OSD Advantages: - Fast RockDB access - up to L3 (assuming spillover does it job) Disadvantage: - 1 defect NVMe impacts max 7 OSDs (1 NVMe assigned to 7 or 6 OSD daemons per host) - 2.7TB of NVMe space not used per host *Option d: *1 NVMe disk for OSDs, 1 for RGW metadata pool Advantages: - Fast RockDB access - up to L3 - Fast RGW metadata pool (though limited to 5,3TB (raw pool size will be 16TB, divided by 3 due to replication) I assume this already gives some possibilities Disadvantages: - 1 defect NVMe might impact a complete host (all OSDs might be using it for the RockDB storage) - 1 TB of NVMe is not used Though menu to choose from, each with it possibilities... The initial idea was too assign 200GB per OSD of the NVMe space per OSD, but this would result in a lot of unused space. I don't know if there is anything on the roadmap to adapt the RocksDB sizing to make better use of the available NVMe disk space. With all the information, I would assume that the best option would be *option A*. Since we will be using erasure coding for the RGW data pool (k=6, m=3), the impact of a defect NVMe would be too significant. The other alternative would be option b, but then again we would be dealing with HW raid which is against all Ceph design rules. Any other options or (dis)advantages I missed? Or any other opinions to choose another option? Regards, Kristof Op vr 15 nov. 2019 om 18:22 schreef <vita...@yourcmc.ru>: > Use 30 GB for all OSDs. Other values are pointless, because > https://yourcmc.ru/wiki/Ceph_performance#About_block.db_sizing > > You can use the rest of free NVMe space for bcache - it's much better > than just allocating it for block.db. >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
