We have 28 OSDs per host and we can only have 2 NVMe per host (one being used for OS) ... and if I remember correctly there is max 10 OSDs/NVMe recommended, that's why we decided to go just for HDD based clusters at the beinging.
We have 2 clusters this way, one being for HPC (no radosgw/s3) and other for "users" (radosgw/s3), running over 4 years now ... works ok, performance is ok, just we have this problem where we have to do gentle reweight of a failed OSDs. Thanks for the info, we will consider NVMe ... although then there is SPOF for those OSDs which have DB on NVMe? Rok On Tue, Feb 3, 2026 at 2:35 PM Robert Sander via ceph-users < [email protected]> wrote: > Am 03.02.26 um 2:31 PM schrieb Rok Jaklič: > > On Tue, Feb 3, 2026 at 2:26 PM Robert Sander via ceph-users <ceph- > > [email protected] <mailto:[email protected]>> wrote: > > > >> Am 03.02.26 um 2:18 PM schrieb Rok Jaklič via ceph-users: > >> > >> > 2 OSD(s) experiencing slow operations in BlueStore > >> > 2 OSD(s) experiencing stalled read in db device of > >> BlueFS > >> > >> Are your OSDs HDD only? > > > > > Yes. > > > > Does not affect users much. Usually those messages appear when we are > > reweighting and changing failed disks. > > These HDDs will be maxed out with the recovery work and cannot serve > anything else any more. > > I have seen HDD only clusters going into the "spiral of death" because > the HDDs cannot answer fast enough. OSDs randomly dropping out making > the whole system unstable. > > The RocksDB is such a random IO application that it is not suitable for > HDDs. It should always be put on flash storage (SSD/NVMe). > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
