Hello, On Thu, 7 Jul 2016 23:19:35 +0200 Zoltan Arnold Nagy wrote:
> Hi Nick, > > How large NVMe drives are you running per 12 disks? > > In my current setup I have 4xP3700 per 36 disks but I feel like I could > get by with 2… Just looking for community experience :-) > This is funny, because you ask Nick about the size and don't mention it yourself. ^o^ As I speculated in my reply, it's the 400GB model and Nick didn't dispute that. And I shall assume the same for you. You could get by with 2 of the 400GB ones, but that depends on a number of things. 1. What's your use case, typical usage pattern? Are you doing a lot of large sequential writes or is it mostly smallish I/Os? HDD OSDs will clock in at about 100MB/s with OSD bench, but realistically not see more than 50-60MB/s, so with 18 of them per one 400GB P3700 you're about on par. 2. What's your network setup? If you have more than 20Gb/s to that node, your journals will likely become the (write) bottleneck. But that's only the case with backfills or again largish sequential writes of course. 3. A repeat of sorts of the previous 2 points, this time with the focus on endurance. How much data are you writing per day to an average OSD? With 18 OSDs per 400GB P3700 NVMe you will want that to be less than 223GB/day/OSD. 4. As usual, failure domains. In the case of a NVMe failure you'll loose twice the amount of OSDs. That all being said, at 36 OSDs I'd venture you'll run out of CPU steam (with small write IOPS) before your journals become the bottleneck. Christian > Cheers, > Zoltan > [snip] -- Christian Balzer Network/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
