Am 30.08.2013 20:33, schrieb Wido den Hollander:
On 08/30/2013 08:19 PM, Geraint Jones wrote:
Hi Guys
We are using Ceph in production backing an LXC cluster. The setup is : 2
x Servers, 24 x 3TB Disks each in groups of 3 as RAID0. SSD for
journals. Bonded 1gbit ethernet (2gbit total).
I think you sized your machines too big. I'd say go for 6 machines with 8
disks each without RAID-0. Let Ceph do it's job and avoid RAID.
I think that would be ok if ceph wouldn't consume that much memory. On a
relatively small cluster with currently 16 OSDs and 8192 PGs (2 pools with 4096
pgs each) each osd consumes around 2-4 GB ram. So with 24 disks the machine
would need at least 96 GB ram, just for the osds. With some memory left for the
page cache etc. 128 GB would be the minimum.
Anyway, I think this is a bug and already filed a report for it here:
http://tracker.ceph.com/issues/5700 To be honest I'm a little disappointed by
the last answer there, because 8192 isn't a lot of pgs? The docs say 50-100 pgs
per osds per pool. So 2 pools with 4096 pgs each just allows for 40 - 80 osds -
which isn't really that much? There's also a big discrepancy with the docs, as
they state an osd would consume 200 - 500 mb (
http://ceph.com/docs/next/install/hardware-recommendations/). I know David is
still waiting for a detailed debug log from me, which I'll provide within a
short while. But if ceph's memory requirements are really that high, fixing the
docs to allow for proper resource planing is really a must.
I'm curious what memory usage other users are experiencing?
Cheers,
Corin
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
