Den fre 10 aug. 2018 kl 04:33 skrev Matthew Pounsett <>:

> First, in my tests and reading I haven't encountered anything that
> suggests I should expect problems from using a small number of large file
> servers in a cluster.  But I recognize that this isn't the preferred
> configuration, and I'm wondering if I should be worried about any
> operational issues.  Obviously I/O won't be as good as it could be, but I
> don't expect it would be worse than software RAID served over NFS.  Is
> there anything there I've missed?  Eventually the plan will be to swap out
> the large file servers for a larger number of smaller servers, but that
> will take years of regular hardware refresh cycles.
Having few but large OSD hosts will mean that a loss of one (for planned or
unplanned maintenance) will have a high impact on the cluster as a whole.
The copies will by default try to place each replica on a separate host,
which is good, but it would also mean that loss of a host requires a lot of
data to
be shuffled and rebuilt from other replicas, and this will of course take

As opposed to the previous setup, this will add some inter-host traffic
aswell, each write to the primary PG will then in turn cause that host to
replicate it
again over the network to X other hosts to form the required amount of
replicas, so where raid/zfs/lvm did the copying internally on the same host
it will now go once or twice over the net. Not a huge problem, but worth
noticing. When repairs happen, like if a host dies, the amount of
traffic will be quite large, given the 80-130T described above.

I still think ceph can be a decent solution for your problem, but it would
be easier to make rolling maintenance on a cluster if the loss is smaller
when a
host is gone if the cluster is made up of many smaller hosts. So, while the
above situation is better than what you came from, it would not be an
ceph setup, but then again, who has an optimal setup anyhow. Everyone wants
to fix some part of the cluster always, if money and time was endless.

May the most significant bit of your life be positive.
ceph-users mailing list

Reply via email to