On Tue, Dec 23, 2014 at 3:54 PM, Stefan G. Weichinger <li...@xunil.at> wrote: > Am 23.12.2014 um 21:40 schrieb Rich Freeman: >> On Tue, Dec 23, 2014 at 3:27 PM, Stefan G. Weichinger <li...@xunil.at> wrote: >>> >>> got my first two demo nodes up and in-sync ... what a success ;-) >> >> I started to look into ceph, and my biggest issue is that they don't >> protect against silent corruption. They do checksum data during >> transit, but not at rest. That means that you could end up with 3 >> different copies of a file and no way to know which one is the right >> one. Simply storing the data on btrfs isn't enough - that will >> protect against files changing on the disk itself, but you could STILL >> end up with 3 different copies of a file on different nodes and no way >> to know which one is right, if the error happens at a higher level >> than the btrfs filesystem/disk. > > but ... oh my. *sigh* > > I assume the devs there have a clever answer to this as well? > > At least for the future ... now that btrfs is declared stable at least > for the more trivial setups (read: not RAID5/6) by Chris Mason himself > ... btrfs should be usable for ceph-OSDs soon.
Proclamations of stability do not stable make. :) I'm using btrfs now, but I've had my share of headaches (especially with the 3.15/16 kernels). I think the rate of change/regressions is a good long-term sign, but I'd stick to longterm kernels if you use it (which is different advice than I used to give). > > In the other direction: what protects against these errors you mention? > If I had a solution I'd be using it. I don't use ceph. Btrfs protects against them just fine for a single system. The problem with ceph is cross-node consistency. -- Rich
