On Wed, Jul 19, 2017 at 7:09 PM, Gregory Farnum <[email protected]> wrote:
> > > On Wed, Jul 19, 2017 at 10:25 AM David <[email protected]> wrote: > >> On Tue, Jul 18, 2017 at 6:54 AM, Blair Bethwaite < >> [email protected]> wrote: >> >>> We are a data-intensive university, with an increasingly large fleet >>> of scientific instruments capturing various types of data (mostly >>> imaging of one kind or another). That data typically needs to be >>> stored, protected, managed, shared, connected/moved to specialised >>> compute for analysis. Given the large variety of use-cases we are >>> being somewhat more circumspect it our CephFS adoption and really only >>> dipping toes in the water, ultimately hoping it will become a >>> long-term default NAS choice from Luminous onwards. >>> >>> On 18 July 2017 at 15:21, Brady Deetz <[email protected]> wrote: >>> > All of that said, you could also consider using rbd and zfs or >>> whatever filesystem you like. That would allow you to gain the benefits of >>> scaleout while still getting a feature rich fs. But, there are some down >>> sides to that architecture too. >>> >>> We do this today (KVMs with a couple of large RBDs attached via >>> librbd+QEMU/KVM), but the throughput able to be achieved this way is >>> nothing like native CephFS - adding more RBDs doesn't seem to help >>> increase overall throughput. Also, if you have NFS clients you will >>> absolutely need SSD ZIL. And of course you then have a single point of >>> failure and downtime for regular updates etc. >>> >>> In terms of small file performance I'm interested to hear about >>> experiences with in-line file storage on the MDS. >>> >>> Also, while we're talking about CephFS - what size metadata pools are >>> people seeing on their production systems with 10s-100s millions of >>> files? >>> >> >> On a system with 10.1 million files, metadata pool is 60MB >> >> > Unfortunately that's not really an accurate assessment, for good but > terrible reasons: > 1) CephFS metadata is principally stored via the omap interface (which is > designed for handling things like the directory storage CephFS needs) > 2) omap is implemented via Level/RocksDB > 3) there is not a good way to determine which pool is responsible for > which portion of RocksDBs data > 4) So the pool stats do not incorporate omap data usage at all in their > reports (it's part of the overall space used, and is one of the things that > can make that larger than the sum of the per-pool spaces) > > You could try and estimate it by looking at how much "lost" space there is > (and subtracting out journal sizes and things, depending on setup). But I > promise there's more than 60MB of CephFS metadata for 10.1 million files! > -Greg > That makes more sense, I did think I'd got my units mixed up or something. For a start my MDS daemon is using about 17GB with 1.5mil inodes in cache. Thanks for shedding some light on this. > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
