On Thu May 28 11:22:52 2015, Christian Balzer wrote: > > We are testing different scenarios before making our final decision > > (cache-tiering, journaling, separate pool,...). > > > Definitely a good idea to test things out and get an idea what Ceph and > your hardware can do. > > From my experience and reading this ML however I think your best bet > (overall performance) is to use those 4 SSDs a 1:5 journal SSDs for your > 20 OSDs HDDs. > > Currently cache-tiering is probably the worst use for those SSD resources, > though the code and strategy is of course improving. > I agree: in our particular enviroment, our tests also conclude that SSD journaling performs far better than cache-tiering, especially when cache becomes close to its capacity and data movement between cache and backing storage occurs frequently.
We also want to test if it is possible to use SSD disks as a "transparent" cache for the HDDs at system (Linux kernel) level, and how reliable/good is it. > Dedicated SSD pools may be a good fit depending on your use case. > However I'd advise against mixing SSD and HDD OSDs on the same node. > To fully utilize those SSDs you'll need a LOT more CPU power than required > by HDD OSDs or SSD journals/HDD OSDs systems. > And you already have 20 OSDs in that box. Good point! We did not consider that, thanks for pointing it out. > What CPUs do you have in those storage nodes anyway? > Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz, according to /proc/cpuinfo. We have only 1 CPU per osd node, so I'm afraid we have another potential bottleneck here. > If you have the budget, I'd deploy the current storage nodes in classic > (SSDs for journals) mode and add a small (2x 8-12 SSDs) pair of pure SSD > nodes, optimized for their task (more CPU power, faster network). > > Then use those SSD nodes to experiment with cache-tiers and pure SSD pools > and switch over things when you're comfortable with this and happy with the > performance. > > > > > > However with 20 OSDs per node, you're likely to go from a being > > > bottlenecked by your HDDs to being CPU limited (when dealing with lots > > > of small IOPS at least). > > > Still, better than now for sure. > > > > > This is very interesting, thanks for pointing it out! > > What would you suggest to use in order to identify the actual > > bottleneck? (disk, CPU, RAM, etc.). Tools like munin? > > > Munin might work, I use collectd to gather all those values (and even more > importantly all Ceph counters) and graphite to visualize it. > For ad-hoc, on the spot analysis I really like atop (in a huge window), > which will make it very clear what is going on. > > > In addition, there are some kernel tunables that may be helpful > > to improve overall performance. Maybe we are filling some kernel > > internals and that limits our results (for instance, we had to increase > > fs.aio-max-nr in sysctl.d to 262144 to be able to use 20 disks per > > host). Which tunables should we observe? > > > I'm no expert for large (not even medium) clusters, so you'll have to > research the archives and net (the CERN Ceph slide is nice). > One thing I remember is "kernel.pid_max", which is something you're likely > to run into at some point with your dense storage nodes: > http://ceph.com/docs/master/start/hardware-recommendations/#additional-considerations > > Christian All you say is really interesting. Thanks for your valuable advice. We surely still have plenty of things to learn and test before going to production. Thanks again for your time and help. Best regards, - Xavier Serrano - LCAC, Laboratori de Càlcul - Departament d'Arquitectura de Computadors, UPC _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
