That's right. I didn't actually use Jewel for very long. I'm glad it worked for you.
On Fri, May 11, 2018, 4:49 PM Webert de Souza Lima <webert.b...@gmail.com> wrote: > Thanks David. > Although you mentioned this was introduced with Luminous, it's working > with Jewel. > > ~# ceph osd pool stats > > Fri May 11 17:41:39 2018 > > pool rbd id 5 > client io 505 kB/s rd, 3801 kB/s wr, 46 op/s rd, 27 op/s wr > > pool rbd_cache id 6 > client io 2538 kB/s rd, 3070 kB/s wr, 601 op/s rd, 758 op/s wr > cache tier io 12225 kB/s flush, 0 op/s promote, 3 PG(s) flushing > > pool cephfs_metadata id 7 > client io 2233 kB/s rd, 2260 kB/s wr, 95 op/s rd, 587 op/s wr > > pool cephfs_data_ssd id 8 > client io 1126 kB/s rd, 94897 B/s wr, 33 op/s rd, 42 op/s wr > > pool cephfs_data id 9 > client io 0 B/s rd, 11203 kB/s wr, 12 op/s rd, 12 op/s wr > > pool cephfs_data_cache id 10 > client io 4383 kB/s rd, 550 kB/s wr, 57 op/s rd, 39 op/s wr > cache tier io 7012 kB/s flush, 4399 kB/s evict, 11 op/s promote > > > Regards, > > Webert Lima > DevOps Engineer at MAV Tecnologia > *Belo Horizonte - Brasil* > *IRC NICK - WebertRLZ* > > > On Fri, May 11, 2018 at 5:14 PM David Turner <drakonst...@gmail.com> > wrote: > >> `ceph osd pool stats` with the option to specify the pool you are >> interested in should get you the breakdown of IO per pool. This was >> introduced with luminous. >> >> On Fri, May 11, 2018 at 2:39 PM Webert de Souza Lima < >> webert.b...@gmail.com> wrote: >> >>> I think ceph doesn't have IO metrics will filters by pool right? I see >>> IO metrics from clients only: >>> >>> ceph_client_io_ops >>> ceph_client_io_read_bytes >>> ceph_client_io_read_ops >>> ceph_client_io_write_bytes >>> ceph_client_io_write_ops >>> >>> and pool "byte" metrics, but not "io": >>> >>> ceph_pool(write/read)_bytes(_total) >>> >>> Regards, >>> >>> Webert Lima >>> DevOps Engineer at MAV Tecnologia >>> *Belo Horizonte - Brasil* >>> *IRC NICK - WebertRLZ* >>> >>> On Wed, May 9, 2018 at 2:23 PM Webert de Souza Lima < >>> webert.b...@gmail.com> wrote: >>> >>>> Hey Jon! >>>> >>>> On Wed, May 9, 2018 at 12:11 PM, John Spray <jsp...@redhat.com> wrote: >>>> >>>>> It depends on the metadata intensity of your workload. It might be >>>>> quite interesting to gather some drive stats on how many IOPS are >>>>> currently hitting your metadata pool over a week of normal activity. >>>>> >>>> >>>> Any ceph built-in tool for this? maybe ceph daemonperf (altoght I'm not >>>> sure what I should be looking at). >>>> My current SSD disks have 2 partitions. >>>> - One is used for cephfs cache tier pool, >>>> - The other is used for both: cephfs meta-data pool and cephfs >>>> data-ssd (this is an additional cephfs data pool with only ssds with file >>>> layout for a specific direcotory to use it) >>>> >>>> Because of this, iostat shows me peaks of 12k IOPS in the metadata >>>> partition, but this could definitely be IO for the data-ssd pool. >>>> >>>> >>>>> If you are doing large file workloads, and the metadata mostly fits in >>>>> RAM, then the number of IOPS from the MDS can be very, very low. On >>>>> the other hand, if you're doing random metadata reads from a small >>>>> file workload where the metadata does not fit in RAM, almost every >>>>> client read could generate a read operation, and each MDS could easily >>>>> generate thousands of ops per second. >>>>> >>>> >>>> I have yet to measure it the right way but I'd assume my metadata fits >>>> in RAM (a few 100s of MB only). >>>> >>>> This is an email hosting cluster with dozens of thousands of users so >>>> there are a lot of random reads and writes, but not too many small files. >>>> Email messages are concatenated together in files up to 4MB in size >>>> (when a rotation happens). >>>> Most user operations are dovecot's INDEX operations and I will keep >>>> index directory in a SSD-dedicaded pool. >>>> >>>> >>>> >>>>> Isolating metadata OSDs is useful if the data OSDs are going to be >>>>> completely saturated: metadata performance will be protected even if >>>>> clients are hitting the data OSDs hard. >>>>> >>>> >>>> This seems to be the case. >>>> >>>> >>>>> If "heavy write" means completely saturating the cluster, then sharing >>>>> the OSDs is risky. If "heavy write" just means that there are more >>>>> writes than reads, then it may be fine if the metadata workload is not >>>>> heavy enough to make good use of SSDs. >>>>> >>>> >>>> Saturarion will only happen in peak workloads, not often. By heavy >>>> write I mean there are much more writes than reads, yes. >>>> So I think I can start sharing the OSDs, if I think this is impacting >>>> performance I can just change the ruleset and move metadata to a SSD-only >>>> pool, right? >>>> >>>> >>>>> The way I'd summarise this is: in the general case, dedicated SSDs are >>>>> the safe way to go -- they're intrinsically better suited to metadata. >>>>> However, in some quite common special cases, the overall number of >>>>> metadata ops is so low that the device doesn't matter. >>>> >>>> >>>> >>>> Thank you very much John! >>>> Webert Lima >>>> DevOps Engineer at MAV Tecnologia >>>> Belo Horizonte - Brasil >>>> IRC NICK - WebertRLZ >>>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
