On 08/08/2014 03:44 PM, Dan Van Der Ster wrote:
Hi,
Here’s what we do to identify our top RBD users.
First, enable log level 10 for the filestore so you can see all the IOs
coming from the VMs. Then use a script like this (used on a dumpling
cluster):
https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl
to summarize the osd logs and identify the top clients.
Then its just a matter of scripting to figure out the ops/sec per
volume, but for us at least the main use-case has been to identify who
is responsible for a new peak in overall ops — and daily-granular
statistics from the above script tends to suffice.
BTW, do you throttle your clients? We found that its absolutely
necessary, since without a throttle just a few active VMs can eat up the
entire iops capacity of the cluster.
+1
I'd strongly advise to set I/O limits for Instances. I've had multiple
occasions where a runaway script inside a VM was hammering on the
underlying storage killing all I/O.
Not only with Ceph, but over the many years I've worked with storage.
I/O == expensive
CloudStack supports I/O limiting, so I recommend you set a limit. Set it
to 750 write IOps for example. That way one Instance can't kill the
whole cluster, but it still has enough I/O to run. (usually).
Wido
Cheers, Dan
-- Dan van der Ster || Data & Storage Services || CERN IT Department --
On 08 Aug 2014, at 13:51, Andrija Panic <andrija.pa...@gmail.com
<mailto:andrija.pa...@gmail.com>> wrote:
Hi,
we just had some new clients, and have suffered very big degradation
in CEPH performance for some reasons (we are using CloudStack).
I'm wondering if there is way to monitor OP/s or similar usage by
client connected, so we can isolate the heavy client ?
Also, what is the general best practice to monitor these kind of
changes in CEPH ? I'm talking about R/W or OP/s change or similar...
Thanks,
--
Andrija Panić
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Wido den Hollander
42on B.V.
Ceph trainer and consultant
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
