I use nmon on each OSD server, this is a really good tool to find out
what is going on regarding CPU, Mem, Disks and Networking
German Anders
--- Original message ---
Asunto: Re: [ceph-users] Performance really drops from 700MB/s to
10MB/s
De: Craig Lewis <[email protected]>
Para: Mariusz Gronczewski <[email protected]>
Cc: German Anders <[email protected]>, Ceph Users
<[email protected]>
Fecha: Thursday, 14/08/2014 15:42
I find graphs really help here. One screen that has all the disk I/O
and latency for all OSDs makes it easy to pin point the bottleneck.
If you don't have that, I'd go low tech: Watch the blinky lights. It's
really easy to see which disk is the hotspot.
On Thu, Aug 14, 2014 at 6:56 AM, Mariusz Gronczewski
<[email protected]> wrote:
Actual OSD (/var/log/ceph/ceph-osd.$id) logs would be more useful.
Few ideas:
* do 'ceph health detail' to get detail of which OSD is stalling
* 'ceph osd perf' to see latency of each osd
* 'ceph --admin-daemon /var/run/ceph/ceph-osd.$id.asok
dump_historic_ops' shows "recent slow" ops
I actually have very similiar problem, cluster goes full speed
(sometimes even for hours) and suddenly everything stops for a minute
or 5, no disk IO, no IO wait (so disks are fine), no IO errors in
kernel log, and OSDs only complain that other OSD subop is slow (but
on that OSD everything looks fine too)
On Wed, 13 Aug 2014 16:04:30 -0400, German Anders
<[email protected]> wrote:
Also, even a "ls -ltr" could be done inside the /mnt of the RBD that
it freeze the prompt. Any ideas? I've attach some syslogs from one of
the OSD servers and also from the client. Both are running Ubuntu
14.04LTS with Kernel 3.15.8.
The cluster is not usable at this point, since I can't run a "ls" on
the rbd.
Thanks in advance,
Best regards,
German Anders
--- Original message ---
Asunto: Re: [ceph-users] Performance really drops from 700MB/s to
10MB/s
De: German Anders <[email protected]>
Para: Mark Nelson <[email protected]>
Cc: <[email protected]>
Fecha: Wednesday, 13/08/2014 11:09
Actually is very strange, since if i run the fio test on the client,
and also un parallel run a iostat on all the OSD servers, i don't see
any workload going on over the disks, I mean... nothing! 0.00....and
also the fio script on the client is reacting very rare too:
$ sudo fio --filename=/dev/rbd1 --direct=1 --rw=write --bs=4m
--size=10G --iodepth=16 --ioengine=libaio --runtime=60
--group_reporting --name=file99
file99: (g=0): rw=write, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio,
iodepth=16
fio-2.1.3
Starting 1 process
Jobs: 1 (f=1): [W] [2.1% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta
01h:26m:43s]
It's seems like is doing nothing..
German Anders
--- Original message ---
Asunto: Re: [ceph-users] Performance really drops from 700MB/s to
10MB/s
De: Mark Nelson <[email protected]>
Para: <[email protected]>
Fecha: Wednesday, 13/08/2014 11:00
On 08/13/2014 08:19 AM, German Anders wrote:
Hi to all,
I'm having a particular behavior on a
new Ceph cluster.
I've map
a RBD to a client and issue some performance tests with fio, at this
point everything goes just fine (also the results :) ), but then I try
to run another new test on a new RBD on the same client, and suddenly
the performance goes below 10MB/s and it took almost 10 minutes to
complete a 10G file test, if I issue a *ceph -w* I don't see anything
suspicious, any idea what can be happening here?
When things are going fast, are your disks actually writing data out
as
fast as your client IO would indicate? (don't forgot to count
replication!) It may be that the great speed is just writing data
into
the tmpfs journals (if the test is only 10GB and spread across 36
OSDs,
it could finish pretty quickly writing to tmpfs!). FWIW, tmpfs
journals
aren't very safe. It's not something you want to use outside of
testing
except in unusual circumstances.
In your tests, when things are bad: it's generally worth checking to
see
if any one disk/osd is backed up relative to the others. There are a
couple of ways to accomplish this. the Ceph admin socket can tell you
information about each OSD ie how many outstanding IOs and a history
of
slow ops. You can also look at per-disk statistics with something
like
iostat or collectl.
Hope this helps!
The cluster is made of:
3 x MON Servers
4 x OSD Servers (3TB SAS 6G disks for OSD daemons & tmpfs for Journal
->
there's one tmpfs of 36GB that is share by 9 OSD daemons, on each
server)
2 x Network SW (Cluster and Public)
10GbE speed on both networks
The ceph.conf file is the following:
[global]
fsid = 56e56e4c-ea59-4157-8b98-acae109bebe1
mon_initial_members = cephmon01, cephmon02, cephmon03
mon_host = 10.97.10.1,10.97.10.2,10.97.10.3
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
filestore_xattr_use_omap = true
public_network = 10.97.0.0/16
cluster_network = 192.168.10.0/24
osd_pool_default_size = 2
glance_api_version = 2
[mon]
debug_optracker = 0
[mon.cephmon01]
host = cephmon01
mon_addr = 10.97.10.1:6789
[mon.cephmon02]
host = cephmon02
mon_addr = 10.97.10.2:6789
[mon.cephmon03]
host = cephmon03
mon_addr = 10.97.10.3:6789
[osd]
journal_dio = false
osd_journal_size = 4096
fstype = btrfs
debug_optracker = 0
[osd.0]
host = cephosd01
devs = /dev/sdc1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.1]
host = cephosd01
devs = /dev/sdd1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.2]
host = cephosd01
devs = /dev/sdf1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.3]
host = cephosd01
devs = /dev/sdg1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.4]
host = cephosd01
devs = /dev/sdi1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.5]
host = cephosd01
devs = /dev/sdj1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.6]
host = cephosd01
devs = /dev/sdl1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.7]
host = cephosd01
devs = /dev/sdm1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.8]
host = cephosd01
devs = /dev/sdn1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.9]
host = cephosd02
devs = /dev/sdc1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.10]
host = cephosd02
devs = /dev/sdd1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.11]
host = cephosd02
devs = /dev/sdf1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.12]
host = cephosd02
devs = /dev/sdg1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.13]
host = cephosd02
devs = /dev/sdi1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.14]
host = cephosd02
devs = /dev/sdj1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.15]
host = cephosd02
devs = /dev/sdl1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.16]
host = cephosd02
devs = /dev/sdm1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.17]
host = cephosd02
devs = /dev/sdn1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.18]
host = cephosd03
devs = /dev/sdc1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.19]
host = cephosd03
devs = /dev/sdd1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.20]
host = cephosd03
devs = /dev/sdf1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.21]
host = cephosd03
devs = /dev/sdg1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.22]
host = cephosd03
devs = /dev/sdi1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.23]
host = cephosd03
devs = /dev/sdj1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.24]
host = cephosd03
devs = /dev/sdl1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.25]
host = cephosd03
devs = /dev/sdm1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.26]
host = cephosd03
devs = /dev/sdn1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.27]
host = cephosd04
devs = /dev/sdc1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.28]
host = cephosd04
devs = /dev/sdd1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.29]
host = cephosd04
devs = /dev/sdf1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.30]
host = cephosd04
devs = /dev/sdg1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.31]
host = cephosd04
devs = /dev/sdi1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.32]
host = cephosd04
devs = /dev/sdj1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.33]
host = cephosd04
devs = /dev/sdl1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.34]
host = cephosd04
devs = /dev/sdm1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[osd.35]
host = cephosd04
devs = /dev/sdn1
osd_journal = /mnt/ramdisk/$cluster-$id-journal
[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring
Thanks in advance,
Best regards,
*German Anders
*
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Mariusz Gronczewski, Administrator
Efigence S. A.
ul. WoĊoska 9a, 02-583 Warszawa
T: [+48] 22 380 13 13
F: [+48] 22 380 13 14
E: [email protected]
<mailto:[email protected]>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
