Re: [ceph-users] Performance really drops from 700MB/s to 10MB/s

[German Anders]

Also, even a "ls -ltr" could be done inside the /mnt of the RBD that it freeze the prompt. Any ideas? I've attach some syslogs from one of the OSD servers and also from the client. Both are running Ubuntu 14.04LTS with Kernel 3.15.8. The cluster is not usable at this point, since I can't run a "ls" on the rbd.

Thanks in advance,

Best regards,


German Anders

--- Original message ---

De: German Anders <gand...@despegar.com>
Para: Mark Nelson <mark.nel...@inktank.com>
Cc: <ceph-users@lists.ceph.com>
Fecha: Wednesday, 13/08/2014 11:09






fio-2.1.3
Starting 1 process


It's seems like is doing nothing..



German Anders

Asunto: Re: [ceph-users] Performance really drops from 700MB/s to 10MB/s Actually is very strange, since if i run the fio test on the client, and also un parallel run a iostat on all the OSD servers, i don't see any workload going on over the disks, I mean... nothing! 0.00....and also the fio script on the client is reacting very rare too: $ sudo fio --filename=/dev/rbd1 --direct=1 --rw=write --bs=4m --size=10G --iodepth=16 --ioengine=libaio --runtime=60 --group_reporting --name=file99 file99: (g=0): rw=write, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=16 Jobs: 1 (f=1): [W] [2.1% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01h:26m:43s]

--- Original message ---

De: Mark Nelson <mark.nel...@inktank.com>
Para: <ceph-users@lists.ceph.com>
Fecha: Wednesday, 13/08/2014 11:00

On 08/13/2014 08:19 AM, German Anders wrote:

Asunto: Re: [ceph-users] Performance really drops from 700MB/s to 10MB/s

Hi to all,


a RBD to a client and issue some performance tests with fio, at this
point everything goes just fine (also the results :) ), but then I try
to run another new test on a new RBD on the same client, and suddenly
the performance goes below 10MB/s and it took almost 10 minutes to
complete a 10G file test, if I issue a *ceph -w* I don't see anything
suspicious, any idea what can be happening here?

I'm having a particular behavior on a new Ceph cluster. I've map

fast as your client IO would indicate? (don't forgot to count

except in unusual circumstances.


if any one disk/osd is backed up relative to the others.  There are a
couple of ways to accomplish this.  the Ceph admin socket can tell you

iostat or collectl.

Hope this helps!

When things are going fast, are your disks actually writing data out as replication!) It may be that the great speed is just writing data into the tmpfs journals (if the test is only 10GB and spread across 36 OSDs, it could finish pretty quickly writing to tmpfs!). FWIW, tmpfs journals aren't very safe. It's not something you want to use outside of testing In your tests, when things are bad: it's generally worth checking to see information about each OSD ie how many outstanding IOs and a history of slow ops. You can also look at per-disk statistics with something like

               The cluster is made of:

3 x MON Servers

2 x Network SW (Cluster and Public)
10GbE speed on both networks

               The ceph.conf file is the following:

[global]
fsid = 56e56e4c-ea59-4157-8b98-acae109bebe1
mon_initial_members = cephmon01, cephmon02, cephmon03
mon_host = 10.97.10.1,10.97.10.2,10.97.10.3
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
filestore_xattr_use_omap = true
public_network = 10.97.0.0/16
cluster_network = 192.168.10.0/24
osd_pool_default_size = 2
glance_api_version = 2

[mon]
debug_optracker = 0

[mon.cephmon01]
host = cephmon01
mon_addr = 10.97.10.1:6789

[mon.cephmon02]
host = cephmon02
mon_addr = 10.97.10.2:6789

[mon.cephmon03]
host = cephmon03
mon_addr = 10.97.10.3:6789

[osd]
journal_dio = false
osd_journal_size = 4096
fstype = btrfs
debug_optracker = 0

[osd.0]
host = cephosd01
devs = /dev/sdc1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.1]
host = cephosd01
devs = /dev/sdd1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.2]
host = cephosd01
devs = /dev/sdf1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.3]
host = cephosd01
devs = /dev/sdg1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.4]
host = cephosd01
devs = /dev/sdi1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.5]
host = cephosd01
devs = /dev/sdj1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.6]
host = cephosd01
devs = /dev/sdl1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.7]
host = cephosd01
devs = /dev/sdm1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.8]
host = cephosd01
devs = /dev/sdn1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.9]
host = cephosd02
devs = /dev/sdc1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.10]
host = cephosd02
devs = /dev/sdd1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.11]
host = cephosd02
devs = /dev/sdf1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.12]
host = cephosd02
devs = /dev/sdg1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.13]
host = cephosd02
devs = /dev/sdi1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.14]
host = cephosd02
devs = /dev/sdj1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.15]
host = cephosd02
devs = /dev/sdl1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.16]
host = cephosd02
devs = /dev/sdm1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.17]
host = cephosd02
devs = /dev/sdn1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.18]
host = cephosd03
devs = /dev/sdc1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.19]
host = cephosd03
devs = /dev/sdd1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.20]
host = cephosd03
devs = /dev/sdf1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.21]
host = cephosd03
devs = /dev/sdg1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.22]
host = cephosd03
devs = /dev/sdi1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.23]
host = cephosd03
devs = /dev/sdj1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.24]
host = cephosd03
devs = /dev/sdl1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.25]
host = cephosd03
devs = /dev/sdm1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.26]
host = cephosd03
devs = /dev/sdn1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.27]
host = cephosd04
devs = /dev/sdc1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.28]
host = cephosd04
devs = /dev/sdd1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.29]
host = cephosd04
devs = /dev/sdf1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.30]
host = cephosd04
devs = /dev/sdg1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.31]
host = cephosd04
devs = /dev/sdi1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.32]
host = cephosd04
devs = /dev/sdj1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.33]
host = cephosd04
devs = /dev/sdl1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.34]
host = cephosd04
devs = /dev/sdm1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[osd.35]
host = cephosd04
devs = /dev/sdn1
osd_journal = /mnt/ramdisk/$cluster-$id-journal

[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring


Thanks in advance,

Best regards,

*German Anders
*


















_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

4 x OSD Servers (3TB SAS 6G disks for OSD daemons & tmpfs for Journal -> there's one tmpfs of 36GB that is share by 9 OSD daemons, on each server)

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

cephosd02.dmesg.out
Description: Binary data

mail02-old.dmesg.out
Description: Binary data

cephosd02.syslog.out
Description: Binary data

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com