It is likely your 2 spinning disks cannot keep up with the load. Things
are likely to improve if you double your OSDs hooking them up to your
existing SSD journal. Technically it would be nice to run a
load/performance tool (either atop/collectl/sysstat) and measure how
busy your resources are, but it is most likely your 2 spinning disks
will show near 100% busy utilization.
filestore_max_sync_interval: i do not recommend decreasing this to 0.1,
i would keep it at 5 sec
osd_op_threads do not increase this unless you have enough cores.
but adding disks is the way to go
Maged
On 2017-08-22 20:08, fcid wrote:
> Hello everyone,
>
> I've been using ceph to provide storage using RBD for 60 KVM virtual machines
> running on proxmox.
>
> The ceph cluster we have is very small (2 OSDs + 1 mon per node, and a total
> of 3 nodes) and we are having some performace issues, like big latency times
> (apply lat:~0.5 s; commit lat: 0.001 s), which get worse by the weekly
> deep-scrubs.
>
> I wonder if doubling the numbers of OSDs would improve latency times, or if
> there is any other configuration tweak recommended for such small cluster.
> Also, I'm looking forward to read any experience of other users using a
> similiar configuration.
>
> Some technical info:
>
> - Ceph version: 10.2.5
>
> - OSDs have SSD journal (one SSD disk per 2 OSDs) and have a spindle for
> backend disk.
>
> - Using CFQ disk queue scheduler
>
> - OSD configuration excerpt:
>
> osd_recovery_max_active = 1
> osd_recovery_op_priority = 63
> osd_client_op_priority = 1
> osd_mkfs_options = -f -i size=2048 -n size=64k
> osd_mount_options_xfs = inode64,noatime,logbsize=256k
> osd_journal_size = 20480
> osd_op_threads = 12
> osd_disk_threads = 1
> osd_disk_thread_ioprio_class = idle
> osd_disk_thread_ioprio_priority = 7
> osd_scrub_begin_hour = 3
> osd_scrub_end_hour = 8
> osd_scrub_during_recovery = false
> filestore_merge_threshold = 40
> filestore_split_multiple = 8
> filestore_xattr_use_omap = true
> filestore_queue_max_ops = 2500
> filestore_min_sync_interval = 0.01
> filestore_max_sync_interval = 0.1
> filestore_journal_writeahead = true
>
> Best regards,
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
