Re: [PVE-User] ceph - slow rbd ls -l

[Eneko Lacunza]

Hi,

El 11/10/16 a las 03:43, Thiago Damas escribió:

2016-10-10 20:47 GMT-03:00 Lindsay Mathieson :


On 11/10/2016 7:59 AM, Thiago Damas wrote:


I'm experiencing some timeouts when creating new disks/VMs, using a ceph
storage.
Is there some way to reduce the long listing of rbd ls, ie "rbd ls -l"?


Hows your "ceph -s" look?


Are the logs showing any particular OSD's as being slow to respond?


Look:

~# ceph -s
 cluster 71b07854-5d50-46e7-a130-2230712cd3aa
  health HEALTH_OK
  monmap e9: 5 mons at
{0=192.168.33.AAA:6789/0,1=192.168.33.BBB:6789/0,2=192.168.33.CCC:6789/0,3=192.168.33.DDD:6789/0,4=192.168.33.EEE:6789/0}
 election epoch 294, quorum 0,1,2,3,4 0,1,2,3,4
  osdmap e3466: 40 osds: 40 up, 40 in
   pgmap v15080315: 2048 pgs, 1 pools, 7910 GB data, 1995 kobjects
 23467 GB used, 122 TB / 145 TB avail
 2048 active+clean
   client io 36159 kB/s rd, 11745 kB/s wr, 1385 op/s

~# ceph osd perf
osd fs_commit_latency(ms) fs_apply_latency(ms)
   0 1   10
   1 1   13
   2 07
   3 1   12
   4 06
   5 07
   6 05
   7 19
   8 1   10
   9 19
  10 1   11
  11 16
  12 1   13
  13 0   22
  14 27
  15 18
  16 18
  17 08
  18 19
  19 17
  20 3   13
  21 2   30
  22 3   16
  23 1   10
  24 18
  25 08
  26 17
  27 18
  28 17
  29 06
  30 1   13
  31 19
  32 1   11
  33 18
  34 17
  35 1   10
  36 1   13
  37 16
  38 19
  39 2   12

~# time rbd ls -l
...lines and lines...
real 0m37.351s
user 0m0.552s
sys 0m0.232s

For our cluster (12 OSDs) a pool with 58 images needs 5s the first time, 
then its <1s


Not sure what makes this so slow?

Cheers

--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943493611
  943324914
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
www.binovo.es

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] ceph - slow rbd ls -l

[Thiago Damas]

2016-10-10 20:47 GMT-03:00 Lindsay Mathieson :

> On 11/10/2016 7:59 AM, Thiago Damas wrote:
>
>> I'm experiencing some timeouts when creating new disks/VMs, using a ceph
>> storage.
>>Is there some way to reduce the long listing of rbd ls, ie "rbd ls -l"?
>>
>
> Hows your "ceph -s" look?
>
>
> Are the logs showing any particular OSD's as being slow to respond?
>
> --
> Lindsay Mathieson
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>


Look:

~# ceph -s
cluster 71b07854-5d50-46e7-a130-2230712cd3aa
 health HEALTH_OK
 monmap e9: 5 mons at
{0=192.168.33.AAA:6789/0,1=192.168.33.BBB:6789/0,2=192.168.33.CCC:6789/0,3=192.168.33.DDD:6789/0,4=192.168.33.EEE:6789/0}
election epoch 294, quorum 0,1,2,3,4 0,1,2,3,4
 osdmap e3466: 40 osds: 40 up, 40 in
  pgmap v15080315: 2048 pgs, 1 pools, 7910 GB data, 1995 kobjects
23467 GB used, 122 TB / 145 TB avail
2048 active+clean
  client io 36159 kB/s rd, 11745 kB/s wr, 1385 op/s

~# ceph osd perf
osd fs_commit_latency(ms) fs_apply_latency(ms)
  0 1   10
  1 1   13
  2 07
  3 1   12
  4 06
  5 07
  6 05
  7 19
  8 1   10
  9 19
 10 1   11
 11 16
 12 1   13
 13 0   22
 14 27
 15 18
 16 18
 17 08
 18 19
 19 17
 20 3   13
 21 2   30
 22 3   16
 23 1   10
 24 18
 25 08
 26 17
 27 18
 28 17
 29 06
 30 1   13
 31 19
 32 1   11
 33 18
 34 17
 35 1   10
 36 1   13
 37 16
 38 19
 39 2   12

~# time rbd ls -l
...lines and lines...
real 0m37.351s
user 0m0.552s
sys 0m0.232s


  Thiago
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] ceph - slow rbd ls -l

[Lindsay Mathieson]

On 11/10/2016 7:59 AM, Thiago Damas wrote:

I'm experiencing some timeouts when creating new disks/VMs, using a ceph
storage.
   Is there some way to reduce the long listing of rbd ls, ie "rbd ls -l"?


Hows your "ceph -s" look?


Are the logs showing any particular OSD's as being slow to respond?

--
Lindsay Mathieson

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user