subject:"Re\: \[ceph\-users\] High apply latency"

Re: [ceph-users] High apply latency

2018-02-06 Thread Frédéric Nass


Hi Jakub,


Le 06/02/2018 à 16:03, Jakub Jaszewski a écrit :

Hi Frederic,

I've not enable debug level logging on all OSDs, just on one for the 
test, need to double check that.
But looks that merging is ongoing on few OSDs or OSDs are faulty, I 
will dig into that tomorrow.

Write bandwidth is very random


I just reread the whole thread:

- Splitting is not happening anymore - if it ever did - that's for sure.
- Regarding the write bandwidth variations, it seems that these 
variations only concern EC 6+3 pools.
- As you get more than a 1.2 GB/s on replicated pools with 4MB iops, I 
would think that neither NVMe, nor PERC or HDDs is to blame.


Did you check CPU load during EC 6+3 writes on pool 
default.rgw.buckets.data ?


If you don't see any 100% CPU load, nor any 100% iostat issues on either 
the NVMe disk or HDDs, then I would benchmark the network for bandwidth 
or latency issues.


BTW, did you see that some of your OSDs were not tagged as 'hdd' (ceph 
osd df tree).





# rados bench -p default.rgw.buckets.data 120 write
hints = 1
Maintaining 16 concurrent writes of 4194432 bytes to objects of size 
4194432 for up to 120 seconds or 0 objects

Object prefix: benchmark_data_sg08-09_59104
sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg 
lat(s)

0       0         0         0         0         0  -           0
1      16       155       139    555.93   556.017  0.0750027     0.10687
2      16       264       248   495.936   436.013 0.154185    0.118693
3      16       330       314   418.616   264.008 0.118476    0.142667
4      16       415       399   398.953    340.01  0.0873379     0.15102
5      16       483       467   373.557   272.008 0.750453    0.159819
6      16       532       516   343.962   196.006  0.0298334    0.171218
7      16       617       601   343.391    340.01 0.192698    0.177288
8      16       700       684   341.963    332.01  0.0281355    0.171277
9      16       762       746   331.521   248.008  0.0962037    0.163734
 10      16       804       788   315.167   168.005  1.40356    0.196298
 11      16       897       881    320.33   372.011  0.0369085     0.19496
 12      16       985       969   322.966   352.011  0.0290563    0.193986
 13      15      1106      1091   335.657   488.015  0.0617642    0.188703
 14      16      1166      1150   328.537   236.007  0.0401884    0.186206
 15      16      1251      1235   329.299    340.01 0.171256    0.190974
 16      16      1339      1323   330.716   352.011 0.024222    0.189901
 17      16      1417      1401   329.613    312.01  0.0289473    0.186562
 18      16      1465      1449   321.967   192.006 0.028123    0.189153
 19      16      1522      1506    317.02   228.007 0.265448    0.188288
2018-02-06 13:43:21.412512 min lat: 0.0204657 max lat: 3.61509 avg 
lat: 0.18918
sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg 
lat(s)

 20      16      1564      1548   309.568   168.005  0.0327581     0.18918
 21      16      1636      1620    308.54   288.009  0.0715159    0.187381
 22      16      1673      1657   301.242   148.005  1.57285    0.191596
 23      16      1762      1746   303.621   356.011  6.00352    0.206217
 24      16      1885      1869   311.468   492.015  0.0298435    0.203874
 25      16      2010      1994   319.008   500.015  0.0258761    0.199652
 26      16      2116      2100   323.044   424.013  0.0533319     0.19631
 27      16      2201      2185    323.67    340.01 0.134796    0.195953
 28      16      2257      2241    320.11   224.007 0.473629    0.196464
 29      16      2333      2317   319.554   304.009  0.0362741    0.198054
 30      16      2371      2355   313.968   152.005 0.438141    0.200265
 31      16      2459      2443   315.194   352.011  0.0610629    0.200858
 32      16      2525      2509   313.593   264.008  0.0234799    0.201008
 33      16      2612      2596   314.635   348.011 0.072019    0.199094
 34      16      2682      2666   313.615   280.009  0.10062    0.197586
 35      16      2757      2741   313.225   300.009  0.0552581    0.196981
 36      16      2849      2833   314.746   368.011 0.257323     0.19565
 37      16      2891      2875   310.779   168.005  0.0918386     0.19556
 38      16      2946      2930    308.39   220.007  0.0276621    0.195792
 39      16      2975      2959   303.456   116.004  0.0588971     0.19952
2018-02-06 13:43:41.415107 min lat: 0.0204657 max lat: 7.9873 avg lat: 
0.198749
sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg 
lat(s)

 40      16      3060      3044   304.369    340.01  0.0217136    0.198749
 41      16      3098      3082   300.652   152.005  0.0717398    0.199052
 42      16      3141      3125   297.589   172.005  0.0257422    0.201899
 43      15      3241      3226   300.063   404.012  0.0733869    0.209446
 44      16      3332      3316   301.424   360.011  0.0327249    0.206686
 45      16      3430      3414   303.436   392.012  0.0413156

Re: [ceph-users] High apply latency

2018-02-06 Thread Jakub Jaszewski

Hi Frederic,

I've not enable debug level logging on all OSDs, just on one for the test,
need to double check that.
But looks that merging is ongoing on few OSDs or OSDs are faulty, I will
dig into that tomorrow.
Write bandwidth is very random

# rados bench -p default.rgw.buckets.data 120 write
hints = 1
Maintaining 16 concurrent writes of 4194432 bytes to objects of size
4194432 for up to 120 seconds or 0 objects
Object prefix: benchmark_data_sg08-09_59104
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
0   0 0 0 0 0   -
 0
1  16   155   139555.93   556.017   0.0750027
 0.10687
2  16   264   248   495.936   436.0130.154185
0.118693
3  16   330   314   418.616   264.0080.118476
0.142667
4  16   415   399   398.953340.01   0.0873379
 0.15102
5  16   483   467   373.557   272.0080.750453
0.159819
6  16   532   516   343.962   196.006   0.0298334
0.171218
7  16   617   601   343.391340.010.192698
0.177288
8  16   700   684   341.963332.01   0.0281355
0.171277
9  16   762   746   331.521   248.008   0.0962037
0.163734
   10  16   804   788   315.167   168.005 1.40356
0.196298
   11  16   897   881320.33   372.011   0.0369085
 0.19496
   12  16   985   969   322.966   352.011   0.0290563
0.193986
   13  15  1106  1091   335.657   488.015   0.0617642
0.188703
   14  16  1166  1150   328.537   236.007   0.0401884
0.186206
   15  16  1251  1235   329.299340.010.171256
0.190974
   16  16  1339  1323   330.716   352.0110.024222
0.189901
   17  16  1417  1401   329.613312.01   0.0289473
0.186562
   18  16  1465  1449   321.967   192.0060.028123
0.189153
   19  16  1522  1506317.02   228.0070.265448
0.188288
2018-02-06 13:43:21.412512 min lat: 0.0204657 max lat: 3.61509 avg lat:
0.18918
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
   20  16  1564  1548   309.568   168.005   0.0327581
 0.18918
   21  16  1636  1620308.54   288.009   0.0715159
0.187381
   22  16  1673  1657   301.242   148.005 1.57285
0.191596
   23  16  1762  1746   303.621   356.011 6.00352
0.206217
   24  16  1885  1869   311.468   492.015   0.0298435
0.203874
   25  16  2010  1994   319.008   500.015   0.0258761
0.199652
   26  16  2116  2100   323.044   424.013   0.0533319
 0.19631
   27  16  2201  2185323.67340.010.134796
0.195953
   28  16  2257  2241320.11   224.0070.473629
0.196464
   29  16  2333  2317   319.554   304.009   0.0362741
0.198054
   30  16  2371  2355   313.968   152.0050.438141
0.200265
   31  16  2459  2443   315.194   352.011   0.0610629
0.200858
   32  16  2525  2509   313.593   264.008   0.0234799
0.201008
   33  16  2612  2596   314.635   348.0110.072019
0.199094
   34  16  2682  2666   313.615   280.009 0.10062
0.197586
   35  16  2757  2741   313.225   300.009   0.0552581
0.196981
   36  16  2849  2833   314.746   368.0110.257323
 0.19565
   37  16  2891  2875   310.779   168.005   0.0918386
 0.19556
   38  16  2946  2930308.39   220.007   0.0276621
0.195792
   39  16  2975  2959   303.456   116.004   0.0588971
 0.19952
2018-02-06 13:43:41.415107 min lat: 0.0204657 max lat: 7.9873 avg lat:
0.198749
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
   40  16  3060  3044   304.369340.01   0.0217136
0.198749
   41  16  3098  3082   300.652   152.005   0.0717398
0.199052
   42  16  3141  3125   297.589   172.005   0.0257422
0.201899
   43  15  3241  3226   300.063   404.012   0.0733869
0.209446
   44  16  3332  3316   301.424   360.011   0.0327249
0.206686
   45  16  3430  3414   303.436   392.012   0.0413156
0.203727
   46  16  3534  3518   305.882   416.0130.033638
0.202182
   47  16  3602  3586   305.161   272.008   0.0453557
0.200028
   48  16  3663  3647   303.886   244.007   0.0779019
0.199777
   49  16  3736  3720   303.643   292.009   0.0285231
0.206274
   50  16  3849  3833   306.609   452.014   0.0537071
0.208127
   51  16  3909  3893   305.303   240.007   0.0366709
0.207793
   52  16  3972  3956   304.277   252.008   0.0289131
0.207989
   53  16  4048  4032   304.272   304.009   0.0348617
0.207844
   54  16  4114  4098   303.525   264.008   0.0799526
 0.20701
   55  16

Re: [ceph-users] High apply latency

2018-02-05 Thread Frédéric Nass

Hi Jakub,

Le 05/02/2018 à 12:26, Jakub Jaszewski a écrit :

Hi Frederic,

Many thanks for your contribution to the topic!

I've just set logging level 20 for filestore via

ceph tell osd.0 config set debug_filestore 20

but so far
found
 nothing by keyword 'split'
 in /var/log/ceph/ceph-osd.0.log

So, if you're running ceph > 12.2.1, that means splitting is not 
happening. Did you check during writes ? Did you check other OSDs logs ?

Actually, splitting should not happen now that you've increased 
filestore_merge_threshold and filestore_split_multiple values.

I've also run your script across the cluster nodes, results as follows

id=3, pool=volumes, objects=10454548, avg=160.28
id=20, pool=default.rgw.buckets.data, objects=22419862, avg=35.2344
id=3, pool=volumes, objects=10454548, avg=159.22
id=20, pool=default.rgw.buckets.data, objects=22419862, avg=35.9994
id=3, pool=volumes, objects=10454548, avg=159.843
id=20, pool=default.rgw.buckets.data, objects=22419862, avg=34.7435
id=3, pool=volumes, objects=10454548, avg=159.695
id=20, pool=default.rgw.buckets.data, objects=22419862, avg=35.0579
id=3, pool=volumes, objects=10454548, avg=160.594
id=20, pool=default.rgw.buckets.data, objects=22419862, avg=34.7757
id=3, pool=volumes, objects=10454548, avg=160.099
id=20, pool=default.rgw.buckets.data, objects=22419862, avg=33.8517
id=3, pool=volumes, objects=10454548, avg=159.912
id=20, pool=default.rgw.buckets.data, objects=22419862, avg=37.5698
id=3, pool=volumes, objects=10454548, avg=159.407
id=20, pool=default.rgw.buckets.data, objects=22419862, avg=35.4991
id=3, pool=volumes, objects=10454548, avg=160.075
id=20, pool=default.rgw.buckets.data, objects=22419862, avg=35.481

Looks like there is nothing to be handled by split, am I right? But 
what about merging ? Avg is less than 40

 , should directories structure be reduced now?

It should, I guess. But then you'd see blocked requests on every object 
deletion. If you do, you might want to set filestore_merge_threshold 
to -40 (negative value) so merging does not happen anymore.

Splitting would still happen over 5120 files per subdirectory.

    "

filestore_merge_threshold": "40",
"filestore_split_multiple": "8",
"filestore_split_rand_factor": "20",

M
ay I ask for the link to documentation where I can read more about 
OSD underlying directory structure?

I'm not aware of any related documentation.

Do you still observe slow or blocked requests now that you've increased 
filestore_merge_threshold and filestore_split_multiple ?

Regards,

Frédéric.

And just noticed log entries in /var/log/ceph/ceph-osd.0.log

2018-02-05 11:22:03.346400 7f3cc94fe700  0 -- 10.212.14.11:6818/4702 
 >> 10.212.14.17:6802/82845 
 conn(0xe254cca800 :6818 
s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 
l=0).handle_connect_msg accept connect_seq 27 vs existing csq=27 
existing_state=STATE_STANDBY
2018-02-05 11:22:03.346583 7f3cc94fe700  0 -- 10.212.14.11:6818/4702 
 >> 10.212.14.17:6802/82845 
 conn(0xe254cca800 :6818 
s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 
l=0).handle_connect_msg accept connect_seq 28 vs existing csq=27 
existing_state=STATE_STANDBY

M

any thanks!
Jakub

On Mon, Feb 5, 2018 at 9:56 AM, Frédéric Nass 
> wrote:

Hi,

In addition, starting with Luminous 12.2.1 (RHCS 3), splitting ops
should be loggued with default setting of debug level messages:
https://github.com/ceph/ceph/blob/v12.2.1/src/os/filestore/HashIndex.cc#L320

There's also a RFE for merging to be loggued as well as splitting:
https://bugzilla.redhat.com/show_bug.cgi?id=1523532

Regards,

Frédéric.

Le 02/02/2018 à 17:00, Frédéric Nass a écrit :

Hi,

Split and merge operations happen during writes only, splitting
on file creation and merging on file deletion.

As you don't see any blocked requests during reads I would guess
your issue happens during splitting. Now that you increased
filestore_merge_threshold and filestore_split_multiple, you
shouldn't expect any splitting operations to happen any soon, nor
any merging operations, unless your workload consists of writing
a huge number of files and removing them.

You should check how many files are in each lower directories of
pool 20's PGs. This would help to confirm that the blocked
requests come with the splitting.

We now use the below script (on one of the OSD nodes) to get an
average value of the number of files in some PGs of each pool and
run this script every 5 minutes with Munin to get a graph out of
the values.
This way, we can anticipate the

Re: [ceph-users] High apply latency

2018-02-02 Thread Piotr Dałek


On 18-02-02 09:55 AM, Jakub Jaszewski wrote:

Hi,

So I have changed merge & split settings to
filestore_merge_threshold = 40
filestore_split_multiple = 8

and restart all OSDs , host by host.

Let me ask a question, although the pool default.rgw.buckets.data that was 
affected prior to the above change has higher write bandwidth it is very 
random now. Writes are random for other pools (same for EC and replicated 
types) too, before the change writes to replicated pools were much more stable.

Reads from pools look fine and stable.

Is it the result of mentioned change ? Is PG directory structure updating or 
...?


The HUGE problem with filestore is that it can't handle large number of 
small objects well. Sure, if the number only grows slowly (case with RBD 
images) then it's probably not that noticeable, but in case of 31 millions 
of objects that come and go at random pace you're going to hit frequent 
problems with filestore collections splitting and merging. Pre-Luminous, it 
happened on all osds hosting particular collection at once, and in Luminous 
there's "filestore split rand factor" which according to docs:


Description:  A random factor added to the split threshold to avoid
  too many filestore splits occurring at once. See
  ``filestore split multiple`` for details.
  This can only be changed for an existing osd offline,
  via ceph-objectstore-tool's apply-layout-settings command.

You may want to try the above as well.

--
Piotr Dałek
piotr.da...@corp.ovh.com
https://www.ovh.com/us/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High apply latency

2018-02-02 Thread Jakub Jaszewski

Hi,

So I have changed merge & split settings to

filestore_merge_threshold = 40
filestore_split_multiple = 8

and restart all OSDs , host by host.

Let me ask a question, although the pool default.rgw.buckets.data that was
affected prior to the above change has higher write bandwidth it is very
random now. Writes are random for other pools (same for EC and replicated
types) too, before the change writes to replicated pools were much more
stable.
Reads from pools look fine and stable.

Is it the result of mentioned change ? Is PG directory structure updating
or ...?




# rados bench -p default.rgw.buckets.data 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194432 bytes to objects of size
4194432 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_sg08-09_19744
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
0   0 0 0 0 0   -
 0
1  16   169   153   611.982   612.019   0.0547835
0.096198
2  16   314   298   595.958   580.018   0.0485904
0.103653
3  16   454   438   583.956   560.017   0.0747779
0.106501
4  16   556   540539.96   408.0120.111734
0.110132
5  16   624   608   486.364   272.0080.69
0.120221
6  16   651   635   423.302   108.003   0.0706783
0.126086
7  16   730   714407.97316.010.362838
0.153117
8  16   802   786392.97   288.009   0.0261249
0.154728
9  16   858   842   374.194   224.007   0.0703766
0.159723
   10  16   913   897   358.773   220.0070.169544
0.173459
Total time run: 10.386646
Total writes made:  913
Write size: 4194432
Object size:4194432
Bandwidth (MB/sec): 351.616
Stddev Bandwidth:   173.421
Max bandwidth (MB/sec): 612.019
Min bandwidth (MB/sec): 108.003
Average IOPS:   87
Stddev IOPS:43
Max IOPS:   153
Min IOPS:   27
Average Latency(s): 0.179969
Stddev Latency(s):  0.321098
Max latency(s): 2.60469
Min latency(s): 0.0209669





# rados bench -p default.rgw.buckets.data 10 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
0   0 0 0 0 0   -
 0
1  15   470   455   1818.75   1820.06   0.0462129
 0.0337452
eTotal time run:   1.960388
Total reads made: 913
Read size:4194432
Object size:  4194432
Bandwidth (MB/sec):   1862.95
Average IOPS: 465
Stddev IOPS:  0
Max IOPS: 455
Min IOPS: 455
Average Latency(s):   0.0335775
Max latency(s):   0.259385
Min latency(s):   0.0169049
#

# rados bench -p default.rgw.buckets.data 10 rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
0   0 0 0 0 0   -
 0
1  15   476   461   1843.81   1844.06   0.0272928
 0.0335396
2  15   954   939   1877.83   1912.06   0.0434087
 0.0332626
3  16  1436  1420   1893.16   1924.06   0.0408819
 0.0330526
4  15  1925  1910   1909.84   1960.06   0.0259839
 0.0328248
5  15  2408  2393   1914.25   1932.06   0.0427577
 0.0328096
6  16  2891  2875   1916.52   1928.06   0.0273633
 0.0327565
7  15  3377  3362  1921   1948.06   0.0294795
0.032736
8  16  3875  3859   1929.36   1988.06   0.0242193
 0.03258
9  16  4351  4335   1926.53   1904.06   0.0203889
 0.0326336
   10  16  4855  4839   1935.46   2016.06   0.0325821
 0.0324852
Total time run:   10.032589
Total reads made: 4856
Read size:4194432
Object size:  4194432
Bandwidth (MB/sec):   1936.15
Average IOPS: 484
Stddev IOPS:  11
Max IOPS: 504
Min IOPS: 461
Average Latency(s):   0.0325118
Max latency(s):   0.250729
Min latency(s):   0.0149525
#








# rados bench -p benchmark_erasure_coded 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4202496 bytes to objects of size
4202496 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_sg08-09_20674
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
0   0 0 0 0 0   -
 0
1  16   400   384   1538.92  1539   0.0234887
 0.0401751
2  16   824   808   1618.98   1699.31   0.0215378
 0.0390947
3  16  1212  11961597.6   1555.03   0.0169675
 0.0394463
4  16  1551  1535   1537.81   1358.65   0.0605608
 0.0413843
5  16  1947  1931   1547.63   1587.09   0.0183851
 0.0409588
6  16  2099  2083   1391.21   609.1880.014569

Re: [ceph-users] High apply latency

2018-02-01 Thread Jakub Jaszewski

Regarding split & merge, I have default values
filestore_merge_threshold = 10
filestore_split_multiple = 2

according to https://bugzilla.redhat.com/show_bug.cgi?id=1219974 the
recommended values are

filestore_merge_threshold = 40
filestore_split_multiple = 8

Is it something that I can easily change to default or lower values than
proposed in case of further performance degradation ?

I did tests of 4 pools: 2 replicated pools (x3 ) and 2 EC  pools (k=6,m=3)

The pool with the lowest bandwidth has osd tree structure like
├── 20.115s1_head
│   └── DIR_5
│   └── DIR_1
│   ├── DIR_1
│   │   ├── DIR_0
│   │   ├── DIR_1
│   │   ├── DIR_2
│   │   │   ├── DIR_0
│   │   │   ├── DIR_1
│   │   │   ├── DIR_2
│   │   │   ├── DIR_3
│   │   │   ├── DIR_4
│   │   │   ├── DIR_5
│   │   │   ├── DIR_6
│   │   │   ├── DIR_7
│   │   │   ├── DIR_8
│   │   │   ├── DIR_9
│   │   │   ├── DIR_A
│   │   │   ├── DIR_B
│   │   │   ├── DIR_C
│   │   │   ├── DIR_D
│   │   │   ├── DIR_E
│   │   │   └── DIR_F


Tests results

# rados bench -p default.rgw.buckets.data 10 write
hints = 1
Maintaining 16 concurrent writes of 4194432 bytes to objects of size
4194432 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_sg08-09_180679
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
0   0 0 0 0 0   -
 0
1  16   129   113   451.975   452.014   0.0376714
0.128027
2  16   209   193   385.964320.010.119609
0.138517
3  16   235   219   291.974   104.003   0.0337624
 0.13731
4  16   235   219   218.981 0   -
 0.13731
5  16   266   250   199.983   62.00190.111673
0.238424
6  16   317   301   200.649   204.006   0.0340569
0.298489
7  16   396   380   217.124316.01   0.0379956
0.283458
8  16   444   428   213.981   192.006   0.0304383
0.274193
9  16   485   469   208.426   164.0050.391956
0.283421
   10  16   496   480   191.983   44.00130.104497
0.292074
   11  16   497   481   174.894   4.000120.85
0.293545
   12  16   497   481160.32 0   -
0.293545
   13  16   497   481   147.987 0   -
0.293545
   14  16   497   481   137.417 0   -
0.293545
Total time run: 14.493353
Total writes made:  497
Write size: 4194432
Object size:4194432
Bandwidth (MB/sec): 137.171
Stddev Bandwidth:   147.001
Max bandwidth (MB/sec): 452.014
Min bandwidth (MB/sec): 0
Average IOPS:   34
Stddev IOPS:36
Max IOPS:   113
Min IOPS:   0
Average Latency(s): 0.464281
Stddev Latency(s):  1.09388
Max latency(s): 6.3723
Min latency(s): 0.023835
Cleaning up (deleting benchmark objects)
Removed 497 objects
Clean up completed and total clean up time :10.622382
#


# rados bench -p benchmark_erasure_coded 10 write
hints = 1
Maintaining 16 concurrent writes of 4202496 bytes to objects of size
4202496 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_sg08-09_180807
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
0   0 0 0 0 0   -
 0
1  16   424   408   1635.11   1635.19   0.0490434
 0.0379616
2  16   828   812   1627.03   1619.16   0.0616501
 0.0388467
3  16  1258  1242   1659.06   1723.36   0.0304412
 0.0384537
4  16  1659  1643   1646.03   1607.13   0.0155402
 0.0387351
5  16  2053  2037   1632.61   1579.08   0.0453354
 0.0390236
6  16  2455  2439  1629   1611.14   0.0485313
 0.0392376
7  16  2649  2633   1507.34   777.516   0.0148972
 0.0393161
8  16  2858  2842   1423.61   837.633   0.0157639
 0.0449088
9  16  3245  3229   1437.75   1551.02   0.0200845
 0.0444847
   10  16  3629  3613   1447.85  1539   0.0654451
 0.0441569
Total time run: 10.229591
Total writes made:  3630
Write size: 4202496
Object size:4202496
Bandwidth (MB/sec): 1422.18
Stddev Bandwidth:   341.609
Max bandwidth (MB/sec): 1723.36
Min bandwidth (MB/sec): 777.516
Average IOPS:   354
Stddev IOPS:85
Max IOPS:   430
Min IOPS:   194
Average Latency(s): 0.0448612
Stddev Latency(s):  0.0712224
Max latency(s): 1.08353
Min latency(s): 0.0134629
Cleaning up (deleting benchmark objects)
Removed 3630 objects
Clean up completed and total clean up time :2.321669
#



# rados bench -p volumes 10 write
hints = 1

Re: [ceph-users] High apply latency

2018-02-01 Thread Jakub Jaszewski

Thanks, I've temporary disabled both scrubbing and deep-scrubbing, things
are getting better I feel.

I just noticed high traffic generated on pool default.rgw.gc

pool default.rgw.gc id 7
  client io 2162 MB/s rd, 0 B/s wr, 3023 op/s rd, 0 op/s wr


There is lot of data written via radosgw

GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
385T  142T 243T 63.08  31384k
POOLS:
NAME   ID QUOTA OBJECTS
 QUOTA BYTES USED   %USED MAX AVAIL OBJECTS  DIRTY
READ   WRITE  RAW USED
volumes3  N/A
 N/A 40352G 72.1915547G 10352564 10109k
  2155M  2574M 118T
default.rgw.gc 7  N/A
 N/A  0 015547G   32 32
  1820M  5352k0
default.rgw.buckets.data   20 N/A
 N/A 80596G 72.1631094G 21235439 20737k
   123M   114M 118T


Although reads stats on default.rgw.gc are high according to ceph osd pool
stats  command, I don't see much rMB/s on HDDs on any cluster node. However
r/s seems to be significant? . Any idea why it is like that ?

# iostat -xdm 5 1000

Device: rrqm/s   wrqm/s r/s w/srMB/swMB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda   0,00 2,200,002,60 0,00 0,0214,77
   0,000,000,000,00   0,00   0,00
sdb   0,00 0,40   35,80   10,40 0,66 1,74   106,04
   0,408,74   10,482,77   6,86  31,68
sdc   0,00 0,60   45,80   15,60 0,78 3,06   127,87
   0,63   10,20   12,842,46   7,10  43,60
sdd   0,00 0,60   27,00   10,60 0,43 1,79   121,25
   0,318,19   11,410,00   6,60  24,80
sde   0,00 1,40   36,80   18,80 0,65 3,96   169,60
   0,519,17   12,851,96   5,99  33,28
sdf   0,00 0,000,000,00 0,00 0,00 0,00
   0,000,000,000,00   0,00   0,00
sdg   0,00 0,40   33,407,00 0,52 0,9373,26
   0,41   10,16   12,100,91   7,37  29,76
sdh   0,00 1,40   53,60   13,60 0,94 2,62   108,32
   0,70   10,45   12,611,94   7,42  49,84
sdi   0,00 1,00   34,40   10,80 0,61 2,14   124,82
   0,388,44   10,372,30   6,57  29,68
sdj   0,00 0,000,000,00 0,00 0,00 0,00
   0,000,000,000,00   0,00   0,00
sdk   0,00 1,00   39,20   13,20 0,69 2,66   131,13
   0,60   11,45   14,312,97   7,21  37,76
sdl   0,00 0,80   25,40   11,20 0,45 1,92   132,86
   0,298,00   10,492,36   6,54  23,92
sdm   0,00 1,00   37,208,00 0,72 1,4698,82
   0,449,73   11,083,50   7,86  35,52
nvme0n1   0,00 0,000,00  700,60 0,0020,3459,47
   0,070,100,000,10   0,02   1,44
dm-0  0,00 0,000,004,80 0,00 0,02 8,00
   0,000,000,000,00   0,00   0,00


Can I schedule garbage collection process to run on particular days/hours ?

Rados bench looks quite fine during garbage collection run I think.

# rados bench -p benchmark_replicated 20 write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size
4194304 for up to 20 seconds or 0 objects
Object prefix: benchmark_data_sg08-09_175315
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg
lat(s)
0   0 0 0 0 0   -
 0
1  16   364   348   1391.88  1392   0.0638181
 0.0443111
2  16   711   695   1389.83  1388   0.0166734
 0.0454365
3  16  1052  1036   1381.15  1364   0.0756525
 0.0458462
4  16  1401  1385   1384.82  1396   0.0800253
 0.0459712
5  16  1764  1748   1398.21  14520.048202
 0.0455071
6  16  2117  2101   1400.48  1412   0.0154943
 0.0455181
7  16  2468  2452   1400.96  1404   0.0324203
 0.0454904
8  16  2809  2793   1396.32  1364   0.0425057
 0.0456571
9  16  3175  3159   1403.82  1464   0.0535376
 0.0454215
   10  16  3541  3525   1409.82  1464   0.0668655
 0.0452501
   11  16  3911  3895   1416.18  1480   0.0506286
 0.0450723
   12  16  4267  4251   1416.82  1424   0.0266732
 0.0450444
   13  16  4615  45991414.9  13920.101581
0.045124
   14  16  4977  4961   1417.25  1448   0.0342007
 0.0450346
   15  16  5351  5335   1422.48  14960.022117
 0.0449238
   16  16  5691  5675   1418.57  13600.022683

Re: [ceph-users] High apply latency

2018-01-31 Thread Sergey Malinin

Deep scrub is I/O-expensive. If deep scrub is unnecessary, you can disable it 
with "ceph osd pool set  nodeep-scrub". 


On Thursday, February 1, 2018 at 00:10, Jakub Jaszewski wrote:

>  3active+clean+scrubbing+deep 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High apply latency

2018-01-31 Thread Jakub Jaszewski

Hi Luis,

Thanks for your comment, I see high %util for few HDDs per each ceph node
but actually there is very low traffic from client.

iostat -xd shows ongoing operations

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda   0,00 1,600,003,00 0,0018,4012,27
   0,000,000,000,00   0,00   0,00
sdb   0,00 0,200,608,0014,40   488,10   116,86
   0,000,568,000,00   0,56   0,48
sdc   0,00 0,00  153,801,80 25304,0031,30   325,65
   1,127,207,280,00   4,21  65,52
sdd   0,00 5,40  406,80   44,00 102275,20  3295,60
 468,37 1,854,124,292,53   2,11  95,12
sde   0,00 0,603,20   12,0051,20  2461,50   330,62
   0,074,32   12,252,20   2,63   4,00
sdf   0,00 0,401,408,2044,00  1424,90   306,02
   0,011,50   10,290,00   1,50   1,44
sdg   0,00 0,60   92,80   19,00  5483,20  2998,90   151,74
   0,988,74   10,360,84   7,40  82,72
sdh   0,00 0,00  154,401,40 25299,2074,20   325,72
   1,076,886,940,00   4,07  63,44
sdi   0,00 0,000,207,8012,80   397,50   102,58
   0,000,30   12,000,00   0,30   0,24
sdj   0,00 0,200,004,00 0,00   645,60   322,80
   0,000,000,000,00   0,00   0,00
sdk   0,00 0,201,40   15,6032,00  1956,50   233,94
   0,021,08   13,140,00   1,08   1,84
sdl   0,00 0,400,604,0016,80   447,00   201,65
   0,024,35   20,002,00   2,78   1,28
sdm   0,00 0,00   10,004,40   232,00   521,80   104,69
   0,085,898,480,00   4,89   7,04
dm-0  0,00 0,000,004,60 0,0018,40 8,00
   0,000,000,000,00   0,00   0,00
nvme0n1   0,00 0,000,00  124,80 0,00 10366,40   166,13
   0,010,120,000,12   0,03   0,32

when ceph -s shows low client traffic

# ceph -s
  cluster:
id: 1023c49f-3b10-42de-9f62-9b122db32a9a
health: HEALTH_OK

  services:
mon: 3 daemons, quorum host01,host02,host03
mgr: host02(active), standbys: host01, host03
osd: 108 osds: 106 up, 106 in
rgw: 3 daemons active

  data:
pools:   22 pools, 4880 pgs
objects: 31121k objects, 119 TB
usage:   241 TB used, 143 TB / 385 TB avail
pgs: 4875 active+clean
 3active+clean+scrubbing+deep
 2active+clean+scrubbing

  io:
client:   17646 B/s rd, 19038 kB/s wr, 4 op/s rd, 175 op/s wr


Is there any background tasks running and utilizing disks? Is it scrubbing
generating this load?
 3active+clean+scrubbing+deep
 2active+clean+scrubbing




 Thanks
Jakub

On Wed, Jan 31, 2018 at 3:59 PM, Luis Periquito  wrote:

> on a cursory look of the information it seems the cluster is
> overloaded with the requests.
>
> Just a guess, but if you look at IO usage on those spindles they'll be
> at or around 100% usage most of the time.
>
> If that is the case then increasing the pg_num and pgp_num won't help,
> and short term, will make it worse.
>
> Metadata pools (like default.rgw.buckets.index) really excel in a SSD
> pool, even if small. I carved a small OSD in the journal SSDs for
> those kinds of workloads.
>
> On Wed, Jan 31, 2018 at 2:26 PM, Jakub Jaszewski
>  wrote:
> > Is it safe to increase pg_num and pgp_num from 1024 up to 2048 for
> volumes
> > and default.rgw.buckets.data pools?
> > How will it impact cluster behavior? I guess cluster rebalancing will
> occur
> > and will take long time considering amount of data we have on it ?
> >
> > Regards
> > Jakub
> >
> >
> >
> > On Wed, Jan 31, 2018 at 1:37 PM, Jakub Jaszewski <
> jaszewski.ja...@gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> I'm wondering why slow requests are being reported mainly when the
> request
> >> has been put into the queue for processing by its PG  (queued_for_pg ,
> >> http://docs.ceph.com/docs/master/rados/troubleshooting/
> troubleshooting-osd/#debugging-slow-request).
> >> Could it be due too low pg_num/pgp_num ?
> >>
> >> It looks that slow requests are mainly addressed to
> >> default.rgw.buckets.data (pool id 20) , volumes (pool id 3) and
> >> default.rgw.buckets.index (pool id 14)
> >>
> >> 2018-01-31 12:06:55.899557 osd.59 osd.59 10.212.32.22:6806/4413 38 :
> >> cluster [WRN] slow request 30.125793 seconds old, received at 2018-01-31
> >> 12:06:25.773675: osd_op(client.857003.0:126171692 3.a4fec1ad 3.a4fec1ad
> >> (undecoded) ack+ondisk+write+known_if_redirected e5722) currently
> >> queued_for_pg
> >>
> >> Btw how can I get more human-friendly client information from log entry
> >> like above ?
> >>
> >> Current pg_num/pgp_num
> >>
> >> pool 3 'volumes'

Re: [ceph-users] High apply latency

2018-01-31 Thread Luis Periquito

on a cursory look of the information it seems the cluster is
overloaded with the requests.

Just a guess, but if you look at IO usage on those spindles they'll be
at or around 100% usage most of the time.

If that is the case then increasing the pg_num and pgp_num won't help,
and short term, will make it worse.

Metadata pools (like default.rgw.buckets.index) really excel in a SSD
pool, even if small. I carved a small OSD in the journal SSDs for
those kinds of workloads.

On Wed, Jan 31, 2018 at 2:26 PM, Jakub Jaszewski
 wrote:
> Is it safe to increase pg_num and pgp_num from 1024 up to 2048 for volumes
> and default.rgw.buckets.data pools?
> How will it impact cluster behavior? I guess cluster rebalancing will occur
> and will take long time considering amount of data we have on it ?
>
> Regards
> Jakub
>
>
>
> On Wed, Jan 31, 2018 at 1:37 PM, Jakub Jaszewski 
> wrote:
>>
>> Hi,
>>
>> I'm wondering why slow requests are being reported mainly when the request
>> has been put into the queue for processing by its PG  (queued_for_pg ,
>> http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#debugging-slow-request).
>> Could it be due too low pg_num/pgp_num ?
>>
>> It looks that slow requests are mainly addressed to
>> default.rgw.buckets.data (pool id 20) , volumes (pool id 3) and
>> default.rgw.buckets.index (pool id 14)
>>
>> 2018-01-31 12:06:55.899557 osd.59 osd.59 10.212.32.22:6806/4413 38 :
>> cluster [WRN] slow request 30.125793 seconds old, received at 2018-01-31
>> 12:06:25.773675: osd_op(client.857003.0:126171692 3.a4fec1ad 3.a4fec1ad
>> (undecoded) ack+ondisk+write+known_if_redirected e5722) currently
>> queued_for_pg
>>
>> Btw how can I get more human-friendly client information from log entry
>> like above ?
>>
>> Current pg_num/pgp_num
>>
>> pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
>> rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
>> stripe_width 0 application rbd
>> removed_snaps [1~3]
>> pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2
>> crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags
>> hashpspool stripe_width 0 application rgw
>> pool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1
>> object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags
>> hashpspool stripe_width 4224 application rgw
>>
>> Usage
>>
>> GLOBAL:
>> SIZE AVAIL RAW USED %RAW USED OBJECTS
>> 385T  144T 241T 62.54  31023k
>> POOLS:
>> NAME ID QUOTA OBJECTS QUOTA BYTES
>> USED   %USED MAX AVAIL OBJECTS  DIRTY  READ   WRITE
>> RAW USED
>> volumes  3  N/A   N/A
>> 40351G 70.9116557G 10352314 10109k  2130M  2520M
>> 118T
>> default.rgw.buckets.index14 N/A   N/A
>> 0 016557G  205205   160M 27945k
>> 0
>> default.rgw.buckets.data 20 N/A   N/A
>> 79190G 70.5133115G 20865953 20376k   122M   113M
>> 116T
>>
>>
>>
>> # ceph osd pool ls detail
>> pool 0 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
>> rjenkins pg_num 64 pgp_num 64 last_change 4502 flags hashpspool stripe_width
>> 0 application rbd
>> pool 1 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash
>> rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
>> stripe_width 0 application rbd
>> pool 2 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
>> rjenkins pg_num 512 pgp_num 512 last_change 5175 flags hashpspool
>> stripe_width 0 application rbd
>> removed_snaps [1~7,14~2]
>> pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
>> rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
>> stripe_width 0 application rbd
>> removed_snaps [1~3]
>> pool 4 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
>> rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0
>> application rgw
>> pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
>> stripe_width 0 application rgw
>> pool 6 'default.rgw.data.root' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
>> stripe_width 0 application rgw
>> pool 7 'default.rgw.gc' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
>> stripe_width 0 application rgw
>> pool 8 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
>> stripe_width 0 application rgw
>> pool 9 'default.rgw.users.uid' replicated size 3 min_size 2

Re: [ceph-users] High apply latency

2018-01-31 Thread Jakub Jaszewski

Is it safe to increase pg_num and pgp_num from 1024 up to 2048 for volumes
and default.rgw.buckets.data pools?
How will it impact cluster behavior? I guess cluster rebalancing will occur
and will take long time considering amount of data we have on it ?

Regards
Jakub



On Wed, Jan 31, 2018 at 1:37 PM, Jakub Jaszewski 
wrote:

> 
> Hi,
>
> I'm wondering why slow requests are being reported mainly when the request
> has been put into the queue for processing by its PG  (queued_for_pg ,
> http://docs.ceph.com/docs/master/rados/troubleshooting/
> troubleshooting-osd/#debugging-slow-request).
> Could it be due too low pg_num/pgp_num ?
>
> It looks that slow requests are mainly addressed to
>  default.rgw.buckets.data (pool id 20) , volumes (pool id 3) and
> default.rgw.buckets.index (pool id 14)
>
> 2018-01-31 12:06:55.899557 osd.59 osd.59 10.212.32.22:6806/4413 38 :
> cluster [WRN] slow request 30.125793 seconds old, received at 2018-01-31
> 12:06:25.773675: osd_op(client.857003.0:126171692 3.a4fec1ad 3.a4fec1ad
> (undecoded) ack+ondisk+write+known_if_redirected e5722) currently
> queued_for_pg
>
> Btw how can I get more human-friendly client information from log entry
> like above ?
>
> Current pg_num/pgp_num
>
> 
> pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
> stripe_width 0 application rbd
> removed_snaps [1~3]
> pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags
> hashpspool stripe_width 0 application rgw
> pool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1
> object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags
> hashpspool stripe_width 4224 application rgw
>
> Usage
>
> GLOBAL:
> SIZE AVAIL RAW USED %RAW USED OBJECTS
> 385T  144T 241T 62.54  31023k
> POOLS:
> NAME ID QUOTA OBJECTS QUOTA BYTES
> USED   %USED MAX AVAIL OBJECTS  DIRTY  READ
> WRITE  RAW USED
> volumes  3  N/A   N/A
> 40351G 70.9116557G 10352314 10109k  2130M
>  2520M 118T
> default.rgw.buckets.index14 N/A   N/A
>  0 016557G  205205   160M
> 27945k0
> default.rgw.buckets.data 20 N/A   N/A
> 79190G 70.5133115G 20865953 20376k   122M
> 113M 116T
>
>
>
> # ceph osd pool ls detail
> pool 0 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 4502 flags hashpspool
> stripe_width 0 application rbd
> pool 1 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
> stripe_width 0 application rbd
> pool 2 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 512 pgp_num 512 last_change 5175 flags hashpspool
> stripe_width 0 application rbd
> removed_snaps [1~7,14~2]
> pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
> stripe_width 0 application rbd
> removed_snaps [1~3]
> pool 4 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width
> 0 application rgw
> pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 6 'default.rgw.data.root' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 7 'default.rgw.gc' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 8 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 9 'default.rgw.users.uid' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 10 'default.rgw.usage' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 11 'default.rgw.users.email' replicated size 3 min_size 2 crush_rule
> 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 owner
> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
> pool 12 'default.rgw.users.keys' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num

Re: [ceph-users] High apply latency

2018-01-31 Thread Jakub Jaszewski


Hi,

I'm wondering why slow requests are being reported mainly when the request
has been put into the queue for processing by its PG  (queued_for_pg ,
http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#debugging-slow-request
).
Could it be due too low pg_num/pgp_num ?

It looks that slow requests are mainly addressed to
 default.rgw.buckets.data (pool id 20) , volumes (pool id 3) and
default.rgw.buckets.index (pool id 14)

2018-01-31 12:06:55.899557 osd.59 osd.59 10.212.32.22:6806/4413 38 :
cluster [WRN] slow request 30.125793 seconds old, received at 2018-01-31
12:06:25.773675: osd_op(client.857003.0:126171692 3.a4fec1ad 3.a4fec1ad
(undecoded) ack+ondisk+write+known_if_redirected e5722) currently
queued_for_pg

Btw how can I get more human-friendly client information from log entry
like above ?

Current pg_num/pgp_num


pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
stripe_width 0 application rbd
removed_snaps [1~3]
pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1
object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags
hashpspool stripe_width 4224 application rgw

Usage

GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
385T  144T 241T 62.54  31023k
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES
  USED   %USED MAX AVAIL OBJECTS  DIRTY  READ
WRITE  RAW USED
volumes  3  N/A   N/A
  40351G 70.9116557G 10352314 10109k  2130M
 2520M 118T
default.rgw.buckets.index14 N/A   N/A
   0 016557G  205205   160M
27945k0
default.rgw.buckets.data 20 N/A   N/A
  79190G 70.5133115G 20865953 20376k   122M
113M 116T



# ceph osd pool ls detail
pool 0 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins
pg_num 64 pgp_num 64 last_change 4502 flags hashpspool stripe_width 0
application rbd
pool 1 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins
pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool stripe_width 0
application rbd
pool 2 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 512 pgp_num 512 last_change 5175 flags hashpspool
stripe_width 0 application rbd
removed_snaps [1~7,14~2]
pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
stripe_width 0 application rbd
removed_snaps [1~3]
pool 4 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width
0 application rgw
pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 6 'default.rgw.data.root' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 7 'default.rgw.gc' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 8 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 9 'default.rgw.users.uid' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 10 'default.rgw.usage' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 11 'default.rgw.users.email' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 owner
18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 12 'default.rgw.users.keys' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 owner
18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 13 'default.rgw.users.swift' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule
0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
stripe_width 0 application rgw
pool 15 'default.rgw.buckets.data.old' replicated size 3 min_size 2
crush_rule 0

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

Re: [ceph-users] High apply latency

12 matches

Site Navigation

Mail list logo

Footer information