Is it safe to increase pg_num and pgp_num from 1024 up to 2048 for volumes
and default.rgw.buckets.data pools?
How will it impact cluster behavior? I guess cluster rebalancing will occur
and will take long time considering amount of data we have on it ?

Regards
Jakub



On Wed, Jan 31, 2018 at 1:37 PM, Jakub Jaszewski <jaszewski.ja...@gmail.com>
wrote:

> ​​
> Hi,
>
> I'm wondering why slow requests are being reported mainly when the request
> has been put into the queue for processing by its PG  (queued_for_pg ,
> http://docs.ceph.com/docs/master/rados/troubleshooting/
> troubleshooting-osd/#debugging-slow-request).
> Could it be due too low pg_num/pgp_num ?
>
> It looks that slow requests are mainly addressed to
>  default.rgw.buckets.data (pool id 20) , volumes (pool id 3) and
> default.rgw.buckets.index (pool id 14)
>
> 2018-01-31 12:06:55.899557 osd.59 osd.59 10.212.32.22:6806/4413 38 :
> cluster [WRN] slow request 30.125793 seconds old, received at 2018-01-31
> 12:06:25.773675: osd_op(client.857003.0:126171692 3.a4fec1ad 3.a4fec1ad
> (undecoded) ack+ondisk+write+known_if_redirected e5722) currently
> queued_for_pg
>
> Btw how can I get more human-friendly client information from log entry
> like above ?
>
> Current pg_num/pgp_num
>
> ​
> pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
> stripe_width 0 application rbd
> removed_snaps [1~3]
> pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags
> hashpspool stripe_width 0 application rgw
> pool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1
> object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags
> hashpspool stripe_width 4224 application rgw
>
> Usage
>
> GLOBAL:
>     SIZE     AVAIL     RAW USED     %RAW USED     OBJECTS
>     385T      144T         241T         62.54      31023k
> POOLS:
>     NAME                             ID     QUOTA OBJECTS     QUOTA BYTES
>     USED       %USED     MAX AVAIL     OBJECTS      DIRTY      READ
> WRITE      RAW USED
>     volumes                          3      N/A               N/A
>     40351G     70.91        16557G     10352314     10109k      2130M
>  2520M         118T
>     default.rgw.buckets.index        14     N/A               N/A
>          0         0        16557G          205        205       160M
> 27945k            0
>     default.rgw.buckets.data         20     N/A               N/A
>     79190G     70.51        33115G     20865953     20376k       122M
> 113M         116T
>
>
>
> # ceph osd pool ls detail
> pool 0 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 64 pgp_num 64 last_change 4502 flags hashpspool
> stripe_width 0 application rbd
> pool 1 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
> stripe_width 0 application rbd
> pool 2 'images' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 512 pgp_num 512 last_change 5175 flags hashpspool
> stripe_width 0 application rbd
> removed_snaps [1~7,14~2]
> pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool
> stripe_width 0 application rbd
> removed_snaps [1~3]
> pool 4 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width
> 0 application rgw
> pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 6 'default.rgw.data.root' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 7 'default.rgw.gc' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 8 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 9 'default.rgw.users.uid' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 10 'default.rgw.usage' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 11 'default.rgw.users.email' replicated size 3 min_size 2 crush_rule
> 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 owner
> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
> pool 12 'default.rgw.users.keys' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 owner
> 18446744073709551615 flags hashpspool stripe_width 0 application rgw
> pool 13 'default.rgw.users.swift' replicated size 3 min_size 2 crush_rule
> 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags
> hashpspool stripe_width 0 application rgw
> pool 15 'default.rgw.buckets.data.old' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 4502
> flags hashpspool stripe_width 0 application rgw
> pool 16 'default.rgw.buckets.non-ec' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags
> hashpspool stripe_width 0 application rgw
> pool 17 'default.rgw.buckets.extra' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags
> hashpspool stripe_width 0 application rgw
> pool 18 '.rgw.buckets.extra' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool
> stripe_width 0 application rgw
> pool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1
> object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags
> hashpspool stripe_width 4224 application rgw
> pool 21 'benchmark_replicated' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4550 flags
> hashpspool stripe_width 0 application rbd
> removed_snaps [1~3]
> pool 22 'benchmark_erasure_coded' erasure size 9 min_size 7 crush_rule 1
> object_hash rjenkins pg_num 32 pgp_num 32 last_change 4552 flags hashpspool
> stripe_width 24576 application rbd
> removed_snaps [1~3]
>
>
> # ceph df detail
> GLOBAL:
>     SIZE     AVAIL     RAW USED     %RAW USED     OBJECTS
>     385T      144T         241T         62.54      31023k
> POOLS:
>     NAME                             ID     QUOTA OBJECTS     QUOTA BYTES
>    USED       %USED     MAX AVAIL     OBJECTS      DIRTY      READ
>  WRITE      RAW USED
>     rbd                              0      N/A               N/A
>         0         0        16557G            0          0          1
>  134k            0
>     vms                              1      N/A               N/A
>         0         0        16557G            0          0          0
>   0            0
>     images                           2      N/A               N/A
>     7659M      0.05        16557G         1022       1022      51247
>  5668       22977M
>     volumes                          3      N/A               N/A
>    40351G     70.91        16557G     10352314     10109k      2130M
> 2520M         118T
>     .rgw.root                        4      N/A               N/A
>      1588         0        16557G            4          4         90
>   4         4764
>     default.rgw.control              5      N/A               N/A
>         0         0        16557G            8          8          0
>   0            0
>     default.rgw.data.root            6      N/A               N/A
>     93943         0        16557G          336        336       239k
>  6393         275k
>     default.rgw.gc                   7      N/A               N/A
>         0         0        16557G           32         32      1773M
> 5281k            0
>     default.rgw.log                  8      N/A               N/A
>         0         0        16557G          185        185     22404k
>  14936k            0
>     default.rgw.users.uid            9      N/A               N/A
>      3815         0        16557G           15         15       187k
> 53303        11445
>     default.rgw.usage                10     N/A               N/A
>         0         0        16557G            7          7       278k
>  556k            0
>     default.rgw.users.email          11     N/A               N/A
>        58         0        16557G            3          3          0
>   3          174
>     default.rgw.users.keys           12     N/A               N/A
>       177         0        16557G           10         10        262
>  22          531
>     default.rgw.users.swift          13     N/A               N/A
>        40         0        16557G            3          3          0
>   3          120
>     default.rgw.buckets.index        14     N/A               N/A
>         0         0        16557G          205        205       160M
>  27945k            0
>     default.rgw.buckets.data.old     15     N/A               N/A
>      668G      3.88        16557G       180867       176k       707k
> 2318k        2004G
>     default.rgw.buckets.non-ec       16     N/A               N/A
>         0         0        16557G          114        114      17960
> 12024            0
>     default.rgw.buckets.extra        17     N/A               N/A
>         0         0        16557G            0          0          0
>   0            0
>     .rgw.buckets.extra               18     N/A               N/A
>         0         0        16557G            0          0          0
>   0            0
>     default.rgw.buckets.data         20     N/A               N/A
>    79190G     70.51        33115G     20865953     20376k       122M
>  113M         116T
>     benchmark_replicated             21     N/A               N/A
>     1415G      7.88        16557G       363800       355k      1338k
> 1251k        4247G
>     benchmark_erasure_coded          22     N/A               N/A
>    11057M      0.03        33115G         2761       2761        398
>  5520       16586M
>
>
> Thanks
> Jakub
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to