When you increase the number of PGs, don't just go to the max value.
Step into it.
You'll want to end up around 2048, so do 400 -> 512, wait for it to
finish, -> 1024, wait, -> 2048.
Also remember that you don't need a lot of PGs if you don't have much
data in the pools. My .rgw.buckets pool has 2k PGs, but the the RGW
metadata pools only have a couple MB and 32 PGs each.
*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email [email protected] <mailto:[email protected]>
*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website <http://www.centraldesktop.com/> | Twitter
<http://www.twitter.com/centraldesktop> | Facebook
<http://www.facebook.com/CentralDesktop> | LinkedIn
<http://www.linkedin.com/groups?gid=147417> | Blog
<http://cdblog.centraldesktop.com/>
On 4/18/14 05:04 , Tyler Brekke wrote:
That is rather low, increasing the pg count should help with the data
distribution.
Documentation recommends starting with (100 * (num of osds))
/(replicas) rounded up to the nearest power of two.
https://ceph.com/docs/master/rados/operations/placement-groups/
On Fri, Apr 18, 2014 at 4:54 AM, Kenneth Waegeman
<[email protected] <mailto:[email protected]>> wrote:
----- Message from Tyler Brekke <[email protected]
<mailto:[email protected]>> ---------
Date: Fri, 18 Apr 2014 04:37:26 -0700
From: Tyler Brekke <[email protected]
<mailto:[email protected]>>
Subject: Re: [ceph-users] OSD distribution unequally
To: Dan Van Der Ster <[email protected]
<mailto:[email protected]>>
Cc: Kenneth Waegeman <[email protected]
<mailto:[email protected]>>, ceph-users
<[email protected] <mailto:[email protected]>>
How many placement groups do you have in your pool containing
the data, and
what is the replication level of that pool?
400 pgs per pool, replication factor is 3
Looks like you have too few placement groups which is causing
the data
distribution to be off.
-Tyler
On Fri, Apr 18, 2014 at 4:12 AM, Dan Van Der Ster
<[email protected] <mailto:[email protected]>
wrote:
ceph osd reweight-by-utilization
Is that still in 0.79?
I'd start with reweight-by-utilization 200 and then adjust
that number
down until you get to 120 or so.
Cheers, Dan
On Apr 18, 2014 12:49 PM, Kenneth Waegeman
<[email protected]>
wrote:
Hi,
Some osds of our cluster filled up:
health HEALTH_ERR 1 full osd(s); 4 near full osd(s)
monmap e1: 3 mons at
{ceph001=
10.141.8.180:6789/0,ceph002=10.141.8.181:6789/0,ceph003=10.141.8.182:6789/0
<http://10.141.8.180:6789/0,ceph002=10.141.8.181:6789/0,ceph003=10.141.8.182:6789/0>},
election epoch 96, quorum 0,1,2
ceph001,ceph002,ceph003
mdsmap e93: 1/1/1 up
{0=ceph001.cubone.os=up:active}, 1 up:standby
osdmap e1974: 42 osds: 42 up, 42 in
flags full
pgmap v286626: 1200 pgs, 3 pools, 31096 GB data,
26259 kobjects
94270 GB used, 40874 GB / 131 TB avail
1 active+clean+scrubbing+deep
1199 active+clean
I knew it is never really uniform, but the differences of
the OSDs
seems very big, one OSD has 96% while another only has 48%
usage,
which is about 1,8 TB difference:
/dev/sdc 3.7T 1.9T 1.8T 51% /var/lib/ceph/osd/sdc
/dev/sdd 3.7T 2.5T 1.2T 68% /var/lib/ceph/osd/sdd
/dev/sde 3.7T 2.3T 1.5T 61% /var/lib/ceph/osd/sde
/dev/sdf 3.7T 2.7T 975G 74% /var/lib/ceph/osd/sdf
/dev/sdg 3.7T 3.2T 491G 87% /var/lib/ceph/osd/sdg
/dev/sdh 3.7T 2.0T 1.8T 53% /var/lib/ceph/osd/sdh
/dev/sdi 3.7T 2.3T 1.4T 63% /var/lib/ceph/osd/sdi
/dev/sdj 3.7T 3.4T 303G 92% /var/lib/ceph/osd/sdj
/dev/sdk 3.7T 2.8T 915G 76% /var/lib/ceph/osd/sdk
/dev/sdl 3.7T 1.8T 2.0T 48% /var/lib/ceph/osd/sdl
/dev/sdm 3.7T 2.8T 917G 76% /var/lib/ceph/osd/sdm
/dev/sdn 3.7T 3.5T 186G 96% /var/lib/ceph/osd/sdn
We are running 0.79 (well precisely a patched version of
it with an
MDS fix of another thread:-) )
I remember hearing something about the hashpgpool having
an effect on
it, but I read this was already default enabled on the latest
versions. osd_pool_default_flag_hashpspool has indeed the
value true,
but I don't know how to check this for a specific pool.
Is this behaviour normal? Or what can be wrong?
Thanks!
Kind regards,
Kenneth Waegeman
_______________________________________________
ceph-users mailing list
[email protected] <mailto:[email protected]>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected] <mailto:[email protected]>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
----- End message from Tyler Brekke <[email protected]
<mailto:[email protected]>> -----
--
Met vriendelijke groeten,
Kenneth Waegeman
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
