I have only one ruleset number 0 and all pools use it. My crushmap is very
simple:
--------------------------------------------------------------------------
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
# devices
device 0 osd.0
device 1 osd.1
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root
# buckets
host storage {
id -2 # do not change unnecessarily
# weight 1.580
alg straw
hash 0 # rjenkins1
item osd.0 weight 1.000
item osd.1 weight 0.750
}
root default {
id -1 # do not change unnecessarily
# weight 1.580
alg straw
hash 0 # rjenkins1
item storage weight 1.750
}
# rules
rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type osd
step emit
}
# end crush map
--------------------------------------------------------------------------
I expect that Ceph will create replicas between 2 OSDs unless the second OSD
becomes full. After that all new objects added to the storage should be
„degradated“ because there is no space to create second replica.
But I get degradated immediately. This is wierd.
Vadim.
From: ceph-users [mailto:[email protected]] On Behalf Of
Vincenzo Pii
Sent: Friday, June 06, 2014 12:34 PM
To: [email protected]
Subject: Re: [ceph-users] Hard drives of different sizes.
Hi Vadim,
Is every pool also using your custom crush_ruleset ("step chooseleaf firstn 0
type osd")?
Otherwise Ceph will use the default rule to replicate data on separate hosts,
which, in your case of a single host, cannot work.
You can check it with
ceph osd dump --format=json-pretty
And in case apply the rule with
ceph osd pool set <pool_name> crush_ruleset <rulesetId>
And you can check your custom rulseteid with
ceph osd crush dump
Hope this helps!
Regards,
Vincenzo.
2014-06-06 8:24 GMT+02:00 Vadim Kimlaychuk
<[email protected]<mailto:[email protected]>>:
Michael, indeed I have pool size = 3. I changed it to 2. After that I have
recompiled crush map to reflect different sizes of hard drives and put 1.0 to
1Tb drive and 0.75 for 750Gb.
Now I have all my PG-s at status "active". It should be „active+clean“ isn’t
it ?
I put object into the cluster and have
health HEALTH_WARN 192 pgs stuck unclean; recovery 1/2 objects degraded
(50.000%)
monmap e1: 1 mons at
{storage=172.16.3.2:6789/0<http://172.16.3.2:6789/0>}, election epoch 2, quorum
0 storage
osdmap e19: 2 osds: 2 up, 2 in
pgmap v42: 192 pgs, 3 pools, 414 bytes data, 1 objects
75584 kB used, 1619 GB / 1619 GB avail
1/2 objects degraded (50.000%)
192 active
Does that mean object is stored at wrong place? Or set up is still incomplete?
Thanks.
-----Original Message-----
From: ceph-users
[mailto:[email protected]<mailto:[email protected]>]
On Behalf Of Jeremy Hanmer
Sent: Thursday, June 05, 2014 9:41 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: [ceph-users] Hard drives of different sizes.
You'll also want to change the crush weights of your OSDs to reflect the
different sizes so that the smaller disks don't get filled up prematurely. See
"weighting bucket items" here:
http://ceph.com/docs/master/rados/operations/crush-map/
On Thu, Jun 5, 2014 at 10:14 AM, Michael
<[email protected]<mailto:[email protected]>> wrote:
> ceph osd dump | grep size
>
> Check that all pools are size 2, min size 2 or 1.
>
> If not you can change on the fly with:
> ceph osd pool set #poolname size/min_size #size
>
> See docs http://ceph.com/docs/master/rados/operations/pools/ for
> alterations to pool attributes.
>
> -Michael
>
>
> On 05/06/2014 17:29, Vadim Kimlaychuk wrote:
>>
>> ____________________________
>>
>> I have
>> osd pool default size = 2
>> at my ceph.conf. Shouldn' it tell ceph to use 2 OSDs ? Or it is
>> somewhere in CRUSH map?
>>
>> Vadim
>> ____________
>> From: Christian Balzer [[email protected]<mailto:[email protected]>]
>> Sent: Thursday, June 05, 2014 18:26
>> To: Vadim Kimlaychuk
>> Cc: [email protected]<mailto:[email protected]>
>> Subject: Re: [ceph-users] Hard drives of different sizes.
>>
>> Hello,
>>
>> On Thu, 5 Jun 2014 14:11:47 +0000 Vadim Kimlaychuk wrote:
>>
>>> Hello,
>>>
>>> Probably this is anti-pattern, but I have to get answer
>>> how this will work / not work. Input:
>>> I have single host for tests with ceph 0.80.1 and 2 OSD:
>>> OSD.0 – 1000 Gb
>>> OSD.1 – 750 Gb
>>>
>>> Recompiled CRUSH map to set „step chooseleaf firstn 0
>>> type osd“
>>>
>> You got it half right.
>>
>> Version .8x aka Firefly has a default replication of 3, so you would
>> need
>> 3 OSDs at least.
>>
>> Christian
>>>
>>> I am expecting, that part of PG-s will have status
>>> „active+clean“ (with size of ~750Gb) another part of PG-s will have
>>> „active+degradated“ (with size of ~250Gb), because there is not
>>> enough place to replicate data on the second OSD.
>>>
>>> Instead I have ALL PG-s „active + degradated“
>>>
>>> Output:
>>> health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean
>>> monmap e1: 1 mons at
>>> {storage=172.16.3.2:6789/0<http://172.16.3.2:6789/0>}, election
>>> epoch 2, quorum 0 storage osdmap e15: 2 osds: 2 up, 2 in
>>> pgmap v29: 192 pgs, 3 pools, 0 bytes data, 0 objects
>>> 71496 kB used, 1619 GB / 1619 GB avail
>>> 192 active+degraded
>>>
>>> What is the logic behind this?? Can I use different
>>> hard drives successfully? If yes – how?
>>>
>>> Thank you for explanation,
>>>
>>> Vadim
>>>
>>
>> --
>> Christian Balzer Network/Systems Engineer
>> [email protected]<mailto:[email protected]> Global OnLine Japan/Fusion
>> Communications
>> http://www.gol.com/
>> _______________________________________________
>> ceph-users mailing list
>> [email protected]<mailto:[email protected]>
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> _______________________________________________
> ceph-users mailing list
> [email protected]<mailto:[email protected]>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected]<mailto:[email protected]>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Vincenzo Pii
Researcher, InIT Cloud Computing Lab
Zurich University of Applied Sciences (ZHAW)
http://www.cloudcomp.ch/
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
