Hi everyone,
Beginner with Ceph, i’m looking for a way to do a 3-way replication between 2
datacenters as mention in ceph docs (but not describe).
My goal is to keep access to the data (at least read-only access) even when the
link between the 2 datacenters is cut and make sure at least one copy of the
data exists in each datacenter.
I’m not sure how to implement such 3-way replication. With a rule?
Based on the CEPH docs, I think of a rule:
rule 3-way-replication_with_2_DC {
ruleset 1
type replicated
min_size 2
max_size 3
step take DC-1
step choose firstn 1 type host
step chooseleaf firstn 1 type osd
step emit
step take DC-2
step choose firstn 1 type host
step chooseleaf firstn 1 type osd
step emit
step take default
step choose firstn 1 type host
step chooseleaf firstn 1 type osd
step emit
}
but what should happen if the link between the 2 datacenters is cut? If someone
has a better solution, I interested by any resources about it (examples, …).
The default rule (see below) keep the pool working when we mark each node of
DC-2 as down (typically maintenance) but if we shut the link down between the 2
datacenters, the pool/rbd hangs (frozen writing dd tool for example).
Does anyone have some insight on how to setup a 3-way replication between 2
datacenters?
Thanks in advance for any advice on the topic.
Current situation:
Mons : host-1, host-2, host-4
Quick network topology:
USERS NETWORK
|
2x10G
|
DC-1-SWITCH <——— 40G ——> DC-2-SWITCH
| | | | | |
host-1 _| | | host-4 _| | |
host-2 ___| | host-5 ___| |
host-3 _____| host-6 _____|
crushmap :
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 147.33325 root default
-20 73.66663 datacenter DC-1
-15 73.66663 rack DC-1-RACK-1
-9 24.55554 host host-1
27 hdd 2.72839 osd.27 up 1.00000 1.00000
28 hdd 2.72839 osd.28 up 1.00000 1.00000
29 hdd 2.72839 osd.29 up 1.00000 1.00000
30 hdd 2.72839 osd.30 up 1.00000 1.00000
31 hdd 2.72839 osd.31 up 1.00000 1.00000
32 hdd 2.72839 osd.32 up 1.00000 1.00000
33 hdd 2.72839 osd.33 up 1.00000 1.00000
34 hdd 2.72839 osd.34 up 1.00000 1.00000
36 hdd 2.72839 osd.36 up 1.00000 1.00000
-11 24.55554 host host-2
35 hdd 2.72839 osd.35 up 1.00000 1.00000
37 hdd 2.72839 osd.37 up 1.00000 1.00000
38 hdd 2.72839 osd.38 up 1.00000 1.00000
39 hdd 2.72839 osd.39 up 1.00000 1.00000
40 hdd 2.72839 osd.40 up 1.00000 1.00000
41 hdd 2.72839 osd.41 up 1.00000 1.00000
42 hdd 2.72839 osd.42 up 1.00000 1.00000
43 hdd 2.72839 osd.43 up 1.00000 1.00000
46 hdd 2.72839 osd.46 up 1.00000 1.00000
-13 24.55554 host host-3
44 hdd 2.72839 osd.44 up 1.00000 1.00000
45 hdd 2.72839 osd.45 up 1.00000 1.00000
47 hdd 2.72839 osd.47 up 1.00000 1.00000
48 hdd 2.72839 osd.48 up 1.00000 1.00000
49 hdd 2.72839 osd.49 up 1.00000 1.00000
50 hdd 2.72839 osd.50 up 1.00000 1.00000
51 hdd 2.72839 osd.51 up 1.00000 1.00000
52 hdd 2.72839 osd.52 up 1.00000 1.00000
53 hdd 2.72839 osd.53 up 1.00000 1.00000
-19 73.66663 datacenter DC-2
-16 73.66663 rack DC-2-RACK-1
-3 24.55554 host host-4
0 hdd 2.72839 osd.0 up 1.00000 1.00000
1 hdd 2.72839 osd.1 up 1.00000 1.00000
2 hdd 2.72839 osd.2 up 1.00000 1.00000
3 hdd 2.72839 osd.3 up 1.00000 1.00000
4 hdd 2.72839 osd.4 up 1.00000 1.00000
5 hdd 2.72839 osd.5 up 1.00000 1.00000
6 hdd 2.72839 osd.6 up 1.00000 1.00000
7 hdd 2.72839 osd.7 up 1.00000 1.00000
8 hdd 2.72839 osd.8 up 1.00000 1.00000
-5 24.55554 host host-5
9 hdd 2.72839 osd.9 up 1.00000 1.00000
10 hdd 2.72839 osd.10 up 1.00000 1.00000
11 hdd 2.72839 osd.11 up 1.00000 1.00000
12 hdd 2.72839 osd.12 up 1.00000 1.00000
13 hdd 2.72839 osd.13 up 1.00000 1.00000
14 hdd 2.72839 osd.14 up 1.00000 1.00000
15 hdd 2.72839 osd.15 up 1.00000 1.00000
16 hdd 2.72839 osd.16 up 1.00000 1.00000
18 hdd 2.72839 osd.18 up 1.00000 1.00000
-7 24.55554 host host-6
19 hdd 2.72839 osd.19 up 1.00000 1.00000
20 hdd 2.72839 osd.20 up 1.00000 1.00000
21 hdd 2.72839 osd.21 up 1.00000 1.00000
22 hdd 2.72839 osd.22 up 1.00000 1.00000
23 hdd 2.72839 osd.23 up 1.00000 1.00000
24 hdd 2.72839 osd.24 up 1.00000 1.00000
25 hdd 2.72839 osd.25 up 1.00000 1.00000
26 hdd 2.72839 osd.26 up 1.00000 1.00000
54 hdd 2.72839 osd.54 up 1.00000 1.00000
current rules (default one) :
# ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
]
Cordialement / Best regards,
Sébastien VIGNERON
CRIANN,
Ingénieur / Engineer
Technopôle du Madrillet
745, avenue de l'Université
76800 Saint-Etienne du Rouvray - France
tél. +33 2 32 91 42 91
fax. +33 2 32 91 42 92
http://www.criann.fr
mailto:[email protected]
support: [email protected]
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
