I have a question about file-systems with replicated an non replicated data.
We have a file-system where metadata is set to copies=2 and data copies=2, we
then use a placement policy to selectively replicate some data only once based
on file-set. We also place the non-replicated data into a specific pool
(6tnlsas) to ensure we know where it is placed.
My understanding was that in doing this, if we took the disks with the non
replicated data offline, we’d still have the FS available for users as the
metadata is replicated. Sure accessing a non-replicated data file would give an
IO error, but the rest of the FS should be up.
We had a situation today where we wanted to take stg01 offline today, so tried
using mmchdisk stop -d …. Once we got to about disk stg01-01_12_12, GPFS would
refuse to stop any more disks and complain about too many disks, similarly if
we shutdown the NSD servers hosting the disks, the filesystem would have an
SGPanic and force unmount.
First, am I correct in thinking that a FS with non-replicated data, but
replicated metadata should still be accessible (not the non-replicated data)
when the LUNS hosting it are down?
If so, any suggestions why my FS is panic-ing when we take down the one set of
disks?
I thought at first we had some non-replicated metadata, tried a mmrestripefs -R
–metadata-only to force it to ensure 2 replicas, but this didn’t help.
Running 5.0.0.2 on the NSD server nodes.
(First time we went round this we didn’t have a FS descriptor disk, but you can
see below that we added this)
Thanks
Simon
[root@nsd01 ~]# mmlsdisk castles -L
disk driver sector failure holds holds
storage
name type size group metadata data status
availability disk id pool remarks
------------ -------- ------ ----------- -------- ----- -------------
------------ ------- ------------ ---------
CASTLES_GPFS_DESCONLY01 nsd 512 310 no no ready
up 1 system desc
stg01-01_3_3 nsd 4096 210 no yes ready down
4 6tnlsas
stg01-01_4_4 nsd 4096 210 no yes ready down
5 6tnlsas
stg01-01_5_5 nsd 4096 210 no yes ready down
6 6tnlsas
stg01-01_6_6 nsd 4096 210 no yes ready down
7 6tnlsas
stg01-01_7_7 nsd 4096 210 no yes ready down
8 6tnlsas
stg01-01_8_8 nsd 4096 210 no yes ready down
9 6tnlsas
stg01-01_9_9 nsd 4096 210 no yes ready down
10 6tnlsas
stg01-01_10_10 nsd 4096 210 no yes ready down
11 6tnlsas
stg01-01_11_11 nsd 4096 210 no yes ready down
12 6tnlsas
stg01-01_12_12 nsd 4096 210 no yes ready down
13 6tnlsas
stg01-01_13_13 nsd 4096 210 no yes ready down
14 6tnlsas
stg01-01_14_14 nsd 4096 210 no yes ready down
15 6tnlsas
stg01-01_15_15 nsd 4096 210 no yes ready down
16 6tnlsas
stg01-01_16_16 nsd 4096 210 no yes ready down
17 6tnlsas
stg01-01_17_17 nsd 4096 210 no yes ready down
18 6tnlsas
stg01-01_18_18 nsd 4096 210 no yes ready down
19 6tnlsas
stg01-01_19_19 nsd 4096 210 no yes ready down
20 6tnlsas
stg01-01_20_20 nsd 4096 210 no yes ready down
21 6tnlsas
stg01-01_21_21 nsd 4096 210 no yes ready down
22 6tnlsas
stg01-01_ssd_54_54 nsd 4096 210 yes no ready
down 23 system
stg01-01_ssd_56_56 nsd 4096 210 yes no ready
down 24 system
stg02-01_0_0 nsd 4096 110 no yes ready up
25 6tnlsas
stg02-01_1_1 nsd 4096 110 no yes ready up
26 6tnlsas
stg02-01_2_2 nsd 4096 110 no yes ready up
27 6tnlsas
stg02-01_3_3 nsd 4096 110 no yes ready up
28 6tnlsas
stg02-01_4_4 nsd 4096 110 no yes ready up
29 6tnlsas
stg02-01_5_5 nsd 4096 110 no yes ready up
30 6tnlsas
stg02-01_6_6 nsd 4096 110 no yes ready up
31 6tnlsas
stg02-01_7_7 nsd 4096 110 no yes ready up
32 6tnlsas
stg02-01_8_8 nsd 4096 110 no yes ready up
33 6tnlsas
stg02-01_9_9 nsd 4096 110 no yes ready up
34 6tnlsas
stg02-01_10_10 nsd 4096 110 no yes ready up
35 6tnlsas
stg02-01_11_11 nsd 4096 110 no yes ready up
36 6tnlsas
stg02-01_12_12 nsd 4096 110 no yes ready up
37 6tnlsas
stg02-01_13_13 nsd 4096 110 no yes ready up
38 6tnlsas
stg02-01_14_14 nsd 4096 110 no yes ready up
39 6tnlsas
stg02-01_15_15 nsd 4096 110 no yes ready up
40 6tnlsas
stg02-01_16_16 nsd 4096 110 no yes ready up
41 6tnlsas
stg02-01_17_17 nsd 4096 110 no yes ready up
42 6tnlsas
stg02-01_18_18 nsd 4096 110 no yes ready up
43 6tnlsas
stg02-01_19_19 nsd 4096 110 no yes ready up
44 6tnlsas
stg02-01_20_20 nsd 4096 110 no yes ready up
45 6tnlsas
stg02-01_21_21 nsd 4096 110 no yes ready up
46 6tnlsas
stg02-01_ssd_22_22 nsd 4096 110 yes no ready up
47 system desc
stg02-01_ssd_23_23 nsd 4096 110 yes no ready up
48 system
stg02-01_ssd_24_24 nsd 4096 110 yes no ready up
49 system
stg02-01_ssd_25_25 nsd 4096 110 yes no ready up
50 system
stg01-01_22_22 nsd 4096 210 no yes ready up
51 6tnlsasnonrepl desc
stg01-01_23_23 nsd 4096 210 no yes ready up
52 6tnlsasnonrepl
stg01-01_24_24 nsd 4096 210 no yes ready up
53 6tnlsasnonrepl
stg01-01_25_25 nsd 4096 210 no yes ready up
54 6tnlsasnonrepl
stg01-01_26_26 nsd 4096 210 no yes ready up
55 6tnlsasnonrepl
stg01-01_27_27 nsd 4096 210 no yes ready up
56 6tnlsasnonrepl
stg01-01_31_31 nsd 4096 210 no yes ready up
58 6tnlsasnonrepl
stg01-01_32_32 nsd 4096 210 no yes ready up
59 6tnlsasnonrepl
stg01-01_33_33 nsd 4096 210 no yes ready up
60 6tnlsasnonrepl
stg01-01_34_34 nsd 4096 210 no yes ready up
61 6tnlsasnonrepl
stg01-01_35_35 nsd 4096 210 no yes ready up
62 6tnlsasnonrepl
stg01-01_36_36 nsd 4096 210 no yes ready up
63 6tnlsasnonrepl
stg01-01_37_37 nsd 4096 210 no yes ready up
64 6tnlsasnonrepl
stg01-01_38_38 nsd 4096 210 no yes ready up
65 6tnlsasnonrepl
stg01-01_39_39 nsd 4096 210 no yes ready up
66 6tnlsasnonrepl
stg01-01_40_40 nsd 4096 210 no yes ready up
67 6tnlsasnonrepl
stg01-01_41_41 nsd 4096 210 no yes ready up
68 6tnlsasnonrepl
stg01-01_42_42 nsd 4096 210 no yes ready up
69 6tnlsasnonrepl
stg01-01_43_43 nsd 4096 210 no yes ready up
70 6tnlsasnonrepl
stg01-01_44_44 nsd 4096 210 no yes ready up
71 6tnlsasnonrepl
stg01-01_45_45 nsd 4096 210 no yes ready up
72 6tnlsasnonrepl
stg01-01_46_46 nsd 4096 210 no yes ready up
73 6tnlsasnonrepl
stg01-01_47_47 nsd 4096 210 no yes ready up
74 6tnlsasnonrepl
stg01-01_48_48 nsd 4096 210 no yes ready up
75 6tnlsasnonrepl
stg01-01_49_49 nsd 4096 210 no yes ready up
76 6tnlsasnonrepl
stg01-01_50_50 nsd 4096 210 no yes ready up
77 6tnlsasnonrepl
stg01-01_51_51 nsd 4096 210 no yes ready up
78 6tnlsasnonrepl
Number of quorum disks: 3
Read quorum value: 2
Write quorum value: 2
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
