Re: [PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
Mandi! Alwin Antreich In chel di` si favelave... > The min_size 1 means in a degraded state, ceph serves the data as long as one > copy is available. Yes, i know. > If they shutdown cleanly, then it shouldn't be a problem, as far as I have > tested it myself. [...] > I don't see this behavior on our test cluster, when we shutdown all hosts and > start them up at a later time. Boh. It's strange... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/25/index.php/component/k2/item/123 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
Hi Marco, On 11/29/2016 03:05 PM, Marco Gaiarin wrote: > Mandi! Alwin Antreich > In chel di` si favelave... > >> What does the following command give you? >> ceph osd pool get min_size > > root@capitanamerica:~# ceph osd pool get DATA min_size > min_size: 1 > root@capitanamerica:~# ceph osd pool get VM min_size > min_size: 1 > root@capitanamerica:~# ceph osd pool get LXC min_size > min_size: 1 The min_size 1 means in a degraded state, ceph serves the data as long as one copy is available. > > >> As a general thought, a HA cluster would be always running, so the event >> that you shutdown all nodes is unlikely to >> happen. > > Ahem, not so unlikely... we have UPSes but not diesel generators... ;-( If they shutdown cleanly, then it shouldn't be a problem, as far as I have tested it myself. > > >> And if you decide to shutdown all nodes, then a couple of minutes should be >> ok to get everything running again >> and ceph needs its time, till all services are running. > > This is not the case. I've started the nodes one by one... > I don't see this behavior on our test cluster, when we shutdown all hosts and start them up at a later time. -- Cheers, Alwin ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
Mandi! Alwin Antreich In chel di` si favelave... > What does the following command give you? > ceph osd pool get min_size root@capitanamerica:~# ceph osd pool get DATA min_size min_size: 1 root@capitanamerica:~# ceph osd pool get VM min_size min_size: 1 root@capitanamerica:~# ceph osd pool get LXC min_size min_size: 1 > As a general thought, a HA cluster would be always running, so the event that > you shutdown all nodes is unlikely to > happen. Ahem, not so unlikely... we have UPSes but not diesel generators... ;-( > And if you decide to shutdown all nodes, then a couple of minutes should be > ok to get everything running again > and ceph needs its time, till all services are running. This is not the case. I've started the nodes one by one... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/25/index.php/component/k2/item/123 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
Hi Marco,
On 11/29/2016 12:17 PM, Marco Gaiarin wrote:
> Mandi! Alwin Antreich
> In chel di` si favelave...
>
>> May you please show us the logs?
>
> Ok, i'm here. With the log.
>
> A bit of legenda: 10.27.251.7 and 10.27.251.8 are the 'ceph' nodes
> (mon+osd); 10.27.251.11 and 10.27.251.12 are the 'cpu' nodes (only
> mon). In order, mon.0, mon.1, mon.2 and mon.3.
>
> These are the logs of 10.27.251.7 (mon.0); Seems to me that ceph logs
> are all similar, so i hope these suffices.
>
>
> I've started my activity at 15.00, but before take down all the stuff
> i've P2V my last server, my Asterisk PBX box. Clearly, cluster worked:
>
> [...]
> 2016-11-26 16:45:51.900445 osd.4 10.27.251.8:6804/3442 5016 : cluster [INF]
> 3.68 scrub starts
> 2016-11-26 16:45:52.047932 osd.4 10.27.251.8:6804/3442 5017 : cluster [INF]
> 3.68 scrub ok
> 2016-11-26 16:45:52.741334 mon.0 10.27.251.7:6789/0 2317313 : cluster [INF]
> pgmap v2410312: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB
> / 9310 GB avail; 20533 B/s rd, 945 kB/s wr, 127 op/s
> 2016-11-26 16:45:54.825603 mon.0 10.27.251.7:6789/0 2317314 : cluster [INF]
> pgmap v2410313: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB
> / 9310 GB avail; 261 kB/s wr, 7 op/s
> [...]
> 2016-11-26 16:47:52.741749 mon.0 10.27.251.7:6789/0 2317382 : cluster [INF]
> pgmap v2410381: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB
> / 9310 GB avail; 11563 B/s rd, 687 kB/s wr, 124 op/s
> 2016-11-26 16:47:55.002485 mon.0 10.27.251.7:6789/0 2317383 : cluster [INF]
> pgmap v2410382: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB
> / 9310 GB avail; 922 kB/s rd, 292 kB/s wr, 28 op/s
>
>
> Finished the P2V, i've started to power off the cluster, starting from
> the cpu nodes. After powering down a node, i've realized that i need it
> to do another thing, so i've re-powered on. ;-)
>
> 2016-11-26 16:48:05.018514 mon.1 10.27.251.8:6789/0 129 : cluster [INF] mon.1
> calling new monitor election
> 2016-11-26 16:48:05.031761 mon.2 10.27.251.11:6789/0 120 : cluster [INF]
> mon.2 calling new monitor election
> 2016-11-26 16:48:05.053262 mon.0 10.27.251.7:6789/0 2317384 : cluster [INF]
> mon.0 calling new monitor election
> 2016-11-26 16:48:10.091773 mon.0 10.27.251.7:6789/0 2317385 : cluster [INF]
> mon.0@0 won leader election with quorum 0,1,2
> 2016-11-26 16:48:10.104535 mon.0 10.27.251.7:6789/0 2317386 : cluster [INF]
> HEALTH_WARN; 1 mons down, quorum 0,1,2 0,1,2
> 2016-11-26 16:48:10.143625 mon.0 10.27.251.7:6789/0 2317387 : cluster [INF]
> monmap e4: 4 mons at
> {0=10.27.251.7:6789/0,1=10.27.251.8:6789/0,2=10.27.251.11:6789/0,3=10.27.251.12:6789/0}
> 2016-11-26 16:48:10.143731 mon.0 10.27.251.7:6789/0 2317388 : cluster [INF]
> pgmap v2410382: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB
> / 9310 GB avail; 922 kB/s rd, 292 kB/s wr, 28 op/s
> 2016-11-26 16:48:10.144828 mon.0 10.27.251.7:6789/0 2317389 : cluster [INF]
> mdsmap e1: 0/0/0 up
> 2016-11-26 16:48:10.148407 mon.0 10.27.251.7:6789/0 2317390 : cluster [INF]
> osdmap e98: 6 osds: 6 up, 6 in
> 2016-11-26 16:48:11.208968 mon.0 10.27.251.7:6789/0 2317391 : cluster [INF]
> pgmap v2410383: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB
> / 9310 GB avail; 2174 kB/s rd, 646 kB/s wr, 130 op/s
> 2016-11-26 16:48:13.309644 mon.0 10.27.251.7:6789/0 2317392 : cluster [INF]
> pgmap v2410384: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB
> / 9310 GB avail; 2210 kB/s rd, 652 kB/s wr, 135 op/s
> [...]
> 2016-11-26 16:50:04.665220 mon.0 10.27.251.7:6789/0 2317466 : cluster [INF]
> pgmap v2410458: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB
> / 9310 GB avail; 2579 B/s rd, 23217 B/s wr, 5 op/s
> 2016-11-26 16:50:05.707271 mon.0 10.27.251.7:6789/0 2317467 : cluster [INF]
> pgmap v2410459: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB
> / 9310 GB avail; 157 kB/s rd, 445 kB/s wr, 82 op/s
> 2016-11-26 16:50:16.786716 mon.1 10.27.251.8:6789/0 130 : cluster [INF] mon.1
> calling new monitor election
> 2016-11-26 16:50:16.815156 mon.0 10.27.251.7:6789/0 2317468 : cluster [INF]
> mon.0 calling new monitor election
> 2016-11-26 16:52:51.536024 osd.0 10.27.251.7:6800/3166 7755 : cluster [INF]
> 1.e8 scrub starts
> 2016-11-26 16:52:53.771169 osd.0 10.27.251.7:6800/3166 7756 : cluster [INF]
> 1.e8 scrub ok
> 2016-11-26 16:54:34.558607 osd.0 10.27.251.7:6800/3166 7757 : cluster [INF]
> 1.ed scrub starts
> 2016-11-26 16:54:36.682207 osd.0 10.27.251.7:6800/3166 7758 : cluster [INF]
> 1.ed scrub ok
> 2016-11-26 16:57:07.816187 mon.3 10.27.251.12:6789/0 1 : cluster [INF] mon.3
> calling new monitor election
> 2016-11-26 16:57:13.242951 mon.0 10.27.251.7:6789/0 2317469 : cluster [INF]
> mon.0@0 won leader election with quorum 0,1,3
> 2016-11-26 16:57:13.252424 mon.0 10.27.251.7:6789/0 2317470 : cluster [INF]
> HEALTH_WARN; 1 mons down, quorum 0,1,3 0,1,3
> 2016-11-26 16:57:13.253143 mon.0
Re: [PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
Mandi! Alwin Antreich
In chel di` si favelave...
> May you please show us the logs?
Ok, i'm here. With the log.
A bit of legenda: 10.27.251.7 and 10.27.251.8 are the 'ceph' nodes
(mon+osd); 10.27.251.11 and 10.27.251.12 are the 'cpu' nodes (only
mon). In order, mon.0, mon.1, mon.2 and mon.3.
These are the logs of 10.27.251.7 (mon.0); Seems to me that ceph logs
are all similar, so i hope these suffices.
I've started my activity at 15.00, but before take down all the stuff
i've P2V my last server, my Asterisk PBX box. Clearly, cluster worked:
[...]
2016-11-26 16:45:51.900445 osd.4 10.27.251.8:6804/3442 5016 : cluster [INF]
3.68 scrub starts
2016-11-26 16:45:52.047932 osd.4 10.27.251.8:6804/3442 5017 : cluster [INF]
3.68 scrub ok
2016-11-26 16:45:52.741334 mon.0 10.27.251.7:6789/0 2317313 : cluster [INF]
pgmap v2410312: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB /
9310 GB avail; 20533 B/s rd, 945 kB/s wr, 127 op/s
2016-11-26 16:45:54.825603 mon.0 10.27.251.7:6789/0 2317314 : cluster [INF]
pgmap v2410313: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB /
9310 GB avail; 261 kB/s wr, 7 op/s
[...]
2016-11-26 16:47:52.741749 mon.0 10.27.251.7:6789/0 2317382 : cluster [INF]
pgmap v2410381: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB /
9310 GB avail; 11563 B/s rd, 687 kB/s wr, 124 op/s
2016-11-26 16:47:55.002485 mon.0 10.27.251.7:6789/0 2317383 : cluster [INF]
pgmap v2410382: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB /
9310 GB avail; 922 kB/s rd, 292 kB/s wr, 28 op/s
Finished the P2V, i've started to power off the cluster, starting from
the cpu nodes. After powering down a node, i've realized that i need it
to do another thing, so i've re-powered on. ;-)
2016-11-26 16:48:05.018514 mon.1 10.27.251.8:6789/0 129 : cluster [INF] mon.1
calling new monitor election
2016-11-26 16:48:05.031761 mon.2 10.27.251.11:6789/0 120 : cluster [INF] mon.2
calling new monitor election
2016-11-26 16:48:05.053262 mon.0 10.27.251.7:6789/0 2317384 : cluster [INF]
mon.0 calling new monitor election
2016-11-26 16:48:10.091773 mon.0 10.27.251.7:6789/0 2317385 : cluster [INF]
mon.0@0 won leader election with quorum 0,1,2
2016-11-26 16:48:10.104535 mon.0 10.27.251.7:6789/0 2317386 : cluster [INF]
HEALTH_WARN; 1 mons down, quorum 0,1,2 0,1,2
2016-11-26 16:48:10.143625 mon.0 10.27.251.7:6789/0 2317387 : cluster [INF]
monmap e4: 4 mons at
{0=10.27.251.7:6789/0,1=10.27.251.8:6789/0,2=10.27.251.11:6789/0,3=10.27.251.12:6789/0}
2016-11-26 16:48:10.143731 mon.0 10.27.251.7:6789/0 2317388 : cluster [INF]
pgmap v2410382: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB /
9310 GB avail; 922 kB/s rd, 292 kB/s wr, 28 op/s
2016-11-26 16:48:10.144828 mon.0 10.27.251.7:6789/0 2317389 : cluster [INF]
mdsmap e1: 0/0/0 up
2016-11-26 16:48:10.148407 mon.0 10.27.251.7:6789/0 2317390 : cluster [INF]
osdmap e98: 6 osds: 6 up, 6 in
2016-11-26 16:48:11.208968 mon.0 10.27.251.7:6789/0 2317391 : cluster [INF]
pgmap v2410383: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB /
9310 GB avail; 2174 kB/s rd, 646 kB/s wr, 130 op/s
2016-11-26 16:48:13.309644 mon.0 10.27.251.7:6789/0 2317392 : cluster [INF]
pgmap v2410384: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB /
9310 GB avail; 2210 kB/s rd, 652 kB/s wr, 135 op/s
[...]
2016-11-26 16:50:04.665220 mon.0 10.27.251.7:6789/0 2317466 : cluster [INF]
pgmap v2410458: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB /
9310 GB avail; 2579 B/s rd, 23217 B/s wr, 5 op/s
2016-11-26 16:50:05.707271 mon.0 10.27.251.7:6789/0 2317467 : cluster [INF]
pgmap v2410459: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB /
9310 GB avail; 157 kB/s rd, 445 kB/s wr, 82 op/s
2016-11-26 16:50:16.786716 mon.1 10.27.251.8:6789/0 130 : cluster [INF] mon.1
calling new monitor election
2016-11-26 16:50:16.815156 mon.0 10.27.251.7:6789/0 2317468 : cluster [INF]
mon.0 calling new monitor election
2016-11-26 16:52:51.536024 osd.0 10.27.251.7:6800/3166 7755 : cluster [INF]
1.e8 scrub starts
2016-11-26 16:52:53.771169 osd.0 10.27.251.7:6800/3166 7756 : cluster [INF]
1.e8 scrub ok
2016-11-26 16:54:34.558607 osd.0 10.27.251.7:6800/3166 7757 : cluster [INF]
1.ed scrub starts
2016-11-26 16:54:36.682207 osd.0 10.27.251.7:6800/3166 7758 : cluster [INF]
1.ed scrub ok
2016-11-26 16:57:07.816187 mon.3 10.27.251.12:6789/0 1 : cluster [INF] mon.3
calling new monitor election
2016-11-26 16:57:13.242951 mon.0 10.27.251.7:6789/0 2317469 : cluster [INF]
mon.0@0 won leader election with quorum 0,1,3
2016-11-26 16:57:13.252424 mon.0 10.27.251.7:6789/0 2317470 : cluster [INF]
HEALTH_WARN; 1 mons down, quorum 0,1,3 0,1,3
2016-11-26 16:57:13.253143 mon.0 10.27.251.7:6789/0 2317471 : cluster [WRN]
mon.3 10.27.251.12:6789/0 clock skew 0.155786s > max 0.05s
2016-11-26 16:57:13.302934 mon.0 10.27.251.7:6789/0 2317472 : cluster [INF]
monmap e4: 4 mons at
Re: [PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
Mandi! Alwin Antreich In chel di` si favelave... > Would have been interesting if all OSDs were up & in. As depending on the > pool config, the min size for serving data out > of that pool might have prevented the storage to serve data. Ouch! I've forgot to specify... not only the status was OK, but effectively all OSDs was up & in, in 'ceph status' and also in PVE interface. Also, for now i've 2 node storage and my pools size is 2. > I see from your ceph status that you have 4 mons, are they all in your > storage conf? And are your storage nodes also mons? Yes. > If your mons aren't peering, then the status wouldn't be OK, so they must > have done it after a while. May you please > show us the logs? Tomorrow. ;-) -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/25/index.php/component/k2/item/123 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
Hi Marco,
On 11/28/2016 03:31 PM, Marco Gaiarin wrote:
> Mandi! Alwin Antreich
> In chel di` si favelave...
>
>> What did the full ceph status show?
>
> Do you mean 'ceph status'? I've not saved it, but was OK, as now:
>
> root@thor:~# ceph status
> cluster 8794c124-c2ec-4e81-8631-742992159bd6
> health HEALTH_OK
> monmap e4: 4 mons at
> {0=10.27.251.7:6789/0,1=10.27.251.8:6789/0,2=10.27.251.11:6789/0,3=10.27.251.12:6789/0}
> election epoch 94, quorum 0,1,2,3 0,1,2,3
> osdmap e114: 6 osds: 6 up, 6 in
> pgmap v2524432: 768 pgs, 3 pools, 944 GB data, 237 kobjects
> 1874 GB used, 7435 GB / 9310 GB avail
> 768 active+clean
> client io 7693 B/s rd, 302 kB/s wr, 65 op/s
>
Would have been interesting if all OSDs were up & in. As depending on the pool
config, the min size for serving data out
of that pool might have prevented the storage to serve data.
>
>> Did you add all the monitors to your storage config in proxmox?
>> A client is speaking to the monitor first to get the proper maps and then
>> connects to the OSDs. The storage would not be
>> available if you only have one monitor configured on the storage tab in
>> proxmox and that mon would be not avialable (eg.
>> 1 mons down).
>
> I've currently 4 nodes in my cluster: all node are pve clusterized, 2
> are cpu only (ceph mon), 2 (and one more to come) storage node
> (mon+osd(s)).
>
> Yes, i've not changed the storage configuration, and when the CPU nodes
> started at least the two storage nodes where online.
I see from your ceph status that you have 4 mons, are they all in your storage
conf? And are your storage nodes also mons?
It is important to have the monitors online, as these are accessed first and if
those aren't then no storage is
available. With only one OSD node running the storage could be still available,
besides a HEALTH_WARN.
>
>
>> Did you configure timesyncd properly?
>> On reboot the time has to be synced by the host, so all ceph hosts share the
>> same time. The ceph map updates require the
>> proper time, so every host knows which map is the current one.
>
> Now, yes. As stated, i've had configured with only a NTP server that was
> a VM in the same cluster; now, they use two NTP server, one remote.
Then a reboot should not do any harm.
>
> Fixed the ntp server, servers get in sync, ceph status got OK but mons
> does not start to peers themself ('pgmap' logs).
If your mons aren't peering, then the status wouldn't be OK, so they must have
done it after a while. May you please
show us the logs?
>
>
> Thanks.
>
--
Cheers,
Alwin
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
Mandi! Alwin Antreich
In chel di` si favelave...
> What did the full ceph status show?
Do you mean 'ceph status'? I've not saved it, but was OK, as now:
root@thor:~# ceph status
cluster 8794c124-c2ec-4e81-8631-742992159bd6
health HEALTH_OK
monmap e4: 4 mons at
{0=10.27.251.7:6789/0,1=10.27.251.8:6789/0,2=10.27.251.11:6789/0,3=10.27.251.12:6789/0}
election epoch 94, quorum 0,1,2,3 0,1,2,3
osdmap e114: 6 osds: 6 up, 6 in
pgmap v2524432: 768 pgs, 3 pools, 944 GB data, 237 kobjects
1874 GB used, 7435 GB / 9310 GB avail
768 active+clean
client io 7693 B/s rd, 302 kB/s wr, 65 op/s
> Did you add all the monitors to your storage config in proxmox?
> A client is speaking to the monitor first to get the proper maps and then
> connects to the OSDs. The storage would not be
> available if you only have one monitor configured on the storage tab in
> proxmox and that mon would be not avialable (eg.
> 1 mons down).
I've currently 4 nodes in my cluster: all node are pve clusterized, 2
are cpu only (ceph mon), 2 (and one more to come) storage node
(mon+osd(s)).
Yes, i've not changed the storage configuration, and when the CPU nodes
started at least the two storage nodes where online.
> Did you configure timesyncd properly?
> On reboot the time has to be synced by the host, so all ceph hosts share the
> same time. The ceph map updates require the
> proper time, so every host knows which map is the current one.
Now, yes. As stated, i've had configured with only a NTP server that was
a VM in the same cluster; now, they use two NTP server, one remote.
Fixed the ntp server, servers get in sync, ceph status got OK but mons
does not start to peers themself ('pgmap' logs).
Thanks.
--
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/
Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN)
marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797
Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
http://www.lanostrafamiglia.it/25/index.php/component/k2/item/123
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
Hi Marco, On 11/28/2016 01:05 PM, Marco Gaiarin wrote: > > A very strange saturday evening. Hardware tooling, hacking, caffeine, > ... > > I'm still completing my CEPH storage cluster (now 2 node storage, > waiting to add the third), but is it mostly ''on production''. > So, after playing with server for some month, saturday i've shut down > all the cluster, setup all the cables, switches, UPS, ... in a more > decent and stable way. > > To simulate a hard power outgage, i've not set the noout and nodown > flags. > > > After that, i've powered up all the cluster (first the 2 ceph storage > node, after the 2 pve host nodes) and i've hit the first trouble: > > 2016-11-26 18:17:29.901353 mon.0 10.27.251.7:6789/0 1218 : cluster > [INF] HEALTH_WARN; clock skew detected on mon.1, mon.2; 1 mons down, quorum > 0,1,2 0,1,2; Monitor clock skew detected > > The trouble came from the fact that... my NTP server was on a VM, and > despite the fact that the status was only 'HEALTH_WARN', i cannot > access anymore the storage. What did the full ceph status show? Did you add all the monitors to your storage config in proxmox? A client is speaking to the monitor first to get the proper maps and then connects to the OSDs. The storage would not be available if you only have one monitor configured on the storage tab in proxmox and that mon would be not avialable (eg. 1 mons down). Did you configure timesyncd properly? On reboot the time has to be synced by the host, so all ceph hosts share the same time. The ceph map updates require the proper time, so every host knows which map is the current one. > > I've solved adding more NTP server from other sites, and after some > time the cluster go OK: > > 2016-11-26 19:11:33.343818 mon.0 10.27.251.7:6789/0 1581 : cluster > [INF] HEALTH_OK > > and here the panic start. > > > PVE interface report the Ceph cluster OK, report correctly all the stuffs > (mon, osd, pools, pool usage, ...) but data cluster was not accessible: > > a) if i try to move a disk, reply with something like 'no available'. > > b) if i try to start VMs, they stalls... > > The only strange things on log was that there's NO pgmap update, like > before: > > 2016-11-26 16:59:31.588695 mon.0 10.27.251.7:6789/0 2317560 : cluster > [INF] pgmap v2410540: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, > 7452 GB / 9310 GB avail; 13569 kB/s rd, 2731 kB/s wr, 565 op/s > > but really, on panic, i've not noted that. > > > After some tests, i've finally do the right thing. > > 1) i've set the noout and nodown flags. > > 2) i've rebooted the ceph nodes, one by one. > > After that, all the cluster start. VMs that was on stalls, immediately > start. > > > After that, i've understood that NTP is a crucial service for ceph, so > it is needed to have a pool of servers. Still, i'm not sure this was > the culprit. > > > The second thing i've understood is that Ceph react badly to a total > shutdown. In a datacenter this is probably acceptable. > > I don't know if it is my fault, or at least there's THE RIGTH WAY to > start a Ceph cluster from cold metal... > > > Thanks. > -- Cheers, Alwin ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Ceph: PANIC or DON'T PANIC? ;-)
A very strange saturday evening. Hardware tooling, hacking, caffeine, ... I'm still completing my CEPH storage cluster (now 2 node storage, waiting to add the third), but is it mostly ''on production''. So, after playing with server for some month, saturday i've shut down all the cluster, setup all the cables, switches, UPS, ... in a more decent and stable way. To simulate a hard power outgage, i've not set the noout and nodown flags. After that, i've powered up all the cluster (first the 2 ceph storage node, after the 2 pve host nodes) and i've hit the first trouble: 2016-11-26 18:17:29.901353 mon.0 10.27.251.7:6789/0 1218 : cluster [INF] HEALTH_WARN; clock skew detected on mon.1, mon.2; 1 mons down, quorum 0,1,2 0,1,2; Monitor clock skew detected The trouble came from the fact that... my NTP server was on a VM, and despite the fact that the status was only 'HEALTH_WARN', i cannot access anymore the storage. I've solved adding more NTP server from other sites, and after some time the cluster go OK: 2016-11-26 19:11:33.343818 mon.0 10.27.251.7:6789/0 1581 : cluster [INF] HEALTH_OK and here the panic start. PVE interface report the Ceph cluster OK, report correctly all the stuffs (mon, osd, pools, pool usage, ...) but data cluster was not accessible: a) if i try to move a disk, reply with something like 'no available'. b) if i try to start VMs, they stalls... The only strange things on log was that there's NO pgmap update, like before: 2016-11-26 16:59:31.588695 mon.0 10.27.251.7:6789/0 2317560 : cluster [INF] pgmap v2410540: 768 pgs: 768 active+clean; 936 GB data, 1858 GB used, 7452 GB / 9310 GB avail; 13569 kB/s rd, 2731 kB/s wr, 565 op/s but really, on panic, i've not noted that. After some tests, i've finally do the right thing. 1) i've set the noout and nodown flags. 2) i've rebooted the ceph nodes, one by one. After that, all the cluster start. VMs that was on stalls, immediately start. After that, i've understood that NTP is a crucial service for ceph, so it is needed to have a pool of servers. Still, i'm not sure this was the culprit. The second thing i've understood is that Ceph react badly to a total shutdown. In a datacenter this is probably acceptable. I don't know if it is my fault, or at least there's THE RIGTH WAY to start a Ceph cluster from cold metal... Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/25/index.php/component/k2/item/123 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
