Folks... I CEPH servers are in the same network: 10.10.10.0/24... There is a optic channel between the builds: buildA and buildB, just to identified! When I create the cluster in first time, 3 servers going down in buildB, and the remain ceph servers continued to worked properly... I do not understand why now this cant happens anymore! Sorry if I sound like a newbie! I still learn about it! --- Gilberto Nunes Ferreira
(47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em sex, 5 de out de 2018 às 09:44, Marcus Haarmann < [email protected]> escreveu: > Gilberto, > > the underlying problem is a ceph problem and not related to VMs or > Proxmox. > The ceph system requires a mayority of monitor nodes to be active. > Your setup seems to have 3 mon nodes, which results in a loss of quorum > when two of these servers are gone. > Check "ceph -s" on each side if you see any reaction of ceph. > If not, probably not enough mons are present. > > Also, when one side is down you should see a non-presence of some OSD > instances. > In this case, ceph might be up but your VMs which are spread over the OSD > disks, > might block because of the non-accessibility of the primary storage. > The distribution of data over the OSD instances is steered by the crush > map. > You should make sure to have enough copies configured and the crush map > set up in a way > that on each side of your cluster is minimum one copy. > In case the crush map is mis-configured, all copies of your data may be on > the wrong side, > esulting in proxmox not being able to access the VM data. > > Marcus Haarmann > > > Von: "Gilberto Nunes" <[email protected]> > An: "pve-user" <[email protected]> > Gesendet: Freitag, 5. Oktober 2018 14:31:20 > Betreff: Re: [PVE-User] Proxmox CEPH 6 servers failures! > > Nice.. Perhaps if I create a VM in Proxmox01 and Proxmox02, and join this > VM into Cluster Ceph, can I solve to quorum problem? > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em sex, 5 de out de 2018 às 09:23, dorsy <[email protected]> escreveu: > > > Your question has already been answered. You need majority to have > quorum. > > > > On 2018. 10. 05. 14:10, Gilberto Nunes wrote: > > > Hi > > > Perhaps this can help: > > > > > > https://imageshack.com/a/img921/6208/X7ha8R.png > > > > > > I was thing about it, and perhaps if I deploy a VM in both side, with > > > Proxmox and add this VM to the CEPH cluster, maybe this can help! > > > > > > thanks > > > --- > > > Gilberto Nunes Ferreira > > > > > > (47) 3025-5907 > > > (47) 99676-7530 - Whatsapp / Telegram > > > > > > Skype: gilberto.nunes36 > > > > > > > > > > > > > > > > > > Em sex, 5 de out de 2018 às 03:55, Alexandre DERUMIER < > > [email protected]> > > > escreveu: > > > > > >> Hi, > > >> > > >> Can you resend your schema, because it's impossible to read. > > >> > > >> > > >> but you need to have to quorum on monitor to have the cluster > working. > > >> > > >> > > >> ----- Mail original ----- > > >> De: "Gilberto Nunes" <[email protected]> > > >> À: "proxmoxve" <[email protected]> > > >> Envoyé: Jeudi 4 Octobre 2018 22:05:16 > > >> Objet: [PVE-User] Proxmox CEPH 6 servers failures! > > >> > > >> Hi there > > >> > > >> I have something like this: > > >> > > >> CEPH01 ----| > > >> |----- CEPH04 > > >> | > > >> | > > >> CEPH02 > ----|-----------------------------------------------------|---- > > >> CEPH05 > > >> | Optic Fiber > > >> | > > >> CEPH03 ----| > > >> |--- CEPH06 > > >> > > >> Sometime, when Optic Fiber not work, and just CEPH01, CEPH02 and > CEPH03 > > >> remains, the entire cluster fail! > > >> I find out the cause! > > >> > > >> ceph.conf > > >> > > >> [global] auth client required = cephx auth cluster required = cephx > auth > > >> service required = cephx cluster network = 10.10.10.0/24 fsid = > > >> e67534b4-0a66-48db-ad6f-aa0868e962d8 keyring = > > >> /etc/pve/priv/$cluster.$name.keyring mon allow pool delete = true osd > > >> journal size = 5120 osd pool default min size = 2 osd pool default > size > > = > > >> 3 > > >> public network = 10.10.10.0/24 [osd] keyring = > > >> /var/lib/ceph/osd/ceph-$id/keyring [mon.pve-ceph01] host = pve-ceph01 > > mon > > >> addr = 10.10.10.100:6789 mon osd allow primary affinity = true > > >> [mon.pve-ceph02] host = pve-ceph02 mon addr = 10.10.10.110:6789 mon > osd > > >> allow primary affinity = true [mon.pve-ceph03] host = pve-ceph03 mon > > addr > > >> = > > >> 10.10.10.120:6789 mon osd allow primary affinity = true > > [mon.pve-ceph04] > > >> host = pve-ceph04 mon addr = 10.10.10.130:6789 mon osd allow primary > > >> affinity = true [mon.pve-ceph05] host = pve-ceph05 mon addr = > > >> 10.10.10.140:6789 mon osd allow primary affinity = true > > [mon.pve-ceph06] > > >> host = pve-ceph06 mon addr = 10.10.10.150:6789 mon osd allow primary > > >> affinity = true > > >> > > >> Any help will be welcome! > > >> > > >> --- > > >> Gilberto Nunes Ferreira > > >> > > >> (47) 3025-5907 > > >> (47) 99676-7530 - Whatsapp / Telegram > > >> > > >> Skype: gilberto.nunes36 > > >> _______________________________________________ > > >> pve-user mailing list > > >> [email protected] > > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >> > > >> _______________________________________________ > > >> pve-user mailing list > > >> [email protected] > > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >> > > > _______________________________________________ > > > pve-user mailing list > > > [email protected] > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > > pve-user mailing list > > [email protected] > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > [email protected] > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > [email protected] > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ pve-user mailing list [email protected] https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
