Since there were no config changes, I would have a look on cluster communication, i.e switch issues ?
On Fri, Aug 11, 2017 at 11:02 AM, Chris Tomkins <chr...@brandwatch.com> wrote: > Hi Proxmox users, > > I have a 4 node cluster. It has been in production for a few months with > few/no issues. > > This morning one of my admins reported that each node appeared isolated > ("Total votes: 1"). All VMs were up and unaffected. Unfortunately I > made the mistake of stopping VMs on 3 of the nodes to apply updates and > reboot as I assumed this would clear the issue. Now the scenario remains > the same but the VMs are down on 3 of the nodes and it won't allow me to > start them as I have no quorum. > > No config changes were made and this cluster was fine and had quorum last > time I looked (last week). > > I don't want to take the wrong action and make this worse - any advice > would be greatly appreciated! > > hypervisors ar1406/ar1600/ar1601 are up to date and have been rebooted this > morning. ar1407 has not been rebooted or updated (yet) as the VMs on it are > critical. > > Thanks, > > Chris > > [LIVE]root@ar1406:~# for i in ar1406 ar1407 ar1600 ar1601; do ssh $i 'cat > /etc/pve/.members'; done > { > "nodename": "ar1406", > "version": 3, > "cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate": > 0 }, > "nodelist": { > "ar1407": { "id": 2, "online": 0}, > "ar1601": { "id": 3, "online": 0}, > "ar1600": { "id": 4, "online": 0}, > "ar1406": { "id": 1, "online": 1, "ip": "10.0.6.201"} > } > } > { > "nodename": "ar1407", > "version": 3, > "cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate": > 0 }, > "nodelist": { > "ar1407": { "id": 2, "online": 1, "ip": "10.0.6.202"}, > "ar1601": { "id": 3, "online": 0}, > "ar1600": { "id": 4, "online": 0}, > "ar1406": { "id": 1, "online": 0} > } > } > { > "nodename": "ar1600", > "version": 3, > "cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate": > 0 }, > "nodelist": { > "ar1407": { "id": 2, "online": 0}, > "ar1601": { "id": 3, "online": 0}, > "ar1600": { "id": 4, "online": 1, "ip": "10.0.6.203"}, > "ar1406": { "id": 1, "online": 0} > } > } > { > "nodename": "ar1601", > "version": 3, > "cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate": > 0 }, > "nodelist": { > "ar1407": { "id": 2, "online": 0}, > "ar1601": { "id": 3, "online": 1, "ip": "10.0.6.204"}, > "ar1600": { "id": 4, "online": 0}, > "ar1406": { "id": 1, "online": 0} > } > } > > -- > > Chris Tomkins > > Brandwatch | Senior Network Engineer (Linux/Network) > > chr...@brandwatch.com | (+44) 01273 448 949 > > @Brandwatch > > New York | San Francisco | Brighton | Singapore | Berlin | > Stuttgart > > > Discover how organizations are using Brandwatch to create their own success > <https://www.brandwatch.com/customer-success/> > > > Email disclaimer <http://www.brandwatch.com/email-disclaimer/> > > > [image: bw-signature logo.png] > _______________________________________________ > pve-user mailing list > pve-user@pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user