That was actually a really BAD ADVICE…. as when node initiate maintenance mode
it will try to migrate hosted vms … and eventually ends up in the same Lock
loop..
what you really need is to remove started vms from ha-manager, so when the node
initiate shutdown it will do firstly do regular shutdown vm per vm.
So, do something like below as first command in your NUT command sequence:
for a in `ha-manager status | grep started|awk '{print $2}'|sed 's/vm://g'`; do
ha-manager remove $a;done
> On Mar 10, 2022, at 2:48 PM, [email protected] wrote:
>
> I don’t remember, search into pvecm and pve[tab][tab] related commands man
> pages
>
>> On Mar 10, 2022, at 2:19 PM, Stefan Radman <[email protected]> wrote:
>>
>> Hi Sto
>>
>> Thanks for the suggestions.
>>
>> The second option is what I was looking for.
>>
>> How do I initiate “pve node maintenance mode”?
>>
>> The “Node Maintenance” paragraph in the HA documentation is quite brief and
>> does not refer to any command or GUI component.
>>
>> Thank you
>>
>> Stefan
>>
>>
>>> On Mar 10, 2022, at 14:50, [email protected]
>>> <mailto:[email protected]> wrote:
>>>
>>> Hi,
>>>
>>> here are two ideas: shutdown sequence -and- command sequence
>>> 1: shutdown sequence you may achieve when you set NUT’s on each node to
>>> only monitor the UPS power, then configure each node to shutdown itself on
>>> a different ups power levels, ex: node1 on 15% battery, node2 on 10%
>>> battery and so on
>>> 2: you can set a cmd sequence to firstly execute pve node maintenance mode
>>> , and then execute shutdown -> this way HA will not try to migrate vm to
>>> node in maintenance, and the chance all nodes to goes into maintenance in
>>> exactly same second seems to be not a risk at all.
>>>
>>> hope thats helpful.
>>>
>>> Regards,
>>> Sto.
>>>
>>>> On Mar 10, 2022, at 1:10 PM, Stefan Radman via pve-user
>>>> <[email protected] <mailto:[email protected]>> wrote:
>>>>
>>>>
>>>> From: Stefan Radman <[email protected] <mailto:[email protected]>>
>>>> Subject: Locking HA during UPS shutdown
>>>> Date: March 10, 2022 at 1:10:09 PM GMT+2
>>>> To: PVE User List <[email protected]
>>>> <mailto:[email protected]>>
>>>>
>>>>
>>>> Hi
>>>>
>>>> I am configuring a 3 node PVE cluster with integrated Ceph storage.
>>>>
>>>> It is powered by 2 UPS that are monitored by NUT (Network UPS Tools).
>>>>
>>>> HA is configured with 3 groups:
>>>> group pve1 nodes pve1:1,pve2,pve3
>>>> group pve2 nodes pve1,pve2:1,pve3
>>>> group pve3 nodes pve1,pve2,pve3:1
>>>>
>>>> That will normally place the VMs in each group on the corresponding node,
>>>> unless that node fails.
>>>>
>>>> The cluster is configured to migrate VMs away from a node before shutting
>>>> it down (Cluster=>Options=>HA Settings: shutdown_policy=migrate).
>>>>
>>>> NUT is configured to shut down the serves once the last of the two UPS is
>>>> running low on battery.
>>>>
>>>> My problem:
>>>> When NUT starts shutting down the 3 nodes, HA will first try to
>>>> live-migrate them to another node.
>>>> That live migration process gets stuck because all the nodes are shutting
>>>> down simultaneously.
>>>> It seems that the whole process runs into a timeout, finally “powers off”
>>>> all the VMs and shuts down the nodes.
>>>>
>>>> My question:
>>>> Is there a way to “lock” or temporarily de-activate HA before shutting
>>>> down a node to avoid that deadlock?
>>>>
>>>> Thank you
>>>>
>>>> Stefan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> pve-user mailing list
>>>> [email protected] <mailto:[email protected]>
>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>> <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>
>>>
>>>
>>> Best Regards,
>>>
>>> Stoyan Stoyanov Sto | Solutions Manager
>>> | Telehouse.Solutions | ICT Department
>>> | phone/viber: +359 894774934 <tel:+359 894774934>
>>> | telegram: @prostoSto
>>> <https://mysignature.io/redirect/skype:prosto.sto?chat>
>>> | skype: prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
>>> | email: [email protected] <mailto:[email protected]>
>>> | website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
>>> | address: Telepoint #2, Sofia, Bulgaria
>>> <https://mysignature.io/editor/?utm_source=freepixel><356841.png>
>>>
>>> <https://mysig.io/ZDNkNWY>
>>> Save paper. Don’t print
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Best Regards,
>>>
>>> Stoyan Stoyanov Sto | Solutions Manager
>>> | Telehouse.Solutions | ICT Department
>>> | phone/viber: +359 894774934 <tel:+359 894774934>
>>> | telegram: @prostoSto
>>> <https://mysignature.io/redirect/skype:prosto.sto?chat>
>>> | skype: prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
>>> | email: [email protected] <mailto:[email protected]>
>>> | website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
>>> | address: Telepoint #2, Sofia, Bulgaria
>>> <https://mysignature.io/editor/?utm_source=freepixel><356841.png>
>>>
>>> <https://mysig.io/ZDNkNWY>
>>> Save paper. Don’t print
>>
>
>
> Best Regards,
>
> Stoyan Stoyanov Sto | Solutions Manager
> | Telehouse.Solutions | ICT Department
> | phone/viber: +359 894774934 <tel:+359 894774934>
> | telegram: @prostoSto
> <https://mysignature.io/redirect/skype:prosto.sto?chat>
> | skype: prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
> | email: [email protected] <mailto:[email protected]>
> | website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
> | address: Telepoint #2, Sofia, Bulgaria
> <https://mysignature.io/editor/?utm_source=freepixel>
>
> <https://mysig.io/ZDNkNWY>
> Save paper. Don’t print
>
>
>
>
> _______________________________________________
> pve-user mailing list
> [email protected]
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Best Regards,
Stoyan Stoyanov Sto | Solutions Manager
| Telehouse.Solutions | ICT Department
| phone/viber: +359 894774934 <tel:+359 894774934>
| telegram: @prostoSto <https://mysignature.io/redirect/skype:prosto.sto?chat>
| skype: prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
| email: [email protected] <mailto:[email protected]>
| website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
| address: Telepoint #2, Sofia, Bulgaria
<https://mysignature.io/editor/?utm_source=freepixel>
<https://mysig.io/ZDNkNWY>
Save paper. Don’t print
_______________________________________________
pve-user mailing list
[email protected]
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
