Dear Community,
i am trying to better understand the Virtual Router Live Patching aka VR Update
without service interruption.
We did update a CloudStack 4.20.0 Cluster to 4.20.1
As expected, shutdown VMs connected to unpatched Isolated networks can't start
and firewall rules etc can't be changed complaining about a required update of
the VR.
When i execute the live patching of an isolated network or of a VR, i always
receive the following error message "Failed to restart network" and i see the
log entry "Unable to send command. Router requires upgrade"
2025-06-19 13:03:26,740 WARN [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Failed to implement network Network {"id": 279, "name": "TESTNETWORK
LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
"networkofferingid": 8} elements and resources as a part of network restart due
to com.cloud.exception.ResourceUnavailableException: Resource
[VirtualRouter:922] is unreachable: Unable to send command. Router requires
upgrade
2025-06-19 13:03:26,785 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Complete async
job-23502, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to restart network"}
>From the logs, shortly before i see a success message "Successfully patched
>router VM":
2025-06-19 13:03:26,606 INFO [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Successfully patched router VM instance
{"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"}
Is it because the systemvm template was updated (Debian 11 to Debian 12) and
live patching can only be done on the same systemvm template version?
2025-06-19 13:03:26,738 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Router requires upgrade. Unable to send command to router: VM instance
{"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"},
router template version: Cloudstack Release 4.20.0 Fri Sep 6 03:45:27 AM UTC
2024, minimal required version: 4.20.1
The only way to get the network/VR fully functional again, allowing stopped VMs
to start and changing firewall rules etc is to execute a network restart with
cleanup or execute "Upgrade router to use newer template" - which comes with a
service impact as expected.
Full log attached to the end of this mail, in case i am missing something
obvious.
Thanks in advance for helping me to understand the requirements for non-service
impacting live patching.
Best regards,
Stephan
############
### full log ###
############
2025-06-19 13:02:57,903 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(qtp1404565079-435:[ctx-ccc0da24, ctx-7ab4ba90]) (logid:64ccbfcd) submit async
job-23502, details: AsyncJob
{"accountId":4,"cmd":"org.apache.cloudstack.api.command.user.network.RestartNetworkCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"9\",\"livepatch\":\"true\",\"sessionkey\":\"1234567890abcdefg\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"82573\",\"id\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"ctxDetails\":\"{\\\"interface
com.cloud.network.Network\\\":\\\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\\\"}\",\"ctxAccountId\":\"4\",\"uuid\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"cmdEventType\":\"NETWORK.RESTART\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":23502,"initMsid":33830401474176,"instanceId":279,"instanceType":"Network","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":9,"uuid":"c02acb59-37e0-4bba-bd8e-82294c6
6c505"}
2025-06-19 13:02:57,908 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Executing
AsyncJob
{"accountId":4,"cmd":"org.apache.cloudstack.api.command.user.network.RestartNetworkCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"9\",\"livepatch\":\"true\",\"sessionkey\":\"1234567890abcdefg\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"82573\",\"id\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"ctxDetails\":\"{\\\"interface
com.cloud.network.Network\\\":\\\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\\\"}\",\"ctxAccountId\":\"4\",\"uuid\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"cmdEventType\":\"NETWORK.RESTART\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":23502,"initMsid":33830401474176,"instanceId":279,"instanceType":"Network","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":9,"uuid":"c02acb59-37e0-4bba-bd8e-82294c66c505"}
2025-06-19 13:02:57,948 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Account [Account
[{"accountName":"Cloudstack_Admins","id":4,"uuid":"742dba95-8709-4aba-92c8-fb472e118ddb"}]]
has access to resource.
2025-06-19 13:02:57,961 DEBUG [c.c.a.ApiServlet]
(qtp1404565079-435:[ctx-d1310f58]) (logid:bf1fd0ff) ===START=== 192.168.1.5 --
GET
jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
2025-06-19 13:02:58,007 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Restarting network Network {"id": 279, "name": "TESTNETWORK LIVEPATCH", "uuid":
"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8}...
2025-06-19 13:02:58,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Wait time setting on com.cloud.agent.api.PatchSystemVmCommand is 600000 seconds
2025-06-19 13:02:58,046 DEBUG [c.c.a.m.ClusteredAgentAttache]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Seq 3902-2730025799116652757: Forwarding Seq 3986-2730025799116652757: { Cmd ,
MgmtId: 33830401474176, via: 3986(kvm-001), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.PatchSystemVmCommand":{"forced":"true","accessDetails":{"router.name":"r-922-VM","router.ip":"169.254.77.6"},"wait":"0","bypassHostMaintenance":"false"}}]
} to 24728157879110
2025-06-19 13:02:58,050 DEBUG [c.c.a.ApiServlet]
(qtp1404565079-435:[ctx-d1310f58, ctx-531f992f]) (logid:bf1fd0ff) ===END===
192.168.1.5 -- GET
jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
2025-06-19 13:03:26,591 DEBUG [c.c.a.t.Request]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Seq 3986-2730025799116652757: Received: { Ans: , MgmtId: 33830401474176, via:
3986(kvm-001), Ver: v1, Flags: 10, { PatchSystemVmAnswer } }
2025-06-19 13:03:26,591 INFO [c.c.s.ManagementServerImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Successfully patched system VM r-922-VM
2025-06-19 13:03:26,606 INFO [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Successfully patched router VM instance
{"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"}
2025-06-19 13:03:26,607 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Implementing the network Network {"id": 279, "name": "TESTNETWORK LIVEPATCH",
"uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8}
elements and resources as a part of network restart without cleanup
2025-06-19 13:03:26,638 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Asking VirtualRouter to implement Network {"id": 279, "name": "TESTNETWORK
LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
"networkofferingid": 8}
2025-06-19 13:03:26,668 DEBUG [c.c.a.ApiServlet]
(qtp1404565079-321:[ctx-fcacff74]) (logid:a66d7bee) ===START=== 192.168.1.5 --
GET
jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
2025-06-19 13:03:26,704 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Service SecurityGroup is not supported in the network Network {"id": 279,
"name": "TESTNETWORK LIVEPATCH", "uuid":
"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8}
2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Looking for vms/cloud-scripts.tgz in the classpath
2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
System resource: file:/usr/share/cloudstack-common/vms/cloud-scripts.tgz
2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Absolute path = /usr/share/cloudstack-common/vms/cloud-scripts.tgz
2025-06-19 13:03:26,738 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Router requires upgrade. Unable to send command to router: VM instance
{"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"},
router template version: Cloudstack Release 4.20.0 Fri Sep 6 03:45:27 AM UTC
2024, minimal required version: 4.20.1
2025-06-19 13:03:26,740 WARN [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Failed to implement network Network {"id": 279, "name": "TESTNETWORK
LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
"networkofferingid": 8} elements and resources as a part of network restart due
to com.cloud.exception.ResourceUnavailableException: Resource
[VirtualRouter:922] is unreachable: Unable to send command. Router requires
upgrade
2025-06-19 13:03:26,741 WARN [c.c.n.NetworkServiceImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Network Network {"id": 279, "name": "TESTNETWORK LIVEPATCH", "uuid":
"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8} failed to
restart.
2025-06-19 13:03:26,741 WARN [o.a.c.m.w.WebhookServiceImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Skipping delivering event Event
{"description":"{\"event\":\"NETWORK.RESTART\",\"status\":\"Completed\"}","eventId":null,"eventType":"NETWORK.RESTART","eventUuid":null,"resourceType":"Network","resourceUUID":null}
to any webhook as account ID is missing
2025-06-19 13:03:26,742 WARN [o.a.c.f.e.EventDistributorImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d]) (logid:c02acb59)
Failed to publish event [category: ActionEvent, type: NETWORK.RESTART] on bus
webhookEventBus
2025-06-19 13:03:26,752 DEBUG [c.c.a.ApiServlet]
(qtp1404565079-321:[ctx-fcacff74, ctx-dc88a930]) (logid:a66d7bee) ===END===
192.168.1.5 -- GET
jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
2025-06-19 13:03:26,785 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Complete async
job-23502, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to restart network"}
2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Publish async
job-23502 complete on message bus
2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Wake up jobs
related to job-23502
2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Update db
status for job-23502
2025-06-19 13:03:26,793 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Wake up jobs
joined with job-23502 and disjoin all subjobs created from job- 23502
2025-06-19 13:03:26,824 DEBUG [c.c.a.ApiServer]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Retrieved
cmdEventType from job info: NETWORK.RESTART
2025-06-19 13:03:26,832 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Done executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-23502
2025-06-19 13:03:26,832 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Remove
job-23502 from job monitoring
