What's the value of the global setting "minreq.sysvmtemplate.version" ?
-Wei
On Fri, Jun 20, 2025 at 6:17 PM Stephan Bienek <stephan....@bienek.org>
wrote:
> Dear Community,
>
> i am trying to better understand the Virtual Router Live Patching aka VR
> Update without service interruption.
>
> We did update a CloudStack 4.20.0 Cluster to 4.20.1
> As expected, shutdown VMs connected to unpatched Isolated networks can't
> start and firewall rules etc can't be changed complaining about a required
> update of the VR.
>
> When i execute the live patching of an isolated network or of a VR, i
> always receive the following error message "Failed to restart network" and
> i see the log entry "Unable to send command. Router requires upgrade"
>
> 2025-06-19 13:03:26,740 WARN [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Failed to implement network Network {"id": 279, "name":
> "TESTNETWORK LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> "networkofferingid": 8} elements and resources as a part of network restart
> due to com.cloud.exception.ResourceUnavailableException: Resource
> [VirtualRouter:922] is unreachable: Unable to send command. Router requires
> upgrade
> 2025-06-19 13:03:26,785 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Complete
> async job-23502, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> to restart network"}
>
> From the logs, shortly before i see a success message "Successfully
> patched router VM":
>
> 2025-06-19 13:03:26,606 INFO [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Successfully patched router VM instance
> {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"}
>
> Is it because the systemvm template was updated (Debian 11 to Debian 12)
> and live patching can only be done on the same systemvm template version?
>
> 2025-06-19 13:03:26,738 DEBUG [c.c.n.r.NetworkHelperImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Router requires upgrade. Unable to send command to router:
> VM instance
> {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"},
> router template version: Cloudstack Release 4.20.0 Fri Sep 6 03:45:27 AM
> UTC 2024, minimal required version: 4.20.1
>
> The only way to get the network/VR fully functional again, allowing
> stopped VMs to start and changing firewall rules etc is to execute a
> network restart with cleanup or execute "Upgrade router to use newer
> template" - which comes with a service impact as expected.
>
> Full log attached to the end of this mail, in case i am missing something
> obvious.
>
> Thanks in advance for helping me to understand the requirements for
> non-service impacting live patching.
>
> Best regards,
> Stephan
>
> ############
> ### full log ###
> ############
>
> 2025-06-19 13:02:57,903 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp1404565079-435:[ctx-ccc0da24, ctx-7ab4ba90]) (logid:64ccbfcd) submit
> async job-23502, details: AsyncJob
> {"accountId":4,"cmd":"org.apache.cloudstack.api.command.user.network.RestartNetworkCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"9\",\"livepatch\":\"true\",\"sessionkey\":\"1234567890abcdefg\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"82573\",\"id\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"ctxDetails\":\"{\\\"interface
> com.cloud.network.Network\\\":\\\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\\\"}\",\"ctxAccountId\":\"4\",\"uuid\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"cmdEventType\":\"NETWORK.RESTART\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":23502,"initMsid":33830401474176,"instanceId":279,"instanceType":"Network","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":9,"uuid":"c02acb59-37e0-4bba-bd8e-82294c6
> 6c505"}
> 2025-06-19 13:02:57,908 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Executing
> AsyncJob
> {"accountId":4,"cmd":"org.apache.cloudstack.api.command.user.network.RestartNetworkCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"9\",\"livepatch\":\"true\",\"sessionkey\":\"1234567890abcdefg\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"82573\",\"id\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"ctxDetails\":\"{\\\"interface
> com.cloud.network.Network\\\":\\\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\\\"}\",\"ctxAccountId\":\"4\",\"uuid\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"cmdEventType\":\"NETWORK.RESTART\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":23502,"initMsid":33830401474176,"instanceId":279,"instanceType":"Network","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":9,"uuid":"c02acb59-37e0-4bba-bd8e-82294c66c505"}
> 2025-06-19 13:02:57,948 DEBUG [c.c.u.AccountManagerImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Account [Account
> [{"accountName":"Cloudstack_Admins","id":4,"uuid":"742dba95-8709-4aba-92c8-fb472e118ddb"}]]
> has access to resource.
> 2025-06-19 13:02:57,961 DEBUG [c.c.a.ApiServlet]
> (qtp1404565079-435:[ctx-d1310f58]) (logid:bf1fd0ff) ===START=== 192.168.1.5
> -- GET
> jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> 2025-06-19 13:02:58,007 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Restarting network Network {"id": 279, "name":
> "TESTNETWORK LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> "networkofferingid": 8}...
> 2025-06-19 13:02:58,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Wait time setting on
> com.cloud.agent.api.PatchSystemVmCommand is 600000 seconds
> 2025-06-19 13:02:58,046 DEBUG [c.c.a.m.ClusteredAgentAttache]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Seq 3902-2730025799116652757: Forwarding Seq
> 3986-2730025799116652757: { Cmd , MgmtId: 33830401474176, via:
> 3986(kvm-001), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.PatchSystemVmCommand":{"forced":"true","accessDetails":{"
> router.name":"r-922-VM","router.ip":"169.254.77.6"},"wait":"0","bypassHostMaintenance":"false"}}]
> } to 24728157879110
> 2025-06-19 13:02:58,050 DEBUG [c.c.a.ApiServlet]
> (qtp1404565079-435:[ctx-d1310f58, ctx-531f992f]) (logid:bf1fd0ff) ===END===
> 192.168.1.5 -- GET
> jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> 2025-06-19 13:03:26,591 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Seq 3986-2730025799116652757: Received: { Ans: , MgmtId:
> 33830401474176, via: 3986(kvm-001), Ver: v1, Flags: 10, {
> PatchSystemVmAnswer } }
> 2025-06-19 13:03:26,591 INFO [c.c.s.ManagementServerImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Successfully patched system VM r-922-VM
> 2025-06-19 13:03:26,606 INFO [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Successfully patched router VM instance
> {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"}
> 2025-06-19 13:03:26,607 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Implementing the network Network {"id": 279, "name":
> "TESTNETWORK LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> "networkofferingid": 8} elements and resources as a part of network restart
> without cleanup
> 2025-06-19 13:03:26,638 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Asking VirtualRouter to implement Network {"id": 279,
> "name": "TESTNETWORK LIVEPATCH", "uuid":
> "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8}
> 2025-06-19 13:03:26,668 DEBUG [c.c.a.ApiServlet]
> (qtp1404565079-321:[ctx-fcacff74]) (logid:a66d7bee) ===START=== 192.168.1.5
> -- GET
> jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> 2025-06-19 13:03:26,704 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Service SecurityGroup is not supported in the network
> Network {"id": 279, "name": "TESTNETWORK LIVEPATCH", "uuid":
> "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8}
> 2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Looking for vms/cloud-scripts.tgz in the classpath
> 2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) System resource:
> file:/usr/share/cloudstack-common/vms/cloud-scripts.tgz
> 2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Absolute path =
> /usr/share/cloudstack-common/vms/cloud-scripts.tgz
> 2025-06-19 13:03:26,738 DEBUG [c.c.n.r.NetworkHelperImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Router requires upgrade. Unable to send command to router:
> VM instance
> {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"},
> router template version: Cloudstack Release 4.20.0 Fri Sep 6 03:45:27 AM
> UTC 2024, minimal required version: 4.20.1
> 2025-06-19 13:03:26,740 WARN [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Failed to implement network Network {"id": 279, "name":
> "TESTNETWORK LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> "networkofferingid": 8} elements and resources as a part of network restart
> due to com.cloud.exception.ResourceUnavailableException: Resource
> [VirtualRouter:922] is unreachable: Unable to send command. Router requires
> upgrade
> 2025-06-19 13:03:26,741 WARN [c.c.n.NetworkServiceImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Network Network {"id": 279, "name": "TESTNETWORK
> LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> "networkofferingid": 8} failed to restart.
> 2025-06-19 13:03:26,741 WARN [o.a.c.m.w.WebhookServiceImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Skipping delivering event Event
> {"description":"{\"event\":\"NETWORK.RESTART\",\"status\":\"Completed\"}","eventId":null,"eventType":"NETWORK.RESTART","eventUuid":null,"resourceType":"Network","resourceUUID":null}
> to any webhook as account ID is missing
> 2025-06-19 13:03:26,742 WARN [o.a.c.f.e.EventDistributorImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> (logid:c02acb59) Failed to publish event [category: ActionEvent, type:
> NETWORK.RESTART] on bus webhookEventBus
> 2025-06-19 13:03:26,752 DEBUG [c.c.a.ApiServlet]
> (qtp1404565079-321:[ctx-fcacff74, ctx-dc88a930]) (logid:a66d7bee) ===END===
> 192.168.1.5 -- GET
> jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> 2025-06-19 13:03:26,785 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Complete
> async job-23502, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> to restart network"}
> 2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Publish
> async job-23502 complete on message bus
> 2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Wake up
> jobs related to job-23502
> 2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Update db
> status for job-23502
> 2025-06-19 13:03:26,793 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Wake up
> jobs joined with job-23502 and disjoin all subjobs created from job- 23502
> 2025-06-19 13:03:26,824 DEBUG [c.c.a.ApiServer]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Retrieved
> cmdEventType from job info: NETWORK.RESTART
> 2025-06-19 13:03:26,832 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Done
> executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> for job-23502
> 2025-06-19 13:03:26,832 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Remove
> job-23502 from job monitoring
