Hi Wei,
thanks a lot for your explanation, which helped me a lot to get clarification
about the Live Patching process and specifically the requirements.
Kind regards,
Stephan
> Wei ZHOU <ustcweiz...@gmail.com> hat am 21.06.2025 12:22 CEST geschrieben:
>
>
> Hi Stephan,
>
> "4.20.0" should be fine as 4.20.0 and 4.20.1 systemvm templates are both
> based on Debian 12.X
>
> The 4.19.x system template is based on Debian 11. live-patch also worked in
> my testing few months ago, but I personally do not recommend it ("4.19.x")
>
>
> Kind regards,
> Wei
>
> On Fri, Jun 20, 2025 at 10:00 PM Stephan Bienek <stephan....@bienek.org>
> wrote:
>
> > Hi Wei,
> >
> > an interesting thought - indeed the setting states "4.20.1".
> > I can't remember manually changing the value after the update.
> >
> > Do you you know if changing it to 4.20.0 would be safe way or could be
> > causing any potential issues?
> >
> > Thanks,
> > Stephan
> >
> > > Wei ZHOU <ustcweiz...@gmail.com> hat am 20.06.2025 19:21 CEST
> > geschrieben:
> > >
> > >
> > > What's the value of the global setting "minreq.sysvmtemplate.version" ?
> > >
> > >
> > > -Wei
> > >
> > > On Fri, Jun 20, 2025 at 6:17 PM Stephan Bienek <stephan....@bienek.org>
> > > wrote:
> > >
> > > > Dear Community,
> > > >
> > > > i am trying to better understand the Virtual Router Live Patching aka
> > VR
> > > > Update without service interruption.
> > > >
> > > > We did update a CloudStack 4.20.0 Cluster to 4.20.1
> > > > As expected, shutdown VMs connected to unpatched Isolated networks
> > can't
> > > > start and firewall rules etc can't be changed complaining about a
> > required
> > > > update of the VR.
> > > >
> > > > When i execute the live patching of an isolated network or of a VR, i
> > > > always receive the following error message "Failed to restart network"
> > and
> > > > i see the log entry "Unable to send command. Router requires upgrade"
> > > >
> > > > 2025-06-19 13:03:26,740 WARN [o.a.c.e.o.NetworkOrchestrator]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Failed to implement network Network {"id": 279,
> > "name":
> > > > "TESTNETWORK LIVEPATCH", "uuid":
> > "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > > "networkofferingid": 8} elements and resources as a part of network
> > restart
> > > > due to com.cloud.exception.ResourceUnavailableException: Resource
> > > > [VirtualRouter:922] is unreachable: Unable to send command. Router
> > requires
> > > > upgrade
> > > > 2025-06-19 13:03:26,785 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> > Complete
> > > > async job-23502, jobStatus: FAILED, resultCode: 530, result:
> > > >
> > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> > > > to restart network"}
> > > >
> > > > From the logs, shortly before i see a success message "Successfully
> > > > patched router VM":
> > > >
> > > > 2025-06-19 13:03:26,606 INFO [o.a.c.e.o.NetworkOrchestrator]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Successfully patched router VM instance
> > > >
> > {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"}
> > > >
> > > > Is it because the systemvm template was updated (Debian 11 to Debian
> > 12)
> > > > and live patching can only be done on the same systemvm template
> > version?
> > > >
> > > > 2025-06-19 13:03:26,738 DEBUG [c.c.n.r.NetworkHelperImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Router requires upgrade. Unable to send command to
> > router:
> > > > VM instance
> > > >
> > {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"},
> > > > router template version: Cloudstack Release 4.20.0 Fri Sep 6 03:45:27
> > AM
> > > > UTC 2024, minimal required version: 4.20.1
> > > >
> > > > The only way to get the network/VR fully functional again, allowing
> > > > stopped VMs to start and changing firewall rules etc is to execute a
> > > > network restart with cleanup or execute "Upgrade router to use newer
> > > > template" - which comes with a service impact as expected.
> > > >
> > > > Full log attached to the end of this mail, in case i am missing
> > something
> > > > obvious.
> > > >
> > > > Thanks in advance for helping me to understand the requirements for
> > > > non-service impacting live patching.
> > > >
> > > > Best regards,
> > > > Stephan
> > > >
> > > > ############
> > > > ### full log ###
> > > > ############
> > > >
> > > > 2025-06-19 13:02:57,903 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (qtp1404565079-435:[ctx-ccc0da24, ctx-7ab4ba90]) (logid:64ccbfcd)
> > submit
> > > > async job-23502, details: AsyncJob
> > > >
> > {"accountId":4,"cmd":"org.apache.cloudstack.api.command.user.network.RestartNetworkCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"9\",\"livepatch\":\"true\",\"sessionkey\":\"1234567890abcdefg\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"82573\",\"id\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"ctxDetails\":\"{\\\"interface
> > > >
> > com.cloud.network.Network\\\":\\\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\\\"}\",\"ctxAccountId\":\"4\",\"uuid\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"cmdEventType\":\"NETWORK.RESTART\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":23502,"initMsid":33830401474176,"instanceId":279,"instanceType":"Network","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":9,"uuid":"c02acb59-37e0-4bba-bd8e-82294c6
> > > > 6c505"}
> > > > 2025-06-19 13:02:57,908 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> > Executing
> > > > AsyncJob
> > > >
> > {"accountId":4,"cmd":"org.apache.cloudstack.api.command.user.network.RestartNetworkCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"9\",\"livepatch\":\"true\",\"sessionkey\":\"1234567890abcdefg\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"82573\",\"id\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"ctxDetails\":\"{\\\"interface
> > > >
> > com.cloud.network.Network\\\":\\\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\\\"}\",\"ctxAccountId\":\"4\",\"uuid\":\"8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7\",\"cmdEventType\":\"NETWORK.RESTART\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":23502,"initMsid":33830401474176,"instanceId":279,"instanceType":"Network","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":9,"uuid":"c02acb59-37e0-4bba-bd8e-82294c66c505"}
> > > > 2025-06-19 13:02:57,948 DEBUG [c.c.u.AccountManagerImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Account [Account
> > > >
> > [{"accountName":"Cloudstack_Admins","id":4,"uuid":"742dba95-8709-4aba-92c8-fb472e118ddb"}]]
> > > > has access to resource.
> > > > 2025-06-19 13:02:57,961 DEBUG [c.c.a.ApiServlet]
> > > > (qtp1404565079-435:[ctx-d1310f58]) (logid:bf1fd0ff) ===START===
> > 192.168.1.5
> > > > -- GET
> > > >
> > jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> > > > 2025-06-19 13:02:58,007 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Restarting network Network {"id": 279, "name":
> > > > "TESTNETWORK LIVEPATCH", "uuid":
> > "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > > "networkofferingid": 8}...
> > > > 2025-06-19 13:02:58,039 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Wait time setting on
> > > > com.cloud.agent.api.PatchSystemVmCommand is 600000 seconds
> > > > 2025-06-19 13:02:58,046 DEBUG [c.c.a.m.ClusteredAgentAttache]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Seq 3902-2730025799116652757: Forwarding Seq
> > > > 3986-2730025799116652757: { Cmd , MgmtId: 33830401474176, via:
> > > > 3986(kvm-001), Ver: v1, Flags: 100011,
> > > >
> > [{"com.cloud.agent.api.PatchSystemVmCommand":{"forced":"true","accessDetails":{"
> > > > router.name
> > ":"r-922-VM","router.ip":"169.254.77.6"},"wait":"0","bypassHostMaintenance":"false"}}]
> > > > } to 24728157879110
> > > > 2025-06-19 13:02:58,050 DEBUG [c.c.a.ApiServlet]
> > > > (qtp1404565079-435:[ctx-d1310f58, ctx-531f992f]) (logid:bf1fd0ff)
> > ===END===
> > > > 192.168.1.5 -- GET
> > > >
> > jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> > > > 2025-06-19 13:03:26,591 DEBUG [c.c.a.t.Request]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Seq 3986-2730025799116652757: Received: { Ans: ,
> > MgmtId:
> > > > 33830401474176, via: 3986(kvm-001), Ver: v1, Flags: 10, {
> > > > PatchSystemVmAnswer } }
> > > > 2025-06-19 13:03:26,591 INFO [c.c.s.ManagementServerImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Successfully patched system VM r-922-VM
> > > > 2025-06-19 13:03:26,606 INFO [o.a.c.e.o.NetworkOrchestrator]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Successfully patched router VM instance
> > > >
> > {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"}
> > > > 2025-06-19 13:03:26,607 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Implementing the network Network {"id": 279, "name":
> > > > "TESTNETWORK LIVEPATCH", "uuid":
> > "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > > "networkofferingid": 8} elements and resources as a part of network
> > restart
> > > > without cleanup
> > > > 2025-06-19 13:03:26,638 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Asking VirtualRouter to implement Network {"id": 279,
> > > > "name": "TESTNETWORK LIVEPATCH", "uuid":
> > > > "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8}
> > > > 2025-06-19 13:03:26,668 DEBUG [c.c.a.ApiServlet]
> > > > (qtp1404565079-321:[ctx-fcacff74]) (logid:a66d7bee) ===START===
> > 192.168.1.5
> > > > -- GET
> > > >
> > jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> > > > 2025-06-19 13:03:26,704 DEBUG [c.c.n.NetworkModelImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Service SecurityGroup is not supported in the network
> > > > Network {"id": 279, "name": "TESTNETWORK LIVEPATCH", "uuid":
> > > > "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7", "networkofferingid": 8}
> > > > 2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Looking for vms/cloud-scripts.tgz in the classpath
> > > > 2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) System resource:
> > > > file:/usr/share/cloudstack-common/vms/cloud-scripts.tgz
> > > > 2025-06-19 13:03:26,733 DEBUG [c.c.u.s.Script]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Absolute path =
> > > > /usr/share/cloudstack-common/vms/cloud-scripts.tgz
> > > > 2025-06-19 13:03:26,738 DEBUG [c.c.n.r.NetworkHelperImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Router requires upgrade. Unable to send command to
> > router:
> > > > VM instance
> > > >
> > {"id":922,"instanceName":"r-922-VM","state":"Running","type":"DomainRouter","uuid":"79a93a76-b1c7-4627-ae4c-d9b1f0a6bdfa"},
> > > > router template version: Cloudstack Release 4.20.0 Fri Sep 6 03:45:27
> > AM
> > > > UTC 2024, minimal required version: 4.20.1
> > > > 2025-06-19 13:03:26,740 WARN [o.a.c.e.o.NetworkOrchestrator]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Failed to implement network Network {"id": 279,
> > "name":
> > > > "TESTNETWORK LIVEPATCH", "uuid":
> > "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > > "networkofferingid": 8} elements and resources as a part of network
> > restart
> > > > due to com.cloud.exception.ResourceUnavailableException: Resource
> > > > [VirtualRouter:922] is unreachable: Unable to send command. Router
> > requires
> > > > upgrade
> > > > 2025-06-19 13:03:26,741 WARN [c.c.n.NetworkServiceImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Network Network {"id": 279, "name": "TESTNETWORK
> > > > LIVEPATCH", "uuid": "8a8d2f9b-98c5-4509-a9a7-8b0a603c55c7",
> > > > "networkofferingid": 8} failed to restart.
> > > > 2025-06-19 13:03:26,741 WARN [o.a.c.m.w.WebhookServiceImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Skipping delivering event Event
> > > >
> > {"description":"{\"event\":\"NETWORK.RESTART\",\"status\":\"Completed\"}","eventId":null,"eventType":"NETWORK.RESTART","eventUuid":null,"resourceType":"Network","resourceUUID":null}
> > > > to any webhook as account ID is missing
> > > > 2025-06-19 13:03:26,742 WARN [o.a.c.f.e.EventDistributorImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502, ctx-c69b9b6d])
> > > > (logid:c02acb59) Failed to publish event [category: ActionEvent, type:
> > > > NETWORK.RESTART] on bus webhookEventBus
> > > > 2025-06-19 13:03:26,752 DEBUG [c.c.a.ApiServlet]
> > > > (qtp1404565079-321:[ctx-fcacff74, ctx-dc88a930]) (logid:a66d7bee)
> > ===END===
> > > > 192.168.1.5 -- GET
> > > >
> > jobId=c02acb59-37e0-4bba-bd8e-82294c66c505&command=queryAsyncJobResult&response=json&sessionkey=1234567890abcdefg
> > > > 2025-06-19 13:03:26,785 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> > Complete
> > > > async job-23502, jobStatus: FAILED, resultCode: 530, result:
> > > >
> > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> > > > to restart network"}
> > > > 2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> > Publish
> > > > async job-23502 complete on message bus
> > > > 2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Wake
> > up
> > > > jobs related to job-23502
> > > > 2025-06-19 13:03:26,789 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> > Update db
> > > > status for job-23502
> > > > 2025-06-19 13:03:26,793 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Wake
> > up
> > > > jobs joined with job-23502 and disjoin all subjobs created from job-
> > 23502
> > > > 2025-06-19 13:03:26,824 DEBUG [c.c.a.ApiServer]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59)
> > Retrieved
> > > > cmdEventType from job info: NETWORK.RESTART
> > > > 2025-06-19 13:03:26,832 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Done
> > > > executing
> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> > > > for job-23502
> > > > 2025-06-19 13:03:26,832 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > > (API-Job-Executor-27:[ctx-1be8a96f, job-23502]) (logid:c02acb59) Remove
> > > > job-23502 from job monitoring
> >
