daviftorres commented on issue #11552:
URL: https://github.com/apache/cloudstack/issues/11552#issuecomment-3250303707
I got this same error after upgrading 3 different non-prod environments. I
even nuked a couple and installed 421 from scratch (no upgrade) and the issue
remains.
```
2025-09-03 14:24:37,591 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Processing command:
com.cloud.agent.api.StartCommand
2025-09-03 14:24:37,591 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Creating VM from specifications
[VM {id: "3", name: "s-3-VM", uuid: "aef0e83b-f8a6-46b5-a77d-85e36c17f2f7",
type: "SecondaryStorageVm"}]
2025-09-03 14:24:37,591 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Configuring VM with UUID
[aef0e83b-f8a6-46b5-a77d-85e36c17f2f7].
2025-09-03 14:24:37,592 DEBUG [agent.properties.AgentPropertiesFileHandler]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Property
[vm.memballoon.stats.period] was altered. Now using the value [5].
2025-09-03 14:24:37,592 WARN [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Setting System VM's [VM {id: "3",
name: "s-3-VM", uuid: "aef0e83b-f8a6-46b5-a77d-85e36c17f2f7", type:
"SecondaryStorageVm"}] current memory as max memory [524288].
2025-09-03 14:24:37,592 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) This host utilizes cgroupv2 (as
the max shares value is [104000]), thus, the VM requested shares of [500] will
be converted to consider the host limits; the new CPU shares value is [49].
2025-09-03 14:24:37,592 DEBUG [agent.properties.AgentPropertiesFileHandler]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Property [hypervisor.uri] has
empty or null value. Using default value [null].
2025-09-03 14:24:37,592 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection
at: qemu:///system
2025-09-03 14:24:37,593 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Trying to fetch storage pool
3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 from libvirt
2025-09-03 14:24:37,593 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection
at: qemu:///system
2025-09-03 14:24:37,598 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully refreshed pool
3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 Capacity: (1.4292 TB) 1571370827776 Used:
(431.42 GB) 463237808128 Available: (1.0078 TB) 1108133019648
2025-09-03 14:24:37,602 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection
at: qemu:///system
2025-09-03 14:24:37,603 DEBUG [utils.script.Script]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Executing command [qemu-img info
-U
/mnt/3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85/84ee5be8-5b28-4f99-b453-afb3fc4f7ae3
].
2025-09-03 14:24:37,611 DEBUG [utils.script.Script]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully executed process
[13862] for command [qemu-img info -U
/mnt/3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85/84ee5be8-5b28-4f99-b453-afb3fc4f7ae3
].
2025-09-03 14:24:37,611 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Disk IO driver policy [null]. The
host supports the io_uring policy [true]
2025-09-03 14:24:37,612 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Trying to fetch storage pool
3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 from libvirt
2025-09-03 14:24:37,612 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection
at: qemu:///system
2025-09-03 14:24:37,616 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully refreshed pool
3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 Capacity: (1.4292 TB) 1571370827776 Used:
(431.42 GB) 463237808128 Available: (1.0078 TB) 1108133019648
2025-09-03 14:24:37,616 DEBUG [kvm.resource.BridgeVifDriver]
(AgentRequest-Handler-5:[]) (logid:9b1fb285)
nic=[Nic:Control-169.254.27.218-null]
2025-09-03 14:24:37,616 DEBUG [utils.script.Script]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Executing command [/bin/bash -c ip
route | grep 169.254.0.0/16 ].
2025-09-03 14:24:37,623 DEBUG [utils.script.Script]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully executed process
[13865] for command [/bin/bash -c ip route | grep 169.254.0.0/16 ].
2025-09-03 14:24:37,623 DEBUG [kvm.resource.BridgeVifDriver]
(AgentRequest-Handler-5:[]) (logid:9b1fb285)
nic=[Nic:Management-10.120.0.218-null]
2025-09-03 14:24:37,623 DEBUG [kvm.resource.BridgeVifDriver]
(AgentRequest-Handler-5:[]) (logid:9b1fb285)
nic=[Nic:Public-10.120.1.222-vlan://1201]
2025-09-03 14:24:37,623 DEBUG [kvm.resource.BridgeVifDriver]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) creating a vNet dev and bridge for
public traffic per traffic label cloudbr1
2025-09-03 14:24:37,623 DEBUG [kvm.resource.BridgeVifDriver]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Executing command
[/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 1201 -p
eth3 -b breth3-1201 -o add ].
2025-09-03 14:24:37,642 DEBUG [kvm.resource.BridgeVifDriver]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully executed process
[13868] for command
[/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 1201 -p
eth3 -b breth3-1201 -o add ].
2025-09-03 14:24:37,642 DEBUG [kvm.resource.BridgeVifDriver]
(AgentRequest-Handler-5:[]) (logid:9b1fb285)
nic=[Nic:Storage-172.25.157.232-null]
2025-09-03 14:24:37,642 DEBUG [kvm.storage.KVMStoragePoolManager]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Disconnecting disk
84ee5be8-5b28-4f99-b453-afb3fc4f7ae3
2025-09-03 14:24:37,642 INFO [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Trying to fetch storage pool
3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 from libvirt
2025-09-03 14:24:37,642 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Looking for libvirtd connection
at: qemu:///system
2025-09-03 14:24:37,647 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Successfully refreshed pool
3c8a09fe-56cb-3263-bf8e-0abcbbd5cf85 Capacity: (1.4292 TB) 1571370827776 Used:
(431.42 GB) 463237808128 Available: (1.0078 TB) 1108133019648
2025-09-03 14:24:37,647 WARN [cloud.agent.Agent]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Caught:
java.lang.NullPointerException: Cannot invoke "java.net.URI.getScheme()"
because "uri" is null
at
com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:180)
at
com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:234)
at
com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:258)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:3816)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3395)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:86)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:52)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2240)
at com.cloud.agent.Agent.processRequest(Agent.java:813)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2025-09-03 14:24:37,648 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-5:[]) (logid:9b1fb285) Seq 1-2231252140385370214: { Ans:
, MgmtId: 90520744490122, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException:
Cannot invoke "java.net.URI.getScheme()" because "uri" is null
at
com.cloud.network.Networks$BroadcastDomainType.getSchemeValue(Networks.java:180)
at
com.cloud.network.Networks$BroadcastDomainType.getValue(Networks.java:234)
at
com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:258)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:3816)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3395)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:86)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:52)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2240)
at com.cloud.agent.Agent.processRequest(Agent.java:813)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.Answer":{"result":"false","details":"Stopped
by previous failure","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-03 14:24:37,713 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Request:Seq 1-2231252140385370215:
{ Cmd , MgmtId: 90520744490122, via: 1, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"169.254.27.218","forceStop":"false","expungeVM":"false","volumesToDisconnect":[],"vmName":"s-3-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2025-09-03 14:24:37,713 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Processing command:
com.cloud.agent.api.StopCommand
2025-09-03 14:24:37,713 DEBUG [resource.wrapper.LibvirtStopCommandWrapper]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) backing up the cmdline
2025-09-03 14:24:40,789 DEBUG [resource.wrapper.LibvirtStopCommandWrapper]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to backup cmdline file due
to There was a problem while connecting to 169.254.27.218:3922
2025-09-03 14:24:40,790 DEBUG [agent.properties.AgentPropertiesFileHandler]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Property [hypervisor.uri] has
empty or null value. Using default value [null].
2025-09-03 14:24:40,790 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Looking for libvirtd connection
at: qemu:///system
2025-09-03 14:24:40,791 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Can not find KVM connection for
Instance: s-3-VM, continuing.
2025-09-03 14:24:40,791 DEBUG [agent.properties.AgentPropertiesFileHandler]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Property [hypervisor.uri] has
empty or null value. Using default value [null].
2025-09-03 14:24:40,791 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Looking for libvirtd connection
at: lxc:///
2025-09-03 14:24:40,791 INFO [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) No existing libvirtd connection
found. Opening a new one
2025-09-03 14:24:40,791 DEBUG [agent.properties.AgentPropertiesFileHandler]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Property [libvirt.events.enabled]
has empty or null value. Using default value [false].
2025-09-03 14:24:40,791 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Libvirt event listening is
disabled, not setting up event loop
2025-09-03 14:24:40,792 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Can not find LXC connection for
Instance: s-3-VM, continuing.
2025-09-03 14:24:40,792 WARN [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Can not find a connection for
Instance s-3-VM. Assuming the default connection.
2025-09-03 14:24:40,792 DEBUG [kvm.resource.LibvirtConnection]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Looking for libvirtd connection
at: qemu:///system
2025-09-03 14:24:40,793 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching name
's-3-VM'
2025-09-03 14:24:40,794 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching name
's-3-VM'
2025-09-03 14:24:40,794 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching name
's-3-VM'
2025-09-03 14:24:40,794 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Executing command
[/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname s-3-VM ].
2025-09-03 14:24:40,905 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Successfully executed process
[13875] for command
[/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname s-3-VM ].
2025-09-03 14:24:40,905 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-4:[]) (logid:9b1fb285)
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:828:
SyntaxWarning: invalid escape sequence '\-'
brName = execute("iptables-save |grep physdev-is-bridged |grep FORWARD
|grep BF |grep '\-o' |awk '{print $4}' | head -1").strip()
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1273:
SyntaxWarning: invalid escape sequence '\-'
cmd = "iptables-save |grep physdev-is-bridged |grep FORWARD |grep BF |grep
'\-o' | grep -w " + brname + "|awk '{print $9}' | head -1"
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1331:
SyntaxWarning: invalid escape sequence '\('
refs = int(execute("""iptables -n -L %s | awk '/%s(.*)references/
{gsub(/\(/, "") ;print $3}'""" % (brfw,brfw)).strip())
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1332:
SyntaxWarning: invalid escape sequence '\('
refs_in = int(execute("""iptables -n -L %s-IN | awk '/%s-IN(.*)references/
{gsub(/\(/, "") ;print $3}'""" % (brfw,brfw)).strip())
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1333:
SyntaxWarning: invalid escape sequence '\('
refs_out = int(execute("""iptables -n -L %s-OUT | awk
'/%s-OUT(.*)references/ {gsub(/\(/, "") ;print $3}'""" % (brfw,brfw)).strip())
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:1334:
SyntaxWarning: invalid escape sequence '\('
refs6 = int(execute("""ip6tables -n -L %s | awk '/%s(.*)references/
{gsub(/\(/, "") ;print $3}'""" % (brfw,brfw)).strip())
2025-09-03 14:24:40,906 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Failed to get vm :Domain not
found: no domain with matching name 's-3-VM'
2025-09-03 14:24:40,906 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Try to stop the vm at first
2025-09-03 14:24:40,906 DEBUG [kvm.resource.LibvirtComputingResource]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) VM s-3-VM doesn't exist, no need
to stop it
2025-09-03 14:24:40,906 WARN [kvm.resource.LibvirtKvmAgentHook]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Groovy script
'/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy' is not available.
Transformations will not be applied.
2025-09-03 14:24:40,906 WARN [kvm.resource.LibvirtKvmAgentHook]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Shell script
'/etc/cloudstack/agent/hooks/libvirt-vm-state-change.sh' is not available.
Transformations will not be applied.
2025-09-03 14:24:40,906 WARN [kvm.resource.LibvirtKvmAgentHook]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Groovy scripting engine is not
initialized. Data transformation skipped.
2025-09-03 14:24:40,906 DEBUG [kvm.resource.LibvirtKvmAgentHook]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) No shell script provided for
transformation. Data transformation skipped.
2025-09-03 14:24:40,906 DEBUG [cloud.agent.Agent]
(AgentRequest-Handler-4:[]) (logid:9b1fb285) Seq 1-2231252140385370215: { Ans:
, MgmtId: 90520744490122, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
