xuanyuanaosheng edited a comment on issue #6169: URL: https://github.com/apache/cloudstack/issues/6169#issuecomment-1080402721
@rohityadavcloud 1. The primary and secondary storage is OK. 2. The zone is enabled and the host is Up.  3. The systemvmtemplate state is OK.  4. The cmd is hang on  5. The storage vm start cmd ``` # ps -ef | grep qemu | grep s-25790-VM root 1624320 1 99 17:32 ? 00:43:10 /usr/libexec/qemu-kvm -name guest=s-25790-VM,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-s-25790-VM/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu qemu64 -m 512 -overcommit mem-lock=off -smp 1,sockets=1,cores=1,threads=1 -uuid b127ae7b-3687-4546-aad1-b90e6b7f931a -smbios type=1,manufacturer=Apache Software Foundation,product=CloudStack KVM Hypervisor,uuid=b127ae7b-3687-4546-aad1-b90e6b7f931a -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=40,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/f502f95a-2360-4b2f-a6e3-bd2ac97fd050","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":fa lse},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/32b9cda5-e967-4965-92d6-7746c91fd335","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"} -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=32b9cda5e967496592d6 -blockdev {"driver":"file","filename":"/usr/share/cloudstack-common/vms/systemvm.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-stora ge"} -device ide-cd,bus=ide.1,unit=0,drive=libvirt-1-format,id=ide0-1-0,bootindex=1 -netdev tap,fd=42,id=hostnet0,vhost=on,vhostfd=43 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=0e:00:a9:fe:a8:6b,bus=pci.0,addr=0x3,rombar=0,romfile= -netdev tap,fd=44,id=hostnet1,vhost=on,vhostfd=45 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=1e:00:df:00:00:56,bus=pci.0,addr=0x4,rombar=0,romfile= -netdev tap,fd=46,id=hostnet2,vhost=on,vhostfd=47 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=1e:00:62:00:00:1c,bus=pci.0,addr=0x5,rombar=0,romfile= -netdev tap,fd=48,id=hostnet3,vhost=on,vhostfd=49 -device virtio-net-pci,netdev=hostnet3,id=net3,mac=1e:00:74:00:00:85,bus=pci.0,addr=0x6,rombar=0,romfile= -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,fd=50,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vn c 10.26.246.8:0,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device i6300esb,id=watchdog0,bus=pci.0,addr=0xa -watchdog-action none -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on ``` The cloudstack agent error logs is ``` 2022-03-28 17:37:08,609 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-5:null) (logid:) Looking for libvirtd connection at: qemu:///system 2022-03-28 17:37:08,627 DEBUG [cloud.agent.Agent] (UgentTask-5:null) (logid:) Sending ping: Seq 30-14: { Cmd , MgmtId: -1, via: 30, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"s-25790-VM":{"state":"PowerOn","host":"whdckvm004.cn.prod"},"v-25789-VM":{"state":"PowerOn","host":"whdckvm004.cn.prod"}},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"30","wait":"0","bypassHostMaintenance":"false"}}] } 2022-03-28 17:37:08,835 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) (logid:) Received response: Seq 30-14: { Ans: , MgmtId: 345052215515, via: 30, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":"30","wait":"0","bypassHostMaintenance":"false"},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2022-03-28 17:37:36,571 WARN [kvm.resource.LibvirtComputingResource] (Script-3:null) (logid:) Interrupting script. 2022-03-28 17:37:36,573 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:32001972) Timed out: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh -n s-25790-VM -c template=domP type=secstorage host=10.25.2.173 port=8250 name=s-25790-VM zone=6 pod=5 guid=s-25790-VM workers=5 resource=com.cloud.storage.resource.PremiumSecondaryStorageResource instance=SecStorage sslcopy=false role=templateProcessor mtu=1500 eth2ip=10.26.13.58 eth2mask=255.255.255.0 gateway=10.26.13.254 eth0ip=169.254.168.107 eth0mask=255.255.0.0 eth1ip=10.26.246.66 eth1mask=255.255.255.0 mgmtcidr=10.25.2.0/24 localgw=10.26.246.254 private.network.device=eth1 eth3ip=10.26.246.14 eth3mask=255.255.255.0 storageip=10.26.246.14 storagenetmask=255.255.255.0 storagegateway=10.26.246.254 internaldns1=10.67.9.201 internaldns2=10.25.9.22 dns1=10.25.9.21 dns2=10.67.230.8 nfsVersion=null . Output is: 2022-03-28 17:37:36,574 ERROR [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:32001972) Passing cmdline failed:timeout 2022-03-28 17:37:36,574 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:32001972) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh -n s-25790-VM -c template=domP type=secstorage host=10.25.2.173 port=8250 name=s-25790-VM zone=6 pod=5 guid=s-25790-VM workers=5 resource=com.cloud.storage.resource.PremiumSecondaryStorageResource instance=SecStorage sslcopy=false role=templateProcessor mtu=1500 eth2ip=10.26.13.58 eth2mask=255.255.255.0 gateway=10.26.13.254 eth0ip=169.254.168.107 eth0mask=255.255.0.0 eth1ip=10.26.246.66 eth1mask=255.255.255.0 mgmtcidr=10.25.2.0/24 localgw=10.26.246.254 private.network.device=eth1 eth3ip=10.26.246.14 eth3mask=255.255.255.0 storageip=10.26.246.14 storagenetmask=255.255.255.0 storagegateway=10.26.246.254 internaldns1=10.67.9.201 internaldns2=10.25.9.22 dns1=10.25.9.21 dns2=10.67.230.8 nfsVersion=null 2022-03-28 17:37:36,576 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:32001972) Executing while with timeout : 300000 2022-03-28 17:37:36,576 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:32001972) We are interrupted but it's not a timeout, just continue 2022-03-28 17:37:36,576 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:32001972) Executing while with timeout : 300000 2022-03-28 17:37:42,026 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:733569db) Processing command: com.cloud.agent.api.GetStorageStatsCommand 2022-03-28 17:37:42,027 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:733569db) Trying to fetch storage pool 8b051a3e-2a52-3f22-8840-85f235a75f40 from libvirt ``` [agent.log](https://github.com/apache/cloudstack/files/8361501/agent.log) @rohityadavcloud Could you please give some advices? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
