I didn't test with lower values, but with 256 RAM and 500Mhz CPU is not working for me. (KVM ubuntu 20.04 ACS 4.15.1 )
Regards, Cristian -----Original Message----- From: Daan Hoogland <daan.hoogl...@gmail.com> Sent: Friday, September 17, 2021 11:14 AM To: users <users@cloudstack.apache.org> Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm thanks for the update Christian, sounds strange that the router would require that much to start but glad it worked. On Wed, Sep 15, 2021 at 1:32 PM <cristian.c@istream.today> wrote: > Hello, > > I have found the issue, after I have changed the CPU and memory for > this " System Offering For Software Router - Local Storage' system > offering the router was deployed in few seconds without any issue. > > CPU mhz from 500 to 2000 and RAM from 256 to 2048. > > > > Regards, > Cristian > > -----Original Message----- > From: cristian.c@istream.today <cristian.c@istream.today> > Sent: Wednesday, September 15, 2021 10:18 AM > To: users@cloudstack.apache.org > Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with > kvm > > Any suggestion on this issue? > > Thank you. > > -----Original Message----- > From: cristian.c@istream.today <cristian.c@istream.today> > Sent: Monday, September 13, 2021 5:40 PM > To: users@cloudstack.apache.org > Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with > kvm > > Hi, > > > Yes, please see : > > root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v > 2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0, > package: 0ubuntu8.13 (Frode Nordahl <frode.nord...@canonical.com> Fri, > 16 Jul 2021 05:16:36 +0000) > 2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001 > 2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutputs:1654 : > outputs=1:stderr > 2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutput:1492 : > output=1:stderr > 2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0, > package: 0ubuntu8.13 (Frode Nordahl <frode.nord...@canonical.com> Fri, > 16 Jul 2021 05:16:36 +0000) > 2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001 > 2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW: > obj=0x55935a296b60 classname=virAccessManager > 2021-09-13 14:33:40.944+0000: 75403: debug : > virAccessManagerNewDriver:111 > : Initialized with stack > 2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW: > obj=0x55935a297f90 classname=virAccessManager > 2021-09-13 14:33:40.944+0000: 75403: debug : > virAccessManagerNewDriver:111 > : Initialized with none > 2021-09-13 14:33:40.944+0000: 75403: info : virObjectRef:386 : OBJECT_REF: > obj=0x55935a296b60 > 2021-09-13 14:33:40.944+0000: 75403: info : virObjectUnref:348 : > OBJECT_UNREF: obj=0x55935a296b60 > 2021-09-13 14:33:40.944+0000: 75403: debug : main:1177 : Decided on > pid file path '/run/libvirtd.pid' > 2021-09-13 14:33:40.944+0000: 75403: debug : main:1187 : Decided on > socket paths '/run/libvirt/libvirt-sock', > '/run/libvirt/libvirt-sock-ro' and '/run/libvirt/libvirt-admin-sock' > 2021-09-13 14:33:40.944+0000: 75403: error : virPidFileAcquirePath:367 : > Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily > unavailable > 2021-09-13 14:33:40.944+0000: 75403: debug : virFileClose:110 : Closed > fd 3 > > > root@usa-nyc-kvm-001:/var/lib/libvirt/images# > /usr/bin/qemu-system-x86_64 --version QEMU emulator version 4.2.1 > (Debian 1:4.2-3ubuntu6.17) Copyright > (c) 2003-2019 Fabrice Bellard and the QEMU Project developers > > > root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a No LSB > modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 20.04.2 LTS > Release: 20.04 > Codename: focal > > > CPU : > vendor_id : AuthenticAMD > cpu family : 23 > model : 49 > model name : AMD EPYC 7552 48-Core Processor > > > BTW: the SSVM and Proxy console is running fine on this node.. > > > > Thank you! > > -----Original Message----- > From: Andrija Panic <andrija.pa...@gmail.com> > Sent: Monday, September 13, 2021 4:40 PM > To: users <users@cloudstack.apache.org> > Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with > kvm > > Can you share qemu/libvirt version/flavour? > > On Mon, 13 Sept 2021 at 14:18, <cristian.c@istream.today> wrote: > > > Hello, > > > > > > > > I'm trying to understand this error, I created new network > > with VPC in a new zone where I added KVM hypervisor, and I have this issue. > > > > > > > > > > > > 2021-09-13 11:36:18,621 DEBUG > > [resource.virtualnetwork.VirtualRoutingResource] > > (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to > > 169.254.52.234 > > > > 2021-09-13 11:36:21,684 DEBUG > > [resource.virtualnetwork.VirtualRoutingResource] > > (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to > > 169.254.52.234 > > > > > > > > Here is the full log : https://pastebin.com/0NwsccUc > > > > > > > > root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922 > > > > Trying 169.254.156.179... > > > > telnet: Unable to connect to remote host: Connection refused ( > > before connection refused, I get no route to host, for sure the nic > > is not up > > ) > > > > root@usa-nyc-kvm-001:~# virsh list > > > > Id Name State > > > > --------------------------- > > > > 2 v-4048-VM running > > > > 4 s-4050-VM running > > > > 7 r-4092-VM running > > > > > > > > root@usa-nyc-kvm-001:~# > > > > > > > > > > > > If I check the log inside the router, I see this (the router is also > > responding very slow, waiting a lot to connect on it via virsh console): > > > > > > > > r-4092-VM login: root > > > > Password: > > > > > > > > Login timed out after 60 seconds. > > > > > > > > > > > > __?.o/ Apache CloudStack SystemVM 4.15.1 > > > > ( )# https://cloudstack.apache.org > > > > (___(_) Debian GNU/Linux 10 r-4093-VM ttyS0 > > > > > > > > r-4093-VM login: root > > > > Password: > > > > Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 > > (2021-01-30) > > x86_64 > > > > > > > > > > > > > > > > root@r-4093-VM:~# cat /var/log/cloud.log > > > > Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config > > > > Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance > > > > Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm > > > > Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module > > already compiled in > > > > Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline > > file from qemu-guest-agent > > > > Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old > > configuration files in /etc/cloudstack/ > > > > Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter > > > > Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system > > vm > > > > Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC > > > > Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1 Enable service = 0 > > > > Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter > > > > Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter > > > > Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1 > > > > Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1 > > > > Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for > > guest VMs > > > > Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm > > > > Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm > > > > 2021-09-13 11:55:13,823 INFO update_config.py :: Processing incoming > > file => cmd_line.json > > > > 2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json > > > > 2021-09-13 11:55:13,858 INFO Continuing with the processing of file > > '/var/cache/cloud/cmd_line.json' > > > > 2021-09-13 11:55:13,877 INFO Command of type cmdline received > > > > 2021-09-13 11:55:13,897 INFO Command of type ips received > > > > 2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0 > > > > 2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0 > > > > 2021-09-13 11:55:13,979 INFO Address found in DataBag ==> > > {u'public_ip': > > u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0', > > u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0', > u'source_nat': > > False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type': > > u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16', > u'gateway': > > u'', u'size': u'16'} > > > > 2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device > eth0 > > already configured > > > > 2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v > secondary > > > > 2021-09-13 11:55:14,010 INFO Wrote edited file > > /etc/dnsmasq.d/cloud.conf > > > > 2021-09-13 11:55:14,010 INFO Updated file in-cache configuration > > > > 2021-09-13 11:55:14,010 INFO Nothing to commit. The > /etc/dhcphosts.txt > > file did not change > > > > 2021-09-13 11:55:14,010 INFO Nothing to commit. The > > /var/lib/misc/dnsmasq.leases file did not change > > > > 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcpopts.txt > > file did not change > > > > 2021-09-13 11:55:14,010 INFO Attempting to delete entries from > > dnsmasq.leases file for VMs which are not on dhcphosts file > > > > 2021-09-13 11:55:14,010 ERROR Caught error while trying to delete > > entries > > from dnsmasq.leases file: [Errno 2] No such file or directory: > > '/etc/dhcphosts.txt' > > > > 2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts > > > > 2021-09-13 11:55:14,044 INFO Updated file in-cache configuration > > > > 2021-09-13 11:55:14,044 INFO Updated hosts file > > > > 2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq > > > > 2021-09-13 11:55:15,409 INFO Service dnsmasq restart > > > > 2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v > secondary > > > > 2021-09-13 11:55:15,832 INFO Nothing to commit. The > > /etc/dnsmasq.d/cloud.conf file did not change > > > > 2021-09-13 11:55:15,957 INFO Nothing to commit. The > /etc/dhcphosts.txt > > file did not change > > > > 2021-09-13 11:55:15,963 INFO Nothing to commit. The > > /var/lib/misc/dnsmasq.leases file did not change > > > > 2021-09-13 11:55:15,963 INFO Nothing to commit. The /etc/dhcpopts.txt > > file did not change > > > > 2021-09-13 11:55:15,967 INFO Executing: systemctl is-active dnsmasq > > > > 2021-09-13 11:55:16,179 INFO Executing: systemctl reload dnsmasq > > > > 2021-09-13 11:55:16,665 INFO Service dnsmasq reload > > > > 2021-09-13 11:55:16,909 INFO Wrote edited file /etc/cron.d/process > > > > 2021-09-13 11:55:17,040 INFO Updated file in-cache configuration > > > > 2021-09-13 11:55:17,063 INFO Executing: iptables-save | grep > > '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES > > > > 2021-09-13 11:55:17,323 INFO Executing: iptables-save | grep '^-A > > FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' | > > bash > > > > 2021-09-13 11:55:17,697 INFO Executing: iptables -F FW_EGRESS_RULES > > > > 2021-09-13 11:55:17,990 INFO Executing: ipset -L | grep Name: | awk > > {'print $2'} | ipset flush > > > > 2021-09-13 11:55:19,272 INFO Executing: ipset -L | grep Name: | awk > > {'print $2'} | ipset destroy > > > > 2021-09-13 11:55:19,833 INFO Executing: iptables-save > > > > 2021-09-13 11:55:20,168 INFO Executing: iptables-save > > > > 2021-09-13 11:55:20,505 INFO Executing: ip6tables-save > > > > 2021-09-13 11:55:20,800 INFO Executing: systemctl stop conntrackd > > > > 2021-09-13 11:55:21,201 INFO Service conntrackd stop > > > > 2021-09-13 11:55:21,230 INFO Executing: systemctl stop keepalived > > > > 2021-09-13 11:55:21,578 INFO Service keepalived stop > > > > 2021-09-13 11:55:21,609 INFO Executing: mount > > > > root@r-4093-VM:~# > > > > > > > > Please ignore the router name, I did multiple tests to extract these > > logs, I always see the same. > > > > > > > > > > > > Regards, > > > > Cristian > > > > > > > > > > > > > > -- > > Andrija Panić > > > > -- Daan
