SSVM and CPVM agent unable to start after console proxy SSL certificate update

[Cloud List]

Hi,

Merry Christmas to all.

We are using Cloudstack with KVM hypervisor. Since our console proxy SSL
certificate has expired, we updated our new SSL certificate using below
method:

http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy

We have done the above method in the past years without any issues, however
this time round, both the SSVM and CPVM agents are not able to start after
the update.

The state for both VMs are up but agents are in "disconnected" state. We
are still able to login to the SSVM, and found out that the cloud service
is not running.

root@s-4200-VM:~# service cloud status
CloudStack cloud service is not running

Tried to start the service:

root@s-4200-VM:~# service cloud start
Starting CloudStack cloud service (type=secstorage) Success

But the service is not started:

root@s-4200-VM:~# service cloud status
CloudStack cloud service is not running

Below is the logs from /var/log/cloud.log:

=
Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config
Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest
Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit
the loop and proceed with configuration.
Sat Dec 26 03:45:04 UTC 2020 Patching  cloud service
Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml
Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm
Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP
Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip
timer=0
Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP
Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP
Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP
Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with
STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR:
Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to 10.19.16.1
Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server
Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of
volume/template
Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file
/etc/apache2/sites-available/default-ssl
Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file:
/etc/apache2/sites-available/default-ssl
Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter
Sat Dec 26 03:45:21 UTC 2020 disable rpfilter
Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0
Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0
Sat Dec 26 03:45:21 UTC 2020 Processors = 1  Enable service  = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1
=

Result of /usr/local/cloud/systemvm/ssvm-check.sh:

=
root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh

First DNS server is  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 48 data bytes
56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms
56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms
Good: Can ping DNS server

Good: DNS resolves download.cloud.com

ERROR: NFS is not currently mounted
Try manually mounting from inside the VM
NFS server is  X.X.201.1
PING X.X.201.1 (X.X.201.1): 48 data bytes
56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms
56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms
--- X.X.201.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms
Good: Can ping nfs server

Management server is 10.237.3.8. Checking connectivity.
Good: Can connect to management server port 8250

ERROR: Java process not running.  Try restarting the SSVM.
root@s-4200-VM:/var/log#
=

The result is OK except the NFS test, but we checked the IP address is not
correct (X.X.201.1 which is the public IP address of the gateway rather
than the actual NFS server IP). We tested mounting to the actual NFS server
and it works fine.

Have tried stopping and starting back the SSVM and the issue still persists.

Anyone can help to advice how we can resolve the problem?

Looking forward to your reply, thank you.

-ip-


Virus-free.
www.avg.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: Network for instance

[Thanh Đức]

Hi Darrin,

Because I deploy cloudstack server. NFS server and KVM server in same local IP, 
can I assign public IP for instance ?

On 2020/12/17 14:00:11, Darrin Hüsselmann  
wrote: 
> Hi Duc,
> 
> Have you set egress rules for your network? It certainly is possible to ping 
> instances on your network.
> 
> Regards
> Darrin
> 
> From: Thanh Đức 
> Sent: Thursday, December 17, 2020 3:28 PM
> To: users@cloudstack.apache.org 
> Subject: Network for instance
> 
> Hi all,
> 
> I tried to create instance on laptop (A) with cloudstack server , nfs server 
> and kvm host same server IP.
> 
> I already create instance. I use another laptop (B) connect same wifi. I 
> tried use laptop (B) ping instance already created by laptop (A) but it can 
> not.
> I ping via LAN
> 
> It is possible ?
> 
> darrin.husselm...@shapeblue.com 
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>   
>  
> 
>