Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

urs - Original Message - From: Ammad Syed [mailto:syedamma...@gmail.com] To: Sent: Wed, 14 Oct 2020 13:16:35 +0500 Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error Hi, I have performed an upgrade on one of my other production environments and have three XS7.0 clusters with latest hotfi

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

ing has also same > md5sum > >> of > >> >>> systemvm iso. The iso is getting copied successfully. The problem > >> looks > >> >>> somewhere else. > >> >>> > >> >>> I have checked in xenserver logs as well

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

gt;>> Sent from my iPhone >> >>> >> >>> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva < >> pearl.dsi...@shapeblue.com> >> >>> wrote: >> >>> > >> >>> > Hi Ammad, >> >>> > >&g

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

;>> > > >>> > Hi Ammad, > >>> > > >>> > Is the understanding right that the steps as mentioned by you in the > >>> previous mail has in-fact worked on one zone? If that's the case, > could you > >>> please ensu

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

timestamps as well and >>> comparing the checksums against the iso on the management server, so that >>> when the system VM's are recreated, they pick up the new iso. >>> > >>> > Thanks, >>> > Pearl >>> > >>> >

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

anagement server, so that >> when the system VM's are recreated, they pick up the new iso. >> > >> > Thanks, >> > Pearl >> > >> > >> > From: Ammad Syed >> > Sent: Sunday, September 1

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

are recreated, they pick up the new iso. > > > > Thanks, > > Pearl > > > > ________________ > > From: Ammad Syed > > Sent: Sunday, September 13, 2020 12:49 PM > > To: users@cloudstack.apache.org > > Subject: Re: Cloudstack 4.1

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

13, 2020 12:49 PM > To: users@cloudstack.apache.org > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error > > Hi Andrija, > > Here is the permission of mount point in SSVM. > > root@s-25437-VM:~# df -h > Filesystem Size Used Avail Use% Mounted on > ude

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

ubject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error Hi Andrija, Here is the permission of mount point in SSVM. root@s-25437-VM:~# df -h Filesystem Size Used Avail Use% Mounted on udev 233M 0 233M 0% /dev tmpfs 98M 19M 80M 19% /run

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Hi Andrija, Here is the permission of mount point in SSVM. root@s-25437-VM:~# df -h Filesystem Size Used Avail Use% Mounted on udev 233M 0 233M 0% /dev tmpfs 98M 19M 80M 19% /run /dev/xvda5 1.1G 773M 282M 74% / tm

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Can you share permissions of your secondary storage (mount it then ls -lah the mount point) On Fri, 11 Sep 2020 at 01:39, Ammad Syed wrote: > Hi Andrija, > > I have performed an upgrade on my production system from 4.11.3 to 4.13.1. > Even I cleared the tags but the issue is still there. > > I h

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Hi Andrija, I have performed an upgrade on my production system from 4.11.3 to 4.13.1. Even I cleared the tags but the issue is still there. I have four zones and only in one zone and on specific host, the systemVM's agent goes up but on all other zones, key injection to systemVM is still failing

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

so should get propagated for all. > Check > > > timestamp. > > > 3. 3. Restart systemvms and they should get the iso inserted. > > > 4. > > > > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso#SystemVm.

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

should get the iso inserted. > > 4. > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso#SystemVm.iso-Xen > > > > Thanks, > > Pearl > > > > > > From: Ammad Syed > > Sent: We

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

tart systemvms and they should get the iso inserted. > 4. > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso#SystemVm.iso-Xen > > Thanks, > Pearl > > > From: Ammad Syed > Sent: Wednesday, September 2, 2020 9:12 PM >

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Ammad Syed Sent: Wednesday, September 2, 2020 9:12 PM To: users@cloudstack.apache.org Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error Guys, please advise how to troubleshoot this further ? How can i enable trace level debugging ? any help in this will be highly appreciated? Ammad pear

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Guys, please advise how to troubleshoot this further ? How can i enable trace level debugging ? any help in this will be highly appreciated? Ammad > On 27-Aug-2020, at 1:41 PM, Vivek Kumar > wrote: > > Pardon, the whole mail chain was not visible so replied accordingly. It > seems like you

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Pardon, the whole mail chain was not visible so replied accordingly. It seems like you are doing SSH using root shell so it doesn’t matter. Vivek Kumar Manager - Cloud & DevOps IndiQus Technologies 24*7 O +91 11 4055 1411 | M +91 7503460090 www.indiqus.com This messa

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Hello Ammad, You haven’t defined the user while logging to the system VM through the Link local IP that’s why you have got the error while logging to the system VM. >> ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922 Try instead below >> ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.25

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Guys, any help to troubleshoot this further would be highly appreciated. Ammad Ali > On 18-Aug-2020, at 1:05 PM, Ammad Syed wrote: > >  > Hi, > > The systemVM is accessible with 169.x.x.x IP from xenserver host. But login > with private key is denied. I am testing this on my test environment

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Hi, The systemVM is accessible with 169.x.x.x IP from xenserver host. But login with private key is denied. I am testing this on my test environment. [root@xenserver ~]# ping 169.254.0.121 PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data. 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

It was my tool (GUI based tool, that still doesn't return any line (???) but I see it. Problem is with 169.254.1.178 not being reachable over SSH (2020-07-25 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command in VR: /opt/cloud/bin/ro

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

I have rechecked the logs are there. Please review below lines. C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors (27 hits) Line 409969: 2020-07-25 01:45:52,320 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-58:ctx-9dc75334 job-1207969/job-1208104

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Ammad, no such log lines in the file you uploaded? On Mon, 17 Aug 2020 at 09:02, Ammad Syed wrote: > Hi Andrija, > > The error occurred only one time. Can you please try to grep "Failed to > setup keystore and generate CSR" you will see all the systemVMs that I > tried to create failed becau

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Hi Andrija, The error occurred only one time. Can you please try to grep "Failed to setup keystore and generate CSR" you will see all the systemVMs that I tried to create failed because of certificate generation failed. OR you can grep job-1208321/job-1208452 this job as well. Also I mentioned ab

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner] (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception Failed to fetch any free public IP address com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any free public IP address ? On Sat, 15 Aug 2020 at 18:05, Ammad

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

I have checked further, the key is successfully injected in systemvm.iso on the management node and copied successfully to xenserver. I have checked on systemVM there is no public key found in authorized_keys of root. How can I troubleshoot this further ? how can I enable trace logging for managem

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Hi, I have setup my test environment, here is what I did: - Installed ACS 4.11.1 and added xenserver 7.0 host in it. SystemVMs are up in the zone and agents are up. - Then upgraded the system to 4.11.3. Recreated systemVMs, agent is up and systemVM are up with 4.11.3 systemVM version. - Then upgr

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Here is the link for download management logs. https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed wrote: > I have reverted the version back to 4.11.3. But I have saved logs starting > from upgrade. > > I think the key h

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Do you get an error while trying to inject ssh key into the systemvm.iso (mgmt logs) , or can you confirm that the systemvm.iso on XS host and the one on the mgmt server are identical, md5sum them (i.e. has the iso been copied over to the XS successfully) - this might explain not being able to logi

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Yes this is exactly the same issue that i faced. Sent from my iPhone > On 12-Aug-2020, at 8:35 AM, Eric Lee Green wrote: > > Correct, 4.11.3 template is used for 4.11.3, 4.12, and 4.13. 4.14 moves to > the 4.14.0 template. > > There seems to be something odd happening key-wise sometimes with

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Correct, 4.11.3 template is used for 4.11.3, 4.12, and 4.13. 4.14 moves to the 4.14.0 template. There seems to be something odd happening key-wise sometimes with upgrades from 4.11.3 to 4.13.1 or 4.14.0.   I managed an upgrade from 4.11.3 to 4.13.1 that *almost* worked, but the secondary stora

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Hi, I think 4.12 and 4.13 uses same systemVM template i.e 4.11.3 version, which I already have registered. Currently I am running 4.11.3 version of ACS. MariaDB [cloud]> SELECT id,name,type,cross_zones,state FROM cloud.vm_template WHERE name like '%systemvm-xenserver%' AND removed IS NULL; +-

Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error

Hi Syed, >From 4.12, the systemvm template had to be upgraded because of OS change in the template, moved to a latest version of Debian. Because of that, some VR scripts have changed and make obsolete older version of VRs, so you will most likely have to register an updated systemvm templates and u