Ammad, no such log lines in the file you uploaded....? On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedamma...@gmail.com> wrote:
> Hi Andrija, > > The error occurred only one time. Can you please try to grep "Failed to > setup keystore and generate CSR" you will see all the systemVMs that I > tried to create failed because of certificate generation failed. OR you can > grep job-1208321/job-1208452 this job as well. > > Also I mentioned above that same thing is happening in my test environment > that I created 4.11.1 then upgraded 4.11.3 > 4.13.1. > > Ammad Ali > > On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic <andrija.pa...@gmail.com> > wrote: > > > 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner] > > (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception Failed > to > > fetch any free public IP address > > com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any free > > public IP address > > > > ? > > > > On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedamma...@gmail.com> wrote: > > > > > I have checked further, the key is successfully injected in > systemvm.iso > > on > > > the management node and copied successfully to xenserver. I have > checked > > on > > > systemVM there is no public key found in authorized_keys of root. > > > > > > How can I troubleshoot this further ? how can I enable trace logging > for > > > management server to see if there is something problematic happening. > > > > > > Ammad Ali > > > > > > On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <syedamma...@gmail.com> > > wrote: > > > > > > > Hi, > > > > > > > > I have setup my test environment, here is what I did: > > > > > > > > - Installed ACS 4.11.1 and added xenserver 7.0 host in it. SystemVMs > > are > > > > up in the zone and agents are up. > > > > - Then upgraded the system to 4.11.3. Recreated systemVMs, agent is > up > > > and > > > > systemVM are up with 4.11.3 systemVM version. > > > > - Then upgraded the system to 4.13.1, recreated systemVMs are running > > but > > > > agents are not up. > > > > > > > > I have checked md5sum of systemvm.iso on xenserver and management > > server, > > > > both are same. > > > > > > > > [root@xenserver iso]# md5sum > /opt/xensource/packages/iso/systemvm.iso > > > > baba18f156395da3a5d8208727d8f421 > > > /opt/xensource/packages/iso/systemvm.iso > > > > > > > > [root@cloudstack-upgrade vms]# md5sum > > > > /usr/share/cloudstack-common/vms/systemvm.iso > > > > baba18f156395da3a5d8208727d8f421 > > > > /usr/share/cloudstack-common/vms/systemvm.iso > > > > > > > > Also the private key on xenserver root and management server are > same. > > > > > > > > management server > > > > /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud > > > > > > > > xenserver path > > > > /root/.ssh/id_rsa.cloud > > > > > > > > - Ammad Ali > > > > > > > > On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed <syedamma...@gmail.com> > > > wrote: > > > > > > > >> Here is the link for download management logs. > > > >> > > > >> > > > >> > > > > > > https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing > > > >> > > > >> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed <syedamma...@gmail.com> > > > >> wrote: > > > >> > > > >>> I have reverted the version back to 4.11.3. But I have saved logs > > > >>> starting from upgrade. > > > >>> > > > >>> I think the key has been copied successfully in system vm iso. > > > >>> > > > >>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl] > > > >>> (main:null) (logid:) Going to update systemvm iso with generated > > > keypairs > > > >>> if needed > > > >>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl] > > > >>> (main:null) (logid:) Trying to inject public and private keys into > > > systemvm > > > >>> iso > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) (logid:) > > > >>> Looking for vms/systemvm.iso in the classpath > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) (logid:) > > > >>> System resource: file:/usr/share/cloudstack-common/vms/systemvm.iso > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) (logid:) > > > >>> Absolute path = /usr/share/cloudstack-common/vms/systemvm.iso > > > >>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl] > > > >>> (main:null) (logid:) Executing: /bin/bash > > > >>> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh > > > >>> /var/cloudstack/management/.ssh/id_rsa.pub > > > >>> /var/cloudstack/management/.ssh/id_rsa > > > >>> /usr/share/cloudstack-common/vms/systemvm.iso > > > >>> 2020-07-25 00:34:17,636 INFO [c.c.s.ConfigurationServerImpl] > > > >>> (main:null) (logid:) Injected public and private keys into systemvm > > iso > > > >>> with result : null > > > >>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase] > > > >>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying > > > >>> > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > >>> to /opt/xensource/packages/iso on 172.16.2.22 with permission 0644 > > > >>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase] > > > >>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying > > > >>> > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission 0644 > > > >>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase] > > > >>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying > > > >>> > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > >>> to /opt/xensource/packages/iso on 172.16.2.9 with permission 0644 > > > >>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase] > > > >>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying > > > >>> > > > > > > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso > > > >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission 0644 > > > >>> > > > >>> I have attached complete management logs. The start of logs is the > > > start > > > >>> of management server after upgrade of management server from 4.11.3 > > to > > > >>> 4.13.1. > > > >>> > > > >>> Ammad Ali > > > >>> > > > >>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic < > > andrija.pa...@gmail.com > > > > > > > >>> wrote: > > > >>> > > > >>>> Do you get an error while trying to inject ssh key into the > > > systemvm.iso > > > >>>> (mgmt logs) , or can you confirm that the systemvm.iso on XS host > > and > > > >>>> the > > > >>>> one on the mgmt server are identical, md5sum them (i.e. has the > iso > > > been > > > >>>> copied over to the XS successfully) - this might explain not being > > > able > > > >>>> to > > > >>>> login via ssh with your private key. > > > >>>> > > > >>>> Best, > > > >>>> > > > >>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <syedamma...@gmail.com> > > wrote: > > > >>>> > > > >>>> > Yes this is exactly the same issue that i faced. > > > >>>> > > > > >>>> > Sent from my iPhone > > > >>>> > > > > >>>> > > On 12-Aug-2020, at 8:35 AM, Eric Lee Green < > > > >>>> eric.lee.gr...@gmail.com> > > > >>>> > wrote: > > > >>>> > > > > > >>>> > > Correct, 4.11.3 template is used for 4.11.3, 4.12, and 4.13. > > 4.14 > > > >>>> moves > > > >>>> > to the 4.14.0 template. > > > >>>> > > > > > >>>> > > There seems to be something odd happening key-wise sometimes > > with > > > >>>> > upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed an upgrade > > > from > > > >>>> > 4.11.3 to 4.13.1 that *almost* worked, but the secondary storage > > VM > > > >>>> > wouldn't work and thus I couldn't spawn new virtual machines. > Same > > > >>>> symptom > > > >>>> > -- key error when the agent tried to ssh into it. And deleting > it > > > and > > > >>>> > making it respawn didn't help. Then I tried 4.11.3 to 4.14.0 and > > > >>>> *all* the > > > >>>> > VM's failed at that point (of course, that was with the new > > > template). > > > >>>> > > > > > >>>> > > Right now I'm back at 4.11.3 until this can be figured out. > > > >>>> > > > > > >>>> > >> On 8/11/2020 5:53 AM, Ammad Syed wrote: > > > >>>> > >> Hi, > > > >>>> > >> > > > >>>> > >> I think 4.12 and 4.13 uses same systemVM template i.e 4.11.3 > > > >>>> version, > > > >>>> > which > > > >>>> > >> I already have registered. Currently I am running 4.11.3 > > version > > > >>>> of ACS. > > > >>>> > >> > > > >>>> > >> MariaDB [cloud]> SELECT id,name,type,cross_zones,state FROM > > > >>>> > >> cloud.vm_template WHERE name like '%systemvm-xenserver%' AND > > > >>>> removed IS > > > >>>> > >> NULL; > > > >>>> > >> > > > >>>> > > > > +------+-----------------------------+--------+-------------+----------+ > > > >>>> > >> | id | name | type | cross_zones | > > > >>>> state | > > > >>>> > >> > > > >>>> > > > > +------+-----------------------------+--------+-------------+----------+ > > > >>>> > >> | 337 | systemvm-xenserver-3.0.0 | SYSTEM | 0 | > > > >>>> Inactive | > > > >>>> > >> | 418 | systemvm-xenserver-4.2 | SYSTEM | 0 | > > > >>>> Active | > > > >>>> > >> | 472 | systemvm-xenserver-4.3 | USER | 1 | > > > >>>> Inactive | > > > >>>> > >> | 473 | systemvm-xenserver-4.3 | USER | 1 | > > > >>>> Inactive | > > > >>>> > >> | 474 | systemvm-xenserver-4.3 | USER | 1 | > > > >>>> Inactive | > > > >>>> > >> | 475 | systemvm-xenserver-4.3 | USER | 1 | > > > >>>> Inactive | > > > >>>> > >> | 476 | systemvm-xenserver-4.3 | USER | 0 | > > > >>>> Inactive | > > > >>>> > >> | 479 | systemvm-xenserver-4.3-2 | USER | 1 | > > > >>>> Inactive | > > > >>>> > >> | 480 | systemvm-xenserver-4.3 | SYSTEM | 0 | > > > >>>> Active | > > > >>>> > >> | 549 | systemvm-xenserver-4.5.1 | USER | 0 | > > > >>>> Active | > > > >>>> > >> | 550 | systemvm-xenserver-4.5.1 | SYSTEM | 0 | > > > >>>> Active | > > > >>>> > >> | 651 | systemvm-xenserver-4.7.0 | USER | 0 | > > > >>>> Inactive | > > > >>>> > >> | 652 | systemvm-xenserver-4.7.0 | USER | 0 | > > > >>>> Inactive | > > > >>>> > >> | 653 | systemvm-xenserver-4.7.0 | SYSTEM | 0 | > > > >>>> Inactive | > > > >>>> > >> | 737 | systemvm-xenserver-4.9.2 | SYSTEM | 1 | > > > >>>> Inactive | > > > >>>> > >> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM | 1 | > > > >>>> Active | > > > >>>> > >> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM | 1 | > > > >>>> Active | > > > >>>> > >> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM | 1 | > > > >>>> Active | > > > >>>> > >> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM | 1 | > > > >>>> Active | > > > >>>> > >> > > > >>>> > > > > +------+-----------------------------+--------+-------------+----------+ > > > >>>> > >> > > > >>>> > >> - Ammad > > > >>>> > >> > > > >>>> > >> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion < > > > >>>> pdion...@apache.org> > > > >>>> > wrote: > > > >>>> > >> db. > > > >>>> > >>> Hi Syed, > > > >>>> > >>> From 4.12, the systemvm template had to be upgraded because > of > > > OS > > > >>>> > change in > > > >>>> > >>> the template, moved to a latest version of Debian. Because > of > > > >>>> that, > > > >>>> > some VR > > > >>>> > >>> scripts have changed and make obsolete older version of VRs, > > so > > > >>>> you > > > >>>> > will > > > >>>> > >>> most likely have to register an updated systemvm templates > and > > > >>>> upgrade > > > >>>> > your > > > >>>> > >>> system VMs and VRs. > > > >>>> > >>> > > > >>>> > >>> Regards, > > > >>>> > >>> > > > >>>> > >>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed < > > > >>>> syedamma...@gmail.com> > > > >>>> > wrote: > > > >>>> > >>> > > > >>>> > >>>> Hi Guys, > > > >>>> > >>>> > > > >>>> > >>>> I was previously on 4.9.3 cloudstack and upgraded to 4.11.1 > > > then > > > >>>> > 4.11.3. > > > >>>> > >>>> The version 4.11.3 is working fine since six months. > > > >>>> > >>>> > > > >>>> > >>>> Now I have tried to upgrade my system from 4.11.3 to > 4.13.1. > > > The > > > >>>> > upgrade > > > >>>> > >>>> goes successful. I didn't uploaded any system VM template. > > > >>>> However the > > > >>>> > >>>> problem occured when I recreated my systemVM of POD, the VM > > > >>>> recreated > > > >>>> > and > > > >>>> > >>>> its state was running but agent state was not getting up, > its > > > >>>> showing > > > >>>> > >>> blank > > > >>>> > >>>> in column. > > > >>>> > >>>> > > > >>>> > >>>> Digging further via job logs, the job is failed with error > > that > > > >>>> > unable to > > > >>>> > >>>> execute command via ssh. Below are the logs. > > > >>>> > >>>> > > > >>>> > >>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper] > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH > execution > > > of > > > >>>> > command > > > >>>> > >>>> /opt/cloud/bin/router_proxy.sh keystore-s > > > >>>> > >>>> etup 169.254.2.199 > > > >>>> /usr/local/cloud/systemvm/conf/agent.properties > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/cloud.jks TJaQYChYBwKh7Cx9 > 365 > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/clou > > > >>>> > >>>> d.csr has an error status code in return. Result output: > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.m.DirectAgentAttache] > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq > > > >>>> > >>>> 906-3195585410596077730: Response Received: > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq > > > >>>> > >>>> 906-3195585410596077730: Processing: { Ans: , MgmtId: > > > 779271079 > > > >>>> > >>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, > > > >>>> > >>>> [{"org.apache.cloudstack.ca > > > >>>> > >>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}] > > > >>>> > >>>> } > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 > > > >>>> > ctx-df740f75) > > > >>>> > >>>> (logid:9fa7dece) Seq 906-319558541059607773 > > > >>>> > >>>> 0: Received: { Ans: , MgmtId: 77927107943497, via: > > > >>>> > >>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, { > > > >>>> SetupKeystoreAnswer } } > > > >>>> > >>>> 2020-07-25 02:30:48,127 ERROR > > [c.c.v.VirtualMachineManagerImpl] > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 > > > >>>> > ctx-df740f75) > > > >>>> > >>>> (logid:9fa7dece) Failed to setup keystore and generate CSR > > for > > > >>>> system > > > >>>> > vm: > > > >>>> > >>>> s-24142-VM > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.v.VmWorkJobHandlerProxy] > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 > > > >>>> > ctx-df740f75) > > > >>>> > >>>> (logid:9fa7dece) Done executing VM work job: > > > >>>> > >>>> > > > >>>> > >>>> > > > >>>> > >>> > > > >>>> > > > > >>>> > > > > > > com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"} > > > >>>> > >>>> 2020-07-25 02:30:48,128 DEBUG > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d job-1208155/job-1208258 > > > >>>> > ctx-df740f75) > > > >>>> > >>>> (logid:9fa7dece) Complete async job-1208258, jobStatus: > > > >>>> SUCCEEDED, > > > >>>> > >>>> resultCode: 0, result: null > > > >>>> > >>>> > > > >>>> > >>>> I tried to dig it further, I was unable to login systemVM > via > > > >>>> ssh from > > > >>>> > >>>> xenserver host with key /root/.ssh/id_rsa.cloud placed. > Look > > > like > > > >>>> > private > > > >>>> > >>>> key issue. However I am able to login on my old systemVMs ( > > i.e > > > >>>> > created > > > >>>> > >>> on > > > >>>> > >>>> ACS 4.11.3) > > > >>>> > >>>> > > > >>>> > >>>> Also I have SSL certificate enabled for console proxy on my > > ACS > > > >>>> 4.11.3 > > > >>>> > >>> and > > > >>>> > >>>> I am using only xenserver 7.0 hosts. > > > >>>> > >>>> > > > >>>> > >>>> I tried to disable SSL on secstorage and console proxy from > > > >>>> global > > > >>>> > >>>> settings, but still didn't worked. > > > >>>> > >>>> > > > >>>> > >>>> I had a fresh installation of ACS 4.13.1 with xenserver > 7.0, > > > >>>> systemVMs > > > >>>> > >>> are > > > >>>> > >>>> working fine in it. > > > >>>> > >>>> > > > >>>> > >>>> Please advise. > > > >>>> > >>>> -- > > > >>>> > >>>> Regards, > > > >>>> > >>>> > > > >>>> > >>>> > > > >>>> > >>>> Syed Ammad Ali > > > >>>> > >>>> > > > >>>> > >> > > > >>>> > > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> Regards, > > > >>> > > > >>> > > > >>> Syed Ammad Ali > > > >>> > > > >> > > > >> > > > >> -- > > > >> Regards, > > > >> > > > >> > > > >> Syed Ammad Ali > > > >> > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > > > > > Syed Ammad Ali > > > > > > > > > > > > > -- > > > Regards, > > > > > > > > > Syed Ammad Ali > > > > > > > > > -- > > > > Andrija Panić > > > > > -- > Regards, > > > Syed Ammad Ali > -- Andrija Panić