Guys, any help to troubleshoot this further would be highly appreciated.
Ammad Ali > On 18-Aug-2020, at 1:05 PM, Ammad Syed <syedamma...@gmail.com> wrote: > > > Hi, > > The systemVM is accessible with 169.x.x.x IP from xenserver host. But login > with private key is denied. I am testing this on my test environment. > > [root@xenserver ~]# ping 169.254.0.121 > PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data. > 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time=0.328 ms > 64 bytes from 169.254.0.121: icmp_seq=2 ttl=64 time=0.124 ms > 64 bytes from 169.254.0.121: icmp_seq=3 ttl=64 time=0.111 ms > 64 bytes from 169.254.0.121: icmp_seq=4 ttl=64 time=0.126 ms > ^C > --- 169.254.0.121 ping statistics --- > 4 packets transmitted, 4 received, 0% packet loss, time 2997ms > rtt min/avg/max/mdev = 0.111/0.172/0.328/0.090 ms > [root@xenserver ~]# > [root@xenserver ~]# > [root@xenserver ~]# ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922 > Permission denied (publickey). > > I have checked by logging into the systemVM, the public key is not found in > /root/.ssh/authorized_keys. > > I have digged further, the systemvm.iso is same on the management node and > xenserver host. I have checked by mounting the ISO as well, the public key is > there in authorized_keys in ISO. > > Restarting / recreating systemVM isn't fixed the issue. > > The systemVM is already running on HVM mode on xenserver host as the template > OS type is Debian 8 64bit is selected. > > On my PRD upgrade, I have tested on one zone only. Currently I am testing on > my test ACS environment which have only one zone. > > Ammad Ali > >> On Mon, Aug 17, 2020 at 11:10 PM Andrija Panic <andrija.pa...@gmail.com> >> wrote: >> It was my tool (GUI based tool, that still doesn't return any line (???) >> but I see it. >> >> Problem is with 169.254.1.178 not being reachable over SSH (2020-07-25 >> 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase] >> (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command in VR: >> /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178 >> /usr/local/cloud/systemvm/conf/agent.properties >> /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365 >> /usr/local/cloud/systemvm/conf/cloud.csr) >> >> Can you try to ssh from that particular XenServer (where the SSVM is >> running) to the IP 169.xxxxx on port 3922 (ssh port) using the private key >> - if you log in to the VM via it's console (root/password) - do you see >> with ifconfig the interface being started with 169..xxxx IP address? I >> assume that destroying the SSVM and starting a new one doesn't fix the >> issue? >> >> You can also try changing the OS type of the systemVM template to Other >> Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need to see ID >> for that OS type from the guest_os table, and set the guest_os_id field in >> the vm_template table for your systemVM template. >> >> Does the issue occur in other zones as well? (assuming same hypervisor type) >> >> >> On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedamma...@gmail.com> wrote: >> >> > I have rechecked the logs are there. Please review below lines. >> > >> > >> > >> > C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors >> > (27 hits) >> > Line 409969: 2020-07-25 01:45:52,320 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-58:ctx-9dc75334 >> > job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to setup >> > keystore and generate CSR for system vm: s-25023-VM >> > Line 437413: 2020-07-25 01:51:33,845 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-70:ctx-457ce4ee >> > job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to setup >> > keystore and generate CSR for system vm: s-25024-VM >> > Line 549538: 2020-07-25 02:06:44,833 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-4634f7e0 >> > job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to setup >> > keystore and generate CSR for system vm: v-25027-VM >> > Line 663111: 2020-07-25 02:23:43,783 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-22:ctx-fe0b857b >> > job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to setup >> > keystore and generate CSR for system vm: v-20771-VM >> > Line 667060: 2020-07-25 02:24:30,221 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-662cfe5d >> > job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to setup >> > keystore and generate CSR for system vm: v-24763-VM >> > Line 674269: 2020-07-25 02:25:47,345 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-b6529613 >> > job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to setup >> > keystore and generate CSR for system vm: v-20773-VM >> > Line 678169: 2020-07-25 02:26:20,355 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-7a062a66 >> > job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to setup >> > keystore and generate CSR for system vm: v-25027-VM >> > Line 697113: 2020-07-25 02:30:09,669 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-40:ctx-3be9fddc >> > job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to setup >> > keystore and generate CSR for system vm: v-20771-VM >> > Line 701161: 2020-07-25 02:30:48,127 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d >> > job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to setup >> > keystore and generate CSR for system vm: s-24142-VM >> > Line 701838: 2020-07-25 02:30:56,759 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-42:ctx-04ac8e71 >> > job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to setup >> > keystore and generate CSR for system vm: v-24763-VM >> > Line 703865: 2020-07-25 02:31:25,123 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-24bdbe8f >> > job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to setup >> > keystore and generate CSR for system vm: v-20773-VM >> > Line 712078: 2020-07-25 02:32:28,666 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d5db5c2 >> > job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to setup >> > keystore and generate CSR for system vm: s-25033-VM >> > Line 715204: 2020-07-25 02:33:02,875 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-51:ctx-594da6d2 >> > job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to setup >> > keystore and generate CSR for system vm: s-25035-VM >> > Line 743140: 2020-07-25 02:38:52,257 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-a3f032da >> > job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to setup >> > keystore and generate CSR for system vm: s-25037-VM >> > Line 817646: 2020-07-25 02:47:07,127 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-ca073bd1 >> > job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to setup >> > keystore and generate CSR for system vm: s-25038-VM >> > Line 847150: 2020-07-25 02:51:46,293 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-9db8c548 >> > job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to setup >> > keystore and generate CSR for system vm: v-25040-VM >> > Line 903949: 2020-07-25 02:57:48,515 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-f35fd59a >> > job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-25041-VM >> > Line 957628: 2020-07-25 03:04:13,286 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-dc18956a >> > job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-20771-VM >> > Line 961606: 2020-07-25 03:05:00,522 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-28:ctx-83e48421 >> > job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-24763-VM >> > Line 966633: 2020-07-25 03:05:29,767 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-30:ctx-897e2a66 >> > job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-20773-VM >> > Line 968892: 2020-07-25 03:05:53,648 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-36:ctx-68e45b80 >> > job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-25041-VM >> > Line 1003036: 2020-07-25 03:13:33,582 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-45:ctx-0f552e0e >> > job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-20771-VM >> > Line 1006496: 2020-07-25 03:14:09,787 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-08e5db24 >> > job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-24763-VM >> > Line 1008408: 2020-07-25 03:14:39,584 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-8b9288a9 >> > job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-20773-VM >> > Line 1023914: 2020-07-25 03:15:58,139 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d54d717 >> > job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-25046-VM >> > Line 1089568: 2020-07-25 03:24:24,534 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-62ff8844 >> > job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to setup >> > keystore and generate CSR for system vm: v-25048-VM >> > Line 1097605: 2020-07-25 03:25:23,179 ERROR >> > [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-62:ctx-24407a9e >> > job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to setup >> > keystore and generate CSR for system vm: s-25050-VM >> > >> > -Ammad Ali >> > >> > >> > >> > On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <andrija.pa...@gmail.com> >> > wrote: >> > >> > > Ammad, no such log lines in the file you uploaded....? >> > > >> > > On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedamma...@gmail.com> wrote: >> > > >> > > > Hi Andrija, >> > > > >> > > > The error occurred only one time. Can you please try to grep "Failed to >> > > > setup keystore and generate CSR" you will see all the systemVMs that I >> > > > tried to create failed because of certificate generation failed. OR you >> > > can >> > > > grep job-1208321/job-1208452 this job as well. >> > > > >> > > > Also I mentioned above that same thing is happening in my test >> > > environment >> > > > that I created 4.11.1 then upgraded 4.11.3 > 4.13.1. >> > > > >> > > > Ammad Ali >> > > > >> > > > On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic < >> > andrija.pa...@gmail.com> >> > > > wrote: >> > > > >> > > > > 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner] >> > > > > (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception >> > > Failed >> > > > to >> > > > > fetch any free public IP address >> > > > > com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any >> > > free >> > > > > public IP address >> > > > > >> > > > > ? >> > > > > >> > > > > On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedamma...@gmail.com> >> > > wrote: >> > > > > >> > > > > > I have checked further, the key is successfully injected in >> > > > systemvm.iso >> > > > > on >> > > > > > the management node and copied successfully to xenserver. I have >> > > > checked >> > > > > on >> > > > > > systemVM there is no public key found in authorized_keys of root. >> > > > > > >> > > > > > How can I troubleshoot this further ? how can I enable trace >> > logging >> > > > for >> > > > > > management server to see if there is something problematic >> > happening. >> > > > > > >> > > > > > Ammad Ali >> > > > > > >> > > > > > On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <syedamma...@gmail.com> >> > > > > wrote: >> > > > > > >> > > > > > > Hi, >> > > > > > > >> > > > > > > I have setup my test environment, here is what I did: >> > > > > > > >> > > > > > > - Installed ACS 4.11.1 and added xenserver 7.0 host in it. >> > > SystemVMs >> > > > > are >> > > > > > > up in the zone and agents are up. >> > > > > > > - Then upgraded the system to 4.11.3. Recreated systemVMs, agent >> > is >> > > > up >> > > > > > and >> > > > > > > systemVM are up with 4.11.3 systemVM version. >> > > > > > > - Then upgraded the system to 4.13.1, recreated systemVMs are >> > > running >> > > > > but >> > > > > > > agents are not up. >> > > > > > > >> > > > > > > I have checked md5sum of systemvm.iso on xenserver and management >> > > > > server, >> > > > > > > both are same. >> > > > > > > >> > > > > > > [root@xenserver iso]# md5sum >> > > > /opt/xensource/packages/iso/systemvm.iso >> > > > > > > baba18f156395da3a5d8208727d8f421 >> > > > > > /opt/xensource/packages/iso/systemvm.iso >> > > > > > > >> > > > > > > [root@cloudstack-upgrade vms]# md5sum >> > > > > > > /usr/share/cloudstack-common/vms/systemvm.iso >> > > > > > > baba18f156395da3a5d8208727d8f421 >> > > > > > > /usr/share/cloudstack-common/vms/systemvm.iso >> > > > > > > >> > > > > > > Also the private key on xenserver root and management server are >> > > > same. >> > > > > > > >> > > > > > > management server >> > > > > > > /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud >> > > > > > > >> > > > > > > xenserver path >> > > > > > > /root/.ssh/id_rsa.cloud >> > > > > > > >> > > > > > > - Ammad Ali >> > > > > > > >> > > > > > > On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed < >> > syedamma...@gmail.com >> > > > >> > > > > > wrote: >> > > > > > > >> > > > > > >> Here is the link for download management logs. >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing >> > > > > > >> >> > > > > > >> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed < >> > > syedamma...@gmail.com> >> > > > > > >> wrote: >> > > > > > >> >> > > > > > >>> I have reverted the version back to 4.11.3. But I have saved >> > logs >> > > > > > >>> starting from upgrade. >> > > > > > >>> >> > > > > > >>> I think the key has been copied successfully in system vm iso. >> > > > > > >>> >> > > > > > >>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl] >> > > > > > >>> (main:null) (logid:) Going to update systemvm iso with >> > generated >> > > > > > keypairs >> > > > > > >>> if needed >> > > > > > >>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl] >> > > > > > >>> (main:null) (logid:) Trying to inject public and private keys >> > > into >> > > > > > systemvm >> > > > > > >>> iso >> > > > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) >> > > (logid:) >> > > > > > >>> Looking for vms/systemvm.iso in the classpath >> > > > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) >> > > (logid:) >> > > > > > >>> System resource: >> > > file:/usr/share/cloudstack-common/vms/systemvm.iso >> > > > > > >>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null) >> > > (logid:) >> > > > > > >>> Absolute path = /usr/share/cloudstack-common/vms/systemvm.iso >> > > > > > >>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl] >> > > > > > >>> (main:null) (logid:) Executing: /bin/bash >> > > > > > >>> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh >> > > > > > >>> /var/cloudstack/management/.ssh/id_rsa.pub >> > > > > > >>> /var/cloudstack/management/.ssh/id_rsa >> > > > > > >>> /usr/share/cloudstack-common/vms/systemvm.iso >> > > > > > >>> 2020-07-25 00:34:17,636 INFO [c.c.s.ConfigurationServerImpl] >> > > > > > >>> (main:null) (logid:) Injected public and private keys into >> > > systemvm >> > > > > iso >> > > > > > >>> with result : null >> > > > > > >>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase] >> > > > > > >>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying >> > > > > > >>> >> > > > > > >> > > > > >> > > > >> > > >> > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso >> > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.22 with permission >> > > 0644 >> > > > > > >>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase] >> > > > > > >>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying >> > > > > > >>> >> > > > > > >> > > > > >> > > > >> > > >> > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso >> > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission >> > 0644 >> > > > > > >>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase] >> > > > > > >>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying >> > > > > > >>> >> > > > > > >> > > > > >> > > > >> > > >> > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso >> > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.9 with permission >> > 0644 >> > > > > > >>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase] >> > > > > > >>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying >> > > > > > >>> >> > > > > > >> > > > > >> > > > >> > > >> > /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso >> > > > > > >>> to /opt/xensource/packages/iso on 172.16.2.5 with permission >> > 0644 >> > > > > > >>> >> > > > > > >>> I have attached complete management logs. The start of logs is >> > > the >> > > > > > start >> > > > > > >>> of management server after upgrade of management server from >> > > 4.11.3 >> > > > > to >> > > > > > >>> 4.13.1. >> > > > > > >>> >> > > > > > >>> Ammad Ali >> > > > > > >>> >> > > > > > >>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic < >> > > > > andrija.pa...@gmail.com >> > > > > > > >> > > > > > >>> wrote: >> > > > > > >>> >> > > > > > >>>> Do you get an error while trying to inject ssh key into the >> > > > > > systemvm.iso >> > > > > > >>>> (mgmt logs) , or can you confirm that the systemvm.iso on XS >> > > host >> > > > > and >> > > > > > >>>> the >> > > > > > >>>> one on the mgmt server are identical, md5sum them (i.e. has >> > the >> > > > iso >> > > > > > been >> > > > > > >>>> copied over to the XS successfully) - this might explain not >> > > being >> > > > > > able >> > > > > > >>>> to >> > > > > > >>>> login via ssh with your private key. >> > > > > > >>>> >> > > > > > >>>> Best, >> > > > > > >>>> >> > > > > > >>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <syedamma...@gmail.com >> > > >> > > > > wrote: >> > > > > > >>>> >> > > > > > >>>> > Yes this is exactly the same issue that i faced. >> > > > > > >>>> > >> > > > > > >>>> > Sent from my iPhone >> > > > > > >>>> > >> > > > > > >>>> > > On 12-Aug-2020, at 8:35 AM, Eric Lee Green < >> > > > > > >>>> eric.lee.gr...@gmail.com> >> > > > > > >>>> > wrote: >> > > > > > >>>> > > >> > > > > > >>>> > > Correct, 4.11.3 template is used for 4.11.3, 4.12, and >> > > 4.13. >> > > > > 4.14 >> > > > > > >>>> moves >> > > > > > >>>> > to the 4.14.0 template. >> > > > > > >>>> > > >> > > > > > >>>> > > There seems to be something odd happening key-wise >> > sometimes >> > > > > with >> > > > > > >>>> > upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed an >> > > upgrade >> > > > > > from >> > > > > > >>>> > 4.11.3 to 4.13.1 that *almost* worked, but the secondary >> > > storage >> > > > > VM >> > > > > > >>>> > wouldn't work and thus I couldn't spawn new virtual >> > machines. >> > > > Same >> > > > > > >>>> symptom >> > > > > > >>>> > -- key error when the agent tried to ssh into it. And >> > deleting >> > > > it >> > > > > > and >> > > > > > >>>> > making it respawn didn't help. Then I tried 4.11.3 to 4.14.0 >> > > and >> > > > > > >>>> *all* the >> > > > > > >>>> > VM's failed at that point (of course, that was with the new >> > > > > > template). >> > > > > > >>>> > > >> > > > > > >>>> > > Right now I'm back at 4.11.3 until this can be figured >> > out. >> > > > > > >>>> > > >> > > > > > >>>> > >> On 8/11/2020 5:53 AM, Ammad Syed wrote: >> > > > > > >>>> > >> Hi, >> > > > > > >>>> > >> >> > > > > > >>>> > >> I think 4.12 and 4.13 uses same systemVM template i.e >> > > 4.11.3 >> > > > > > >>>> version, >> > > > > > >>>> > which >> > > > > > >>>> > >> I already have registered. Currently I am running 4.11.3 >> > > > > version >> > > > > > >>>> of ACS. >> > > > > > >>>> > >> >> > > > > > >>>> > >> MariaDB [cloud]> SELECT id,name,type,cross_zones,state >> > FROM >> > > > > > >>>> > >> cloud.vm_template WHERE name like '%systemvm-xenserver%' >> > > AND >> > > > > > >>>> removed IS >> > > > > > >>>> > >> NULL; >> > > > > > >>>> > >> >> > > > > > >>>> >> > > > > > >> > > > >> > +------+-----------------------------+--------+-------------+----------+ >> > > > > > >>>> > >> | id | name | type | >> > > cross_zones | >> > > > > > >>>> state | >> > > > > > >>>> > >> >> > > > > > >>>> >> > > > > > >> > > > >> > +------+-----------------------------+--------+-------------+----------+ >> > > > > > >>>> > >> | 337 | systemvm-xenserver-3.0.0 | SYSTEM | >> > > 0 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 418 | systemvm-xenserver-4.2 | SYSTEM | >> > > 0 | >> > > > > > >>>> Active | >> > > > > > >>>> > >> | 472 | systemvm-xenserver-4.3 | USER | >> > > 1 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 473 | systemvm-xenserver-4.3 | USER | >> > > 1 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 474 | systemvm-xenserver-4.3 | USER | >> > > 1 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 475 | systemvm-xenserver-4.3 | USER | >> > > 1 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 476 | systemvm-xenserver-4.3 | USER | >> > > 0 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 479 | systemvm-xenserver-4.3-2 | USER | >> > > 1 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 480 | systemvm-xenserver-4.3 | SYSTEM | >> > > 0 | >> > > > > > >>>> Active | >> > > > > > >>>> > >> | 549 | systemvm-xenserver-4.5.1 | USER | >> > > 0 | >> > > > > > >>>> Active | >> > > > > > >>>> > >> | 550 | systemvm-xenserver-4.5.1 | SYSTEM | >> > > 0 | >> > > > > > >>>> Active | >> > > > > > >>>> > >> | 651 | systemvm-xenserver-4.7.0 | USER | >> > > 0 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 652 | systemvm-xenserver-4.7.0 | USER | >> > > 0 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 653 | systemvm-xenserver-4.7.0 | SYSTEM | >> > > 0 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 737 | systemvm-xenserver-4.9.2 | SYSTEM | >> > > 1 | >> > > > > > >>>> Inactive | >> > > > > > >>>> > >> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM | >> > > 1 | >> > > > > > >>>> Active | >> > > > > > >>>> > >> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM | >> > > 1 | >> > > > > > >>>> Active | >> > > > > > >>>> > >> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM | >> > > 1 | >> > > > > > >>>> Active | >> > > > > > >>>> > >> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM | >> > > 1 | >> > > > > > >>>> Active | >> > > > > > >>>> > >> >> > > > > > >>>> >> > > > > > >> > > > >> > +------+-----------------------------+--------+-------------+----------+ >> > > > > > >>>> > >> >> > > > > > >>>> > >> - Ammad >> > > > > > >>>> > >> >> > > > > > >>>> > >> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion < >> > > > > > >>>> pdion...@apache.org> >> > > > > > >>>> > wrote: >> > > > > > >>>> > >> db. >> > > > > > >>>> > >>> Hi Syed, >> > > > > > >>>> > >>> From 4.12, the systemvm template had to be upgraded >> > > because >> > > > of >> > > > > > OS >> > > > > > >>>> > change in >> > > > > > >>>> > >>> the template, moved to a latest version of Debian. >> > Because >> > > > of >> > > > > > >>>> that, >> > > > > > >>>> > some VR >> > > > > > >>>> > >>> scripts have changed and make obsolete older version of >> > > VRs, >> > > > > so >> > > > > > >>>> you >> > > > > > >>>> > will >> > > > > > >>>> > >>> most likely have to register an updated systemvm >> > templates >> > > > and >> > > > > > >>>> upgrade >> > > > > > >>>> > your >> > > > > > >>>> > >>> system VMs and VRs. >> > > > > > >>>> > >>> >> > > > > > >>>> > >>> Regards, >> > > > > > >>>> > >>> >> > > > > > >>>> > >>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed < >> > > > > > >>>> syedamma...@gmail.com> >> > > > > > >>>> > wrote: >> > > > > > >>>> > >>> >> > > > > > >>>> > >>>> Hi Guys, >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> I was previously on 4.9.3 cloudstack and upgraded to >> > > 4.11.1 >> > > > > > then >> > > > > > >>>> > 4.11.3. >> > > > > > >>>> > >>>> The version 4.11.3 is working fine since six months. >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> Now I have tried to upgrade my system from 4.11.3 to >> > > > 4.13.1. >> > > > > > The >> > > > > > >>>> > upgrade >> > > > > > >>>> > >>>> goes successful. I didn't uploaded any system VM >> > > template. >> > > > > > >>>> However the >> > > > > > >>>> > >>>> problem occured when I recreated my systemVM of POD, >> > the >> > > VM >> > > > > > >>>> recreated >> > > > > > >>>> > and >> > > > > > >>>> > >>>> its state was running but agent state was not getting >> > up, >> > > > its >> > > > > > >>>> showing >> > > > > > >>>> > >>> blank >> > > > > > >>>> > >>>> in column. >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> Digging further via job logs, the job is failed with >> > > error >> > > > > that >> > > > > > >>>> > unable to >> > > > > > >>>> > >>>> execute command via ssh. Below are the logs. >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper] >> > > > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH >> > > > execution >> > > > > > of >> > > > > > >>>> > command >> > > > > > >>>> > >>>> /opt/cloud/bin/router_proxy.sh keystore-s >> > > > > > >>>> > >>>> etup 169.254.2.199 >> > > > > > >>>> /usr/local/cloud/systemvm/conf/agent.properties >> > > > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/cloud.jks >> > TJaQYChYBwKh7Cx9 >> > > > 365 >> > > > > > >>>> > >>>> /usr/local/cloud/systemvm/conf/clou >> > > > > > >>>> > >>>> d.csr has an error status code in return. Result >> > output: >> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG >> > > [c.c.a.m.DirectAgentAttache] >> > > > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq >> > > > > > >>>> > >>>> 906-3195585410596077730: Response Received: >> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] >> > > > > > >>>> > >>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq >> > > > > > >>>> > >>>> 906-3195585410596077730: Processing: { Ans: , MgmtId: >> > > > > > 779271079 >> > > > > > >>>> > >>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags: >> > 10, >> > > > > > >>>> > >>>> [{"org.apache.cloudstack.ca >> > > > > > >>>> > >>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}] >> > > > > > >>>> > >>>> } >> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request] >> > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d >> > > job-1208155/job-1208258 >> > > > > > >>>> > ctx-df740f75) >> > > > > > >>>> > >>>> (logid:9fa7dece) Seq 906-319558541059607773 >> > > > > > >>>> > >>>> 0: Received: { Ans: , MgmtId: 77927107943497, via: >> > > > > > >>>> > >>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, { >> > > > > > >>>> SetupKeystoreAnswer } } >> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 ERROR >> > > > > [c.c.v.VirtualMachineManagerImpl] >> > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d >> > > job-1208155/job-1208258 >> > > > > > >>>> > ctx-df740f75) >> > > > > > >>>> > >>>> (logid:9fa7dece) Failed to setup keystore and generate >> > > CSR >> > > > > for >> > > > > > >>>> system >> > > > > > >>>> > vm: >> > > > > > >>>> > >>>> s-24142-VM >> > > > > > >>>> > >>>> 2020-07-25 02:30:48,127 DEBUG >> > > [c.c.v.VmWorkJobHandlerProxy] >> > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d >> > > job-1208155/job-1208258 >> > > > > > >>>> > ctx-df740f75) >> > > > > > >>>> > >>>> (logid:9fa7dece) Done executing VM work job: >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>> >> > > > > > >>>> > >> > > > > > >>>> >> > > > > > >> > > > > >> > > > >> > > >> > com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"} >> > > > > > >>>> > >>>> 2020-07-25 02:30:48,128 DEBUG >> > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] >> > > > > > >>>> > >>>> (Work-Job-Executor-41:ctx-4e3c666d >> > > job-1208155/job-1208258 >> > > > > > >>>> > ctx-df740f75) >> > > > > > >>>> > >>>> (logid:9fa7dece) Complete async job-1208258, jobStatus: >> > > > > > >>>> SUCCEEDED, >> > > > > > >>>> > >>>> resultCode: 0, result: null >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> I tried to dig it further, I was unable to login >> > systemVM >> > > > via >> > > > > > >>>> ssh from >> > > > > > >>>> > >>>> xenserver host with key /root/.ssh/id_rsa.cloud placed. >> > > > Look >> > > > > > like >> > > > > > >>>> > private >> > > > > > >>>> > >>>> key issue. However I am able to login on my old >> > > systemVMs ( >> > > > > i.e >> > > > > > >>>> > created >> > > > > > >>>> > >>> on >> > > > > > >>>> > >>>> ACS 4.11.3) >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> Also I have SSL certificate enabled for console proxy >> > on >> > > my >> > > > > ACS >> > > > > > >>>> 4.11.3 >> > > > > > >>>> > >>> and >> > > > > > >>>> > >>>> I am using only xenserver 7.0 hosts. >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> I tried to disable SSL on secstorage and console proxy >> > > from >> > > > > > >>>> global >> > > > > > >>>> > >>>> settings, but still didn't worked. >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> I had a fresh installation of ACS 4.13.1 with xenserver >> > > > 7.0, >> > > > > > >>>> systemVMs >> > > > > > >>>> > >>> are >> > > > > > >>>> > >>>> working fine in it. >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> Please advise. >> > > > > > >>>> > >>>> -- >> > > > > > >>>> > >>>> Regards, >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> >> > > > > > >>>> > >>>> Syed Ammad Ali >> > > > > > >>>> > >>>> >> > > > > > >>>> > >> >> > > > > > >>>> > >> > > > > > >>>> >> > > > > > >>> >> > > > > > >>> >> > > > > > >>> -- >> > > > > > >>> Regards, >> > > > > > >>> >> > > > > > >>> >> > > > > > >>> Syed Ammad Ali >> > > > > > >>> >> > > > > > >> >> > > > > > >> >> > > > > > >> -- >> > > > > > >> Regards, >> > > > > > >> >> > > > > > >> >> > > > > > >> Syed Ammad Ali >> > > > > > >> >> > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > Regards, >> > > > > > > >> > > > > > > >> > > > > > > Syed Ammad Ali >> > > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > Regards, >> > > > > > >> > > > > > >> > > > > > Syed Ammad Ali >> > > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > Andrija Panić >> > > > > >> > > > >> > > > >> > > > -- >> > > > Regards, >> > > > >> > > > >> > > > Syed Ammad Ali >> > > > >> > > >> > > >> > > -- >> > > >> > > Andrija Panić >> > > >> > >> > >> > -- >> > Regards, >> > >> > >> > Syed Ammad Ali >> > >> >> >> -- >> >> Andrija Panić > > > -- > Regards, > > > Syed Ammad Ali
