Hello,
I'm not sure why I get this error (added new vmware zone), there are not permission issues for systemvms folder, and I do not see any error above this. If I try to login to login into SSVM/Proxy console via cloudstack management, I get this access denied. Any suggestion? 021-09-23 05:22:15,012 ERROR [c.c.u.s.SshHelper] (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, job-34950/job-34972, cmd: SetupKeyStoreCommand) (logid:f3cd2252) Failed to authentication SSH user root on host 10.15.0.160 2021-09-23 05:22:15,012 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, job-34950/job-34972, cmd: SetupKeyStoreCommand) (logid:f3cd2252) [ignored]failed to get message for exception: Failed to authentication SSH user root on host 10.15.0.160 2021-09-23 05:22:15,012 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, job-34950/job-34972, cmd: SetupKeyStoreCommand) (logid:f3cd2252) Command failed due to Exception: java.lang.Exception Message: Failed to authentication SSH user root on host 10.15.0.160 2021-09-23 05:22:15,012 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-283:ctx-b1e493bd lnd-uk-001.shape.host, job-34950/job-34972, cmd: SetupKeyStoreCommand) (logid:f3cd2252) keystore-setup execution result: false 2021-09-23 05:22:15,014 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq 40-1394708509601300577: Response Received: 2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request] (DirectAgent-283:ctx-b1e493bd) (logid:f3cd2252) Seq 40-1394708509601300577: Processing: { Ans: , MgmtId: 345049356158, via: 40(lnd-uk-001.shape.host), Ver: v1, Flags: 10, [{"org.apache.cloudstack.ca.SetupKeystoreAnswer":{"result":"true","wait":"0" ,"bypassHostMaintenance":"false"}}] } 2021-09-23 05:22:15,016 DEBUG [c.c.a.t.Request] (Work-Job-Executor-14:ctx-10c822ed job-34950/job-34972 ctx-67aa0df0) (logid:f3cd2252) Seq 40-1394708509601300577: Received: { Ans: , MgmtId: 345049356158, via: 40(lnd-uk-001.test.host), Ver: v1, Flags: 10, { SetupKeystoreAnswer } } 2021-09-23 05:22:15,022 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-14:ctx-10c822ed job-34950/job-34972 ctx-67aa0df0) (logid:f3cd2252) Retrying after catching exception while trying to secure agent for systemvm id=1955 com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR: Command failed due to Exception: java.lang.Exception Message: Failed to authentication SSH user root on host 10.15.0.160 at org.apache.cloudstack.ca.provider.RootCAProvider.generateCertificateUsingCsr (RootCAProvider.java:170) at org.apache.cloudstack.ca.provider.RootCAProvider.issueCertificate(RootCAProv ider.java:230) at org.apache.cloudstack.ca.CAManagerImpl.issueCertificate(CAManagerImpl.java:1 64) at jdk.internal.reflect.GeneratedMethodAccessor318.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin gMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU tils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint (ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect iveMethodInvocation.java:163) at org.apache.cloudstack.network.contrail.management.EventUtils$EventIntercepto r.invoke(EventUtils.java:107) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect iveMethodInvocation.java:175) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51 ) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect iveMethodInvocation.java:175) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos eInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect iveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro xy.java:215) at com.sun.proxy.$Proxy292.issueCertificate(Unknown Source) at com.cloud.vm.VirtualMachineManagerImpl.setupAgentSecurity(VirtualMachineMana gerImpl.java:970) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage rImpl.java:1241) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage rImpl.java:5502) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodA ccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin gMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.jav a:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManager Impl.java:5669) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext (AsyncJobManagerImpl.java:620) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo ntextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa ultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon text(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont ext(DefaultManagedContext.java:52) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont extRunnable.java:45) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJob ManagerImpl.java:568) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java :515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut or.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu tor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Regards, Cristian