Looks like some error occurred while generating the keystore. Can you check if you see any .jks and crt/key files at /etc/cloudstack/agent/ directory?
Also share output of: netstat -nl | grep 16509 # if you get any listening libvirtd, then your libvirtd is NOT secured netstat -nl | grep 16514 # if you get this, then libvirtd is secured ls -lahi /etc/cloudstack/agent Did you upgrade to the latest LTS minor 4.11.2.0 release? If not please do that, we've some bugs around CA certificates fixed. - Rohit <https://cloudstack.apache.org> ________________________________ From: Ugo Vasi <ugo.v...@procne.it> Sent: Thursday, January 31, 2019 4:56:28 PM To: users@cloudstack.apache.org; Rohit Yadav Subject: Re: secure hosts communications Update: by rebooting the host system, the libvirt is restarted and the ACS-agent has been reconnected to management. The host remains in "unsecure" mode.... If I set to false "ca.plugin.root.auth.strictness" can I migrate the VM? Il 31/01/19 11:50, Ugo Vasi ha scritto: > Hi Rohit, > I tryed renew certificate but it failed! > Now libvirt does not restart and agent is disconnected: > > agent log: > 2019-01-31 11:17:07,530 INFO > [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] > (Certificate Renewal Timer:null) (logid:fe1554cc) Restarting libvirt > after certificate provisioning/renewal > 2019-01-31 11:17:07,567 INFO [cloud.agent.Agent] > (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill > 2019-01-31 11:17:07,568 WARN [cloud.agent.Agent] (Certificate Renewal > Timer:null) (logid:fe1554cc) Failed to execute post certificate > renewal command: > java.lang.IllegalStateException: Shutdown in progress > at > java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) > at java.lang.Runtime.removeShutdownHook(Runtime.java:239) > at > com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157) > at > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > 2019-01-31 11:17:09,797 INFO [cloud.agent.AgentShell] (main:null) > (logid:) Agent started > 2019-01-31 11:17:09,800 INFO [cloud.agent.AgentShell] (main:null) > (logid:) Implementation Version is 4.11.2.0 > 2019-01-31 11:17:09,802 INFO [cloud.agent.AgentShell] (main:null) > (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties > 2019-01-31 11:17:09,815 INFO [cloud.agent.AgentShell] (main:null) > (logid:) Defaulting to using properties file for storage > 2019-01-31 11:17:09,816 INFO [cloud.agent.AgentShell] (main:null) > (logid:) Defaulting to the constant time backoff algorithm > 2019-01-31 11:17:09,828 INFO [cloud.utils.LogUtils] (main:null) > (logid:) log4j configuration found at > /etc/cloudstack/agent/log4j-cloud.xml > 2019-01-31 11:17:09,850 INFO [cloud.agent.AgentShell] (main:null) > (logid:) Using default Java settings for IPv6 preference for agent > connection > 2019-01-31 11:17:09,998 INFO [cloud.agent.Agent] (main:null) (logid:) > id is 5 > 2019-01-31 11:17:10,030 INFO [kvm.resource.LibvirtConnection] > (main:null) (logid:) No existing libvirtd connection found. Opening a > new one > 2019-01-31 11:17:10,175 ERROR [cloud.agent.AgentShell] (main:null) > (logid:) Unable to start agent: > com.cloud.utils.exception.CloudRuntimeException: Failed to connect > socket to '/var/run/libvirt/libvirt-sock': No such file or directory > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914) > at com.cloud.agent.Agent.<init>(Agent.java:190) > at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > at com.cloud.agent.AgentShell.start(AgentShell.java:512) > at com.cloud.agent.AgentShell.main(AgentShell.java:547) > (logs repeat) > > syslog: > > > Jan 31 11:17:07 cshp214 sh[5065]: INFO > [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] > (Certificate Renewal Timer:) (logid:fe1554cc) Restarting libvirt after > certificate provisioning/renewal > Jan 31 11:17:07 cshp214 systemd[1]: Stopping CloudStack Agent... > Jan 31 11:17:07 cshp214 sh[5065]: INFO [cloud.agent.Agent] > (AgentShutdownThread:) (logid:) Stopping the agent: Reason = sig.kill > Jan 31 11:17:07 cshp214 sh[5065]: WARN [cloud.agent.Agent] > (Certificate Renewal Timer:) (logid:fe1554cc) Failed to execute post > certificate renewal command: > Jan 31 11:17:07 cshp214 sh[5065]: java.lang.IllegalStateException: > Shutdown in progress > Jan 31 11:17:07 cshp214 sh[5065]: #011at > java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > java.lang.Runtime.removeShutdownHook(Runtime.java:239) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > java.util.TimerThread.mainLoop(Timer.java:555) > Jan 31 11:17:07 cshp214 sh[5065]: #011at > java.util.TimerThread.run(Timer.java:505) > Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading > data: Input/output error > Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading > data: Input/output error > Jan 31 11:17:08 cshp214 systemd[1]: Stopped CloudStack Agent. > Jan 31 11:17:08 cshp214 systemd[1]: Stopping Virtualization daemon... > Jan 31 11:17:08 cshp214 systemd[1]: Stopped Virtualization daemon. > Jan 31 11:17:08 cshp214 systemd[1]: Starting Virtualization daemon... > Jan 31 11:17:08 cshp214 systemd[1]: Started Virtualization daemon. > Jan 31 11:17:08 cshp214 systemd[1]: Started CloudStack Agent. > Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN No appenders could be > found for logger (com.cloud.agent.AgentShell). > Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN Please initialize the > log4j system properly. > Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN See > http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. > Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Agent started > Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Implementation Version is 4.11.2.0 > Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell] > (main:) (logid:) agent.properties found at > /etc/cloudstack/agent/agent.properties > Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Defaulting to using properties file for storage > Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Defaulting to the constant time backoff algorithm > Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.utils.LogUtils] > (main:) (logid:) log4j configuration found at > /etc/cloudstack/agent/log4j-cloud.xml > Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Using default Java settings for IPv6 preference for > agent connection > Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.Agent] (main:) > (logid:) id is 5 > Jan 31 11:17:10 cshp214 sh[25387]: INFO > [kvm.resource.LibvirtConnection] (main:) (logid:) No existing libvirtd > connection found. Opening a new one > -- > Jan 31 11:17:16 cshp214 snmpd[2460]: error on subcontainer 'ia_addr' > insert (-1) > Jan 31 11:17:16 cshp214 snmpd[2460]: message repeated 3 times: [ error > on subcontainer 'ia_addr' insert (-1)] > Jan 31 11:17:20 cshp214 systemd[1]: cloudstack-agent.service: Service > hold-off time over, scheduling restart. > Jan 31 11:17:20 cshp214 systemd[1]: Stopped CloudStack Agent. > Jan 31 11:17:20 cshp214 systemd[1]: Started CloudStack Agent. > Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN No appenders could be > found for logger (com.cloud.agent.AgentShell). > Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN Please initialize the > log4j system properly. > Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN See > http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. > Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Agent started > Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Implementation Version is 4.11.2.0 > Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell] > (main:) (logid:) agent.properties found at > /etc/cloudstack/agent/agent.properties > Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Defaulting to using properties file for storage > Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Defaulting to the constant time backoff algorithm > Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.utils.LogUtils] > (main:) (logid:) log4j configuration found at > /etc/cloudstack/agent/log4j-cloud.xml > Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell] > (main:) (logid:) Using default Java settings for IPv6 preference for > agent connection > Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.Agent] (main:) > (logid:) id is 5 > Jan 31 11:17:21 cshp214 sh[25457]: INFO > [kvm.resource.LibvirtConnection] (main:) (logid:) No existing libvirtd > connection found. Opening a new one > Jan 31 11:17:21 cshp214 sh[25457]: libvirt: XML-RPC error : Failed to > connect socket to '/var/run/libvirt/libvirt-sock': No such file or > directory > Jan 31 11:17:21 cshp214 sh[25457]: ERROR [cloud.agent.AgentShell] > (main:) (logid:) Unable to start agent: > Jan 31 11:17:21 cshp214 sh[25457]: > com.cloud.utils.exception.CloudRuntimeException: Failed to connect > socket to '/var/run/libvirt/libvirt-sock': No such file or directory > Jan 31 11:17:21 cshp214 sh[25457]: #011at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914) > Jan 31 11:17:21 cshp214 sh[25457]: #011at > com.cloud.agent.Agent.<init>(Agent.java:190) > Jan 31 11:17:21 cshp214 sh[25457]: #011at > com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453) > Jan 31 11:17:21 cshp214 sh[25457]: #011at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422) > Jan 31 11:17:21 cshp214 sh[25457]: #011at > com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406) > Jan 31 11:17:21 cshp214 sh[25457]: #011at > com.cloud.agent.AgentShell.start(AgentShell.java:512) > Jan 31 11:17:21 cshp214 sh[25457]: #011at > com.cloud.agent.AgentShell.main(AgentShell.java:547) > Jan 31 11:17:21 cshp214 sh[25457]: Unable to start agent: Failed to > connect socket to '/var/run/libvirt/libvirt-sock': No such file or > directory > Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Main > process exited, code=exited, status=67/n/a > Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Unit > entered failed state. > Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Failed > with result 'exit-code'. > Jan 31 11:17:21 cshp214 dnsmasq[4000]: read /etc/hosts - 13 addresses > Jan 31 11:17:21 cshp214 dnsmasq[4000]: read > /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses > Jan 31 11:17:21 cshp214 dnsmasq-dhcp[4000]: read > /var/lib/libvirt/dnsmasq/default.hostsfile > Jan 31 11:17:22 cshp214 snmpd[2460]: Connection from UDP: > [127.0.0.1]:37699->[127.0.0.1]:161 > Jan 31 11:17:24 cshp214 snmpd[2460]: message repeated 2 times: [ > Connection from UDP: [127.0.0.1]:37699->[127.0.0.1]:161] > Jan 31 11:17:24 cshp214 libvirtd[25368]: libvirt version: 1.3.1, > package: 1ubuntu10.24 (Marc Deslauriers <marc.deslauri...@ubuntu.com> > Wed, 23 May 2018 13:29:29 -0400) > Jan 31 11:17:24 cshp214 libvirtd[25368]: hostname: cshp214 > Jan 31 11:17:24 cshp214 libvirtd[25368]: Configured security driver > "none" disables default policy to create confined guests > Jan 31 11:17:25 cshp214 libvirtd[25368]: unsupported configuration: > Security driver apparmor not enabled > > > Can anyone help me? > > Il 30/01/19 13:37, Rohit Yadav ha scritto: >> >> Hi Ugo, >> >> >> This will be a one-time procedure, and the KVM host and the VMs do >> not need a reboot but the provisionCertificate API will restart the >> libvirtd process (just check if that can have any side effects for >> your VMs/distro, on most modern distros restarting libvirtd does not >> have any side-effects on existing running VMs). >> >> >> - Rohit >> >> >> >> rohit.ya...@shapeblue.com >> www.shapeblue.com<http://www.shapeblue.com> >> @shapeblue >> >> ------------------------------------------------------------------------ >> *From:* Ugo Vasi <ugo.v...@procne.it> >> *Sent:* Wednesday, January 30, 2019 4:47:09 PM >> *To:* users@cloudstack.apache.org; Rohit Yadav >> *Subject:* Re: secure hosts communications >> Hi Rohit, >> I have a 4.11.2.0 ACS infrastructure (Ubuntu 16.04 with KVM hypervisor) >> I see that all the hosts are in unsecure state from the UI and so the >> live migration don't works (we had trubles with mgmt server). >> >> I read in the documentation that launching the provisionCertificate API >> (by pressing the appropriate button in the UI) the certificates will be >> renewed/regenerated for already connected agents/hosts. >> >> I do not understand if provisioning should be done manually on each host >> or if the procedure should be done only once. >> >> Do this procedure reboot the host or the instances that it contains? >> >> >> Thanks >> >> >> >> Il 27/11/18 09:49, Rohit Yadav ha scritto: >> > Hi Richard, >> > >> > >> > Please read: >> http://docs.cloudstack.apache.org/en/4.11.2.0/adminguide/hosts.html#security >> > >> > >> > 4.11.2 is out, please consider using it instead of 4.11.1 as it has >> several bugfixes etc. >> > >> > In short, with all of your KVM hosts up and connected to mgmt >> server, first change the auth strictness global setting to true, then >> using API secure the hosts using the provisionCertificate API. In the >> UI, go to your hosts that don't show up as secure and click on the >> key button (a new button) to secure the host which calls the >> provisionCertificate API as well. >> > >> > >> > - Rohit >> > >> > <https://cloudstack.apache.org> >> > >> > >> > >> > ________________________________ >> > From: Richard Persaud <richard.pers...@macys.com> >> > Sent: Monday, November 26, 2018 8:19:56 PM >> > To: users@cloudstack.apache.org >> > Subject: RE: secure hosts communications >> > >> > Thank you, Rohit. >> > >> > I am using 4.11.1 with a full KVM environment. They are showing >> unsecure with strictness set to true. >> > >> > What configuration needs to be adjusted to have the KVM hosts show >> secure? >> > >> > Regards, >> > >> > Richard Persaud >> > >> > From: Rohit Yadav <rohit.ya...@shapeblue.com> >> > Sent: Saturday, November 24, 2018 2:02 PM >> > To: users@cloudstack.apache.org >> > Subject: Re: secure hosts communications >> > >> > ⚠ EXT MSG: >> > >> > Richard, >> > >> > >> > Starting 4.11, agent and management servers will use an in-built CA >> framework to secured hosts. Only in case of KVM hosts you may see an >> insecure state, otherwise all KVM hosts (agents) and SSVM/CPVM agents >> will by default in Up state will be secured. There is an auth >> strictness setting that should be true. >> > >> > >> > >> > - Rohit >> > >> > <https://cloudstack.apache.org> >> > >> > >> > >> > ________________________________ >> > From: Richard Persaud >> <richard.pers...@macys.com<mailto:richard.pers...@macys.com>> >> > Sent: Saturday, November 24, 2018 4:21:24 AM >> > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> >> > Subject: secure hosts communications >> > >> > Hello, >> > >> > Is there straight-forward to enable secure communications between >> the management and the hosts? >> > >> > I have looked at many documentations but am still unable to get the >> hosts to show a "secure" state. >> > >> > Regards, >> > >> > Richard Persaud >> > >> > >> > rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com> >> > >> www.shapeblue.com<https://isolate.menlosecurity.com/0/eJyrViotylGyUsooKSmw0tcvLy_XK85ILEhNyilN1UvOz1XSUSrKV7Iy1FEqyUwBqjM0MFaqBQDf4BCe> >> > Amadeus House, Floral Street, London WC2E 9DPUK >> > @shapeblue >> > >> > >> > >> > >> > * This is an EXTERNAL EMAIL. Stop and think before clicking a link >> or opening attachments. >> > >> > rohit.ya...@shapeblue.com >> > www.shapeblue.com<http://www.shapeblue.com> <http://www.shapeblue.com> >> > Amadeus House, Floral Street, London WC2E 9DPUK >> > @shapeblue >> > >> > >> > >> > >> > >> > >> >> >> -- >> >> *Ugo Vasi* / System Administrator >> ugo.v...@procne.it <mailto:ugo.v...@procne.it> >> >> >> >> >> *Procne S.r.l.* >> +39 0432 486 523 >> via Cotonificio, 45 >> 33010 Tavagnacco (UD) >> www.procne.it<http://www.procne.it> <http://www.procne.it> >> <http://www.procne.it/> >> >> >> Le informazioni contenute nella presente comunicazione ed i relativi >> allegati possono essere riservate e sono, comunque, destinate >> esclusivamente alle persone od alla Società sopraindicati. La >> diffusione, distribuzione e/o copiatura del documento trasmesso da parte >> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi >> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003 >> "Codice in materia di protezione dei dati personali". Se avete ricevuto >> questo messaggio per errore, vi preghiamo di distruggerlo e di informare >> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail >> i...@procne.it <mailto:i...@procne.it>. >> > > -- *Ugo Vasi* / System Administrator ugo.v...@procne.it <mailto:ugo.v...@procne.it> *Procne S.r.l.* +39 0432 486 523 via Cotonificio, 45 33010 Tavagnacco (UD) www.procne.it<http://www.procne.it> <http://www.procne.it/> Le informazioni contenute nella presente comunicazione ed i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone od alla Società sopraindicati. La diffusione, distribuzione e/o copiatura del documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003 "Codice in materia di protezione dei dati personali". Se avete ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di informare immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail i...@procne.it <mailto:i...@procne.it>. rohit.ya...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue
