Hi Rohit,
I tryed renew certificate but it failed!
Now libvirt does not restart and agent is disconnected:
agent log:
2019-01-31 11:17:07,530 INFO
[resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper]
(Certificate Renewal Timer:null) (logid:fe1554cc) Restarting libvirt
after certificate provisioning/renewal
2019-01-31 11:17:07,567 INFO [cloud.agent.Agent]
(AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2019-01-31 11:17:07,568 WARN [cloud.agent.Agent] (Certificate Renewal
Timer:null) (logid:fe1554cc) Failed to execute post certificate
renewal command:
java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
at
com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157)
at
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
2019-01-31 11:17:09,797 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Agent started
2019-01-31 11:17:09,800 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Implementation Version is 4.11.2.0
2019-01-31 11:17:09,802 INFO [cloud.agent.AgentShell] (main:null)
(logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2019-01-31 11:17:09,815 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Defaulting to using properties file for storage
2019-01-31 11:17:09,816 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Defaulting to the constant time backoff algorithm
2019-01-31 11:17:09,828 INFO [cloud.utils.LogUtils] (main:null)
(logid:) log4j configuration found at
/etc/cloudstack/agent/log4j-cloud.xml
2019-01-31 11:17:09,850 INFO [cloud.agent.AgentShell] (main:null)
(logid:) Using default Java settings for IPv6 preference for agent
connection
2019-01-31 11:17:09,998 INFO [cloud.agent.Agent] (main:null) (logid:)
id is 5
2019-01-31 11:17:10,030 INFO [kvm.resource.LibvirtConnection]
(main:null) (logid:) No existing libvirtd connection found. Opening a
new one
2019-01-31 11:17:10,175 ERROR [cloud.agent.AgentShell] (main:null)
(logid:) Unable to start agent:
com.cloud.utils.exception.CloudRuntimeException: Failed to connect
socket to '/var/run/libvirt/libvirt-sock': No such file or directory
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914)
at com.cloud.agent.Agent.<init>(Agent.java:190)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
(logs repeat)
syslog:
Jan 31 11:17:07 cshp214 sh[5065]: INFO
[resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper]
(Certificate Renewal Timer:) (logid:fe1554cc) Restarting libvirt after
certificate provisioning/renewal
Jan 31 11:17:07 cshp214 systemd[1]: Stopping CloudStack Agent...
Jan 31 11:17:07 cshp214 sh[5065]: INFO [cloud.agent.Agent]
(AgentShutdownThread:) (logid:) Stopping the agent: Reason = sig.kill
Jan 31 11:17:07 cshp214 sh[5065]: WARN [cloud.agent.Agent]
(Certificate Renewal Timer:) (logid:fe1554cc) Failed to execute post
certificate renewal command:
Jan 31 11:17:07 cshp214 sh[5065]: java.lang.IllegalStateException:
Shutdown in progress
Jan 31 11:17:07 cshp214 sh[5065]: #011at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
java.lang.Runtime.removeShutdownHook(Runtime.java:239)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1157)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:32)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
java.util.TimerThread.mainLoop(Timer.java:555)
Jan 31 11:17:07 cshp214 sh[5065]: #011at
java.util.TimerThread.run(Timer.java:505)
Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading
data: Input/output error
Jan 31 11:17:08 cshp214 libvirtd[4700]: End of file while reading
data: Input/output error
Jan 31 11:17:08 cshp214 systemd[1]: Stopped CloudStack Agent.
Jan 31 11:17:08 cshp214 systemd[1]: Stopping Virtualization daemon...
Jan 31 11:17:08 cshp214 systemd[1]: Stopped Virtualization daemon.
Jan 31 11:17:08 cshp214 systemd[1]: Starting Virtualization daemon...
Jan 31 11:17:08 cshp214 systemd[1]: Started Virtualization daemon.
Jan 31 11:17:08 cshp214 systemd[1]: Started CloudStack Agent.
Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN No appenders could be
found for logger (com.cloud.agent.AgentShell).
Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN Please initialize the
log4j system properly.
Jan 31 11:17:09 cshp214 sh[25387]: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Agent started
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Implementation Version is 4.11.2.0
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) agent.properties found at
/etc/cloudstack/agent/agent.properties
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Defaulting to using properties file for storage
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Defaulting to the constant time backoff algorithm
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.utils.LogUtils]
(main:) (logid:) log4j configuration found at
/etc/cloudstack/agent/log4j-cloud.xml
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Using default Java settings for IPv6 preference for
agent connection
Jan 31 11:17:09 cshp214 sh[25387]: INFO [cloud.agent.Agent] (main:)
(logid:) id is 5
Jan 31 11:17:10 cshp214 sh[25387]: INFO
[kvm.resource.LibvirtConnection] (main:) (logid:) No existing libvirtd
connection found. Opening a new one
--
Jan 31 11:17:16 cshp214 snmpd[2460]: error on subcontainer 'ia_addr'
insert (-1)
Jan 31 11:17:16 cshp214 snmpd[2460]: message repeated 3 times: [ error
on subcontainer 'ia_addr' insert (-1)]
Jan 31 11:17:20 cshp214 systemd[1]: cloudstack-agent.service: Service
hold-off time over, scheduling restart.
Jan 31 11:17:20 cshp214 systemd[1]: Stopped CloudStack Agent.
Jan 31 11:17:20 cshp214 systemd[1]: Started CloudStack Agent.
Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN No appenders could be
found for logger (com.cloud.agent.AgentShell).
Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN Please initialize the
log4j system properly.
Jan 31 11:17:20 cshp214 sh[25457]: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Agent started
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Implementation Version is 4.11.2.0
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) agent.properties found at
/etc/cloudstack/agent/agent.properties
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Defaulting to using properties file for storage
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Defaulting to the constant time backoff algorithm
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.utils.LogUtils]
(main:) (logid:) log4j configuration found at
/etc/cloudstack/agent/log4j-cloud.xml
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.AgentShell]
(main:) (logid:) Using default Java settings for IPv6 preference for
agent connection
Jan 31 11:17:21 cshp214 sh[25457]: INFO [cloud.agent.Agent] (main:)
(logid:) id is 5
Jan 31 11:17:21 cshp214 sh[25457]: INFO
[kvm.resource.LibvirtConnection] (main:) (logid:) No existing libvirtd
connection found. Opening a new one
Jan 31 11:17:21 cshp214 sh[25457]: libvirt: XML-RPC error : Failed to
connect socket to '/var/run/libvirt/libvirt-sock': No such file or
directory
Jan 31 11:17:21 cshp214 sh[25457]: ERROR [cloud.agent.AgentShell]
(main:) (logid:) Unable to start agent:
Jan 31 11:17:21 cshp214 sh[25457]:
com.cloud.utils.exception.CloudRuntimeException: Failed to connect
socket to '/var/run/libvirt/libvirt-sock': No such file or directory
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:914)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.Agent.<init>(Agent.java:190)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:453)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.start(AgentShell.java:512)
Jan 31 11:17:21 cshp214 sh[25457]: #011at
com.cloud.agent.AgentShell.main(AgentShell.java:547)
Jan 31 11:17:21 cshp214 sh[25457]: Unable to start agent: Failed to
connect socket to '/var/run/libvirt/libvirt-sock': No such file or
directory
Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Main
process exited, code=exited, status=67/n/a
Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Unit
entered failed state.
Jan 31 11:17:21 cshp214 systemd[1]: cloudstack-agent.service: Failed
with result 'exit-code'.
Jan 31 11:17:21 cshp214 dnsmasq[4000]: read /etc/hosts - 13 addresses
Jan 31 11:17:21 cshp214 dnsmasq[4000]: read
/var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jan 31 11:17:21 cshp214 dnsmasq-dhcp[4000]: read
/var/lib/libvirt/dnsmasq/default.hostsfile
Jan 31 11:17:22 cshp214 snmpd[2460]: Connection from UDP:
[127.0.0.1]:37699->[127.0.0.1]:161
Jan 31 11:17:24 cshp214 snmpd[2460]: message repeated 2 times: [
Connection from UDP: [127.0.0.1]:37699->[127.0.0.1]:161]
Jan 31 11:17:24 cshp214 libvirtd[25368]: libvirt version: 1.3.1,
package: 1ubuntu10.24 (Marc Deslauriers <marc.deslauri...@ubuntu.com>
Wed, 23 May 2018 13:29:29 -0400)
Jan 31 11:17:24 cshp214 libvirtd[25368]: hostname: cshp214
Jan 31 11:17:24 cshp214 libvirtd[25368]: Configured security driver
"none" disables default policy to create confined guests
Jan 31 11:17:25 cshp214 libvirtd[25368]: unsupported configuration:
Security driver apparmor not enabled
Can anyone help me?
Il 30/01/19 13:37, Rohit Yadav ha scritto:
Hi Ugo,
This will be a one-time procedure, and the KVM host and the VMs do
not need a reboot but the provisionCertificate API will restart the
libvirtd process (just check if that can have any side effects for
your VMs/distro, on most modern distros restarting libvirtd does not
have any side-effects on existing running VMs).
- Rohit
rohit.ya...@shapeblue.com
www.shapeblue.com
@shapeblue
------------------------------------------------------------------------
*From:* Ugo Vasi <ugo.v...@procne.it>
*Sent:* Wednesday, January 30, 2019 4:47:09 PM
*To:* users@cloudstack.apache.org; Rohit Yadav
*Subject:* Re: secure hosts communications
Hi Rohit,
I have a 4.11.2.0 ACS infrastructure (Ubuntu 16.04 with KVM hypervisor)
I see that all the hosts are in unsecure state from the UI and so the
live migration don't works (we had trubles with mgmt server).
I read in the documentation that launching the provisionCertificate API
(by pressing the appropriate button in the UI) the certificates will be
renewed/regenerated for already connected agents/hosts.
I do not understand if provisioning should be done manually on each host
or if the procedure should be done only once.
Do this procedure reboot the host or the instances that it contains?
Thanks
Il 27/11/18 09:49, Rohit Yadav ha scritto:
> Hi Richard,
>
>
> Please read:
http://docs.cloudstack.apache.org/en/4.11.2.0/adminguide/hosts.html#security
>
>
> 4.11.2 is out, please consider using it instead of 4.11.1 as it has
several bugfixes etc.
>
> In short, with all of your KVM hosts up and connected to mgmt
server, first change the auth strictness global setting to true, then
using API secure the hosts using the provisionCertificate API. In the
UI, go to your hosts that don't show up as secure and click on the
key button (a new button) to secure the host which calls the
provisionCertificate API as well.
>
>
> - Rohit
>
> <https://cloudstack.apache.org>
>
>
>
> ________________________________
> From: Richard Persaud <richard.pers...@macys.com>
> Sent: Monday, November 26, 2018 8:19:56 PM
> To: users@cloudstack.apache.org
> Subject: RE: secure hosts communications
>
> Thank you, Rohit.
>
> I am using 4.11.1 with a full KVM environment. They are showing
unsecure with strictness set to true.
>
> What configuration needs to be adjusted to have the KVM hosts show
secure?
>
> Regards,
>
> Richard Persaud
>
> From: Rohit Yadav <rohit.ya...@shapeblue.com>
> Sent: Saturday, November 24, 2018 2:02 PM
> To: users@cloudstack.apache.org
> Subject: Re: secure hosts communications
>
> ⚠ EXT MSG:
>
> Richard,
>
>
> Starting 4.11, agent and management servers will use an in-built CA
framework to secured hosts. Only in case of KVM hosts you may see an
insecure state, otherwise all KVM hosts (agents) and SSVM/CPVM agents
will by default in Up state will be secured. There is an auth
strictness setting that should be true.
>
>
>
> - Rohit
>
> <https://cloudstack.apache.org>
>
>
>
> ________________________________
> From: Richard Persaud
<richard.pers...@macys.com<mailto:richard.pers...@macys.com>>
> Sent: Saturday, November 24, 2018 4:21:24 AM
> To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
> Subject: secure hosts communications
>
> Hello,
>
> Is there straight-forward to enable secure communications between
the management and the hosts?
>
> I have looked at many documentations but am still unable to get the
hosts to show a "secure" state.
>
> Regards,
>
> Richard Persaud
>
>
> rohit.ya...@shapeblue.com<mailto:rohit.ya...@shapeblue.com>
>
www.shapeblue.com<https://isolate.menlosecurity.com/0/eJyrViotylGyUsooKSmw0tcvLy_XK85ILEhNyilN1UvOz1XSUSrKV7Iy1FEqyUwBqjM0MFaqBQDf4BCe>
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
>
> * This is an EXTERNAL EMAIL. Stop and think before clicking a link
or opening attachments.
>
> rohit.ya...@shapeblue.com
> www.shapeblue.com <http://www.shapeblue.com>
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
>
>
>
--
*Ugo Vasi* / System Administrator
ugo.v...@procne.it <mailto:ugo.v...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it> <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
i...@procne.it <mailto:i...@procne.it>.
