Hi Adam, the mentioned bug seems to be fixed in a 4.11 release and you're on 4.13.1 so ideally you shouldn't hit that.
The issue seems to be that the agents got certificates created with a root CA that is not validated by the (additional) management servers. Some hints and checks you can perform: * Did you add all the three management servers simultaneously? * Can you restart all the management servers one by one and see if agents still fail to connect * To manually re-key the agents, you can set ca.plugin.root.auth.strictness global setting to false (no need to restart the mgmt server) which will allow the agents to connect and then using API or UI->Infra-> KVM hosts -> provision certificates again (or use API provisionCertificate for hosts and cpvm/ssvm) * Last resort, backup DB and delete the ca.plugin.root.public.key, ca.plugin.root.private.key, ca.plugin.root.ca.certificate and stop all mgmt server, start one mgmt server and when it's online start remaining. This will re-create root CA keypair and cert and perform the previous step (change auth strictness to false and re-key the agents; Hope this helps. Regards. ________________________________ From: Adam Witwicki <awitwi...@oakfordis.com> Sent: Monday, August 17, 2020 15:52 To: users@cloudstack.apache.org <users@cloudstack.apache.org> Subject: Cant add additional management servers with multiple IPs Hi Guys Trying to set up cloudstack 4.13.1, but I am getting SSL cert errors on the 2 additional management servers I'm trying to setup. These servers have more than one IP - could it be related to this bug https://github.com/apache/cloudstack/issues/2530 Name : cloudstack-management Arch : x86_64 Version : 4.13.1.0 Release : shapeblue0.el7 Error from 1st management server 2020-08-17 10:43:56,747 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-60-thread-1:null) (logid:) Certificate ownership verification failed for client: 10.10.216.221 2020-08-17 10:43:56,747 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-5:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/10.10.216.200:8250, remote address=/10.10.216.221:53568. 2020-08-17 10:43:56,797 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-61-thread-1:null) (logid:) Certificate ownership verification failed for client: 10.10.216.221 2020-08-17 10:43:56,798 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-5:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/10.10.216.200:8250, remote address=/10.10.216.221:53570. Error from additional management server I'm trying to add 2020-08-17 10:43:56,640 ERROR [c.c.u.n.Link] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/10.10.216.221:53564, remote address=/10.10.216.200:8250. The client may have invalid ca-certificates. 2020-08-17 10:43:56,641 WARN [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) Unable to connect to peer management server: 168482836, ip: 10.10.216.200 due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '168482836' on 10.10.216.200:8250 java.io.IOException: SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '168482836' on 10.10.216.200:8250 2020-08-17 10:43:56,641 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) Seq 66-1928103590467993603: Unable to forward null 2020-08-17 10:43:56,641 WARN [c.c.a.m.AgentManagerImpl] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) Resource [Host:66] is unreachable: Host 66: Unable to reach the peer that the agent is connected 2020-08-17 10:43:56,641 WARN [c.c.r.ResourceManagerImpl] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) Unable to obtain host 66 statistics. 2020-08-17 10:43:56,641 WARN [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) The Host stats is null for host: 66 2020-08-17 10:43:56,698 ERROR [c.c.u.n.Link] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/10.10.216.221:53566, remote address=/10.10.216.200:8250. The client may have invalid ca-certificates. 2020-08-17 10:43:56,698 WARN [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) Unable to connect to peer management server: 168482836, ip: 10.10.216.200 due to SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '168482836' on 10.10.216.200:8250 java.io.IOException: SSL: Fail to init SSL! java.io.IOException: SSL: Handshake failed with peer management server '168482836' on 10.10.216.200:8250 2020-08-17 10:43:56,699 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) Seq 69-2867104112774742021: Unable to forward null 2020-08-17 10:43:56,748 ERROR [c.c.u.n.Link] (StatsCollector-2:ctx-aa7d0a75) (logid:10ec5992) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/10.10.216.221:53568, remote address=/10.10.216.200:8250. The client may have invalid ca-certificates. I thought I solved this by following http://mail-archives.apache.org/mod_mbox/cloudstack-users/201805.mbox/%3cvi1pr0701mb186911b8e6ba4b81e00ea963e9...@vi1pr0701mb1869.eurprd07.prod.outlook.com%3E But when adding KVM agents I get this on the management server address=/10.10.216.222:38570. 2020-08-17 11:18:13,195 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-13-thread-1:null) (logid:) Certificate ownership verification failed for client: 10.10.216.221 2020-08-17 11:18:13,196 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/10.10.216.200:8250, remote address=/10.10.216.221:33998. 2020-08-17 11:18:13,277 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-14-thread-1:null) (logid:) Certificate ownership verification failed for client: 10.10.216.221 2020-08-17 11:18:13,278 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: General SSLEngine problem, for local address=/10.10.216.200:8250, remote address=/10.10.216.221:34000. Any help appricated Thanks Adam Disclaimer Notice: This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking. This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited. Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited. Registered address: Oakford Technology Limited, The Manor House, Potterne, Wiltshire. SN10 5PN. Registered in England and Wales No. 5971519 rohit.ya...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue
