Certs are important, it's time to make sure that clocks are correct on all hosts and client/middleware/server all have certs that each other can validate.
If your certs are fine and you are still seeing "Connection reset by peer". It's time to crank up debug logs everywhere (mcollective server and activemq middleware and mcollective client) to see what's going on. If you don't get a useful answer from the debug logs then it's time to capture packets (tcpdump) and look at the initial ssl handshake. However, from somebody who has been there, save yourself a bunch of pain and use this thing for your initial setup: http://choria.io/ On Mon, Jan 30, 2017 at 05:35:23PM -0800, Isabell Cowan wrote: > I've been trying all do to set up MCollective on my puppet cluster. No > matter what I do, I can't seem to get the MCollective server to connect. > The MCollective server(s) are running mostly on Ubnutu Xenial. The > ActiveMQ broker (5.14.3) is running on Debian Stretch. I'm running puppet > 4.x on all nodes. I've used every transport connector I can thing if, and > they all fail to connect. Let me dump some log files at you. > In mcollective.log I'm getting `Connection reset by peer`: > I, [2017-01-27T15:43:59.869501 #18729] INFO -- : activemq.rb:139:in > `on_ssl_connecting' Establishing SSL session with > stomp+ssl://[email protected]:61614 > E, [2017-01-27T15:44:00.070995 #18729] ERROR -- : activemq.rb:149:in > `on_ssl_connectfail' SSL session creation with > stomp+ssl://[email protected]:61614 failed: Connection reset > by peer - SSL_connect > I, [2017-01-27T15:44:00.071371 #18729] INFO -- : activemq.rb:129:in > `on_connectfail' TCP Connection to > stomp+ssl://[email protected]:61614 failed on attempt 24 > Oddly enough, in the ActiveMQ log, I also seem to be getting `Connection > reset by peer`: > ERROR | Could not accept connection from null : {} > java.io.IOException: java.io.IOException: Connection reset by peer > at > > org.apache.activemq.transport.nio.NIOSSLTransport.initializeStreams(NIOSSLTransport.java:188)[activemq-client.jar:] > at > > org.apache.activemq.transport.stomp.StompNIOSSLTransport.initializeStreams(StompNIOSSLTransport.java:57)[activemq-stomp.jar:] > at > > org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:543)[activemq-client.jar:] > at > > org.apache.activemq.transport.nio.NIOTransport.doStart(NIOTransport.java:174)[activemq-client.jar:] > at > > org.apache.activemq.transport.nio.NIOSSLTransport.doStart(NIOSSLTransport.java:462)[activemq-client.jar:] > at > > org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client.jar:] > at > > org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)[activemq-client.jar:] > at > > org.apache.activemq.transport.stomp.StompTransportFilter.start(StompTransportFilter.java:65)[activemq-stomp.jar:] > at > > org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:169)[activemq-client.jar:] > at > > org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)[activemq-client.jar:] > at > > org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072)[activemq-broker.jar:] > at > > org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218)[activemq-broker.jar:] > at java.lang.Thread.run(Thread.java:745)[:1.8.0_111] > So they're both resetting the connection. Huh. Before you ask, no: there > are no iptables rules, and yes: there is a route between the two nodes. > Let's take a peak at `lsof -i` just to be sure and then I'll throw some > config files at you. > java 20833 activemq 84u IPv6 53552 0t0 TCP *:61614 (LISTEN) > activemq.xml: > <!DOCTYPE activemq [ > <!ENTITY keyStores SYSTEM "keyStores.xml"> > ]> > <beans > xmlns="http://www.springframework.org/schema/beans"; > xmlns:amq="http://activemq.apache.org/schema/core"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd";> > <bean > > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/> > <broker xmlns="http://activemq.apache.org/schema/core"; > useJmx="false" > brokerName="broker" > dataDirectory="${activemq.base}/data"> > <persistenceAdapter> > <kahaDB directory="${activemq.base}/data/kahadb"/> > </persistenceAdapter> > <sslContext> > &keyStores; > </sslContext> > <transportConnectors> > <transportConnector > name="stomp+nio" > > > uri="stomp+nio+ssl://0.0.0.0:61614?needClientAuth=true&transport.enabledProtocols=TLSv1,TLSv1.1,TLSv1.2"/> > </transportConnectors> > <plugins> > <simpleAuthenticationPlugin> > <users> > <authenticationUser username="mcollective" > password="password" groups="mcollective,everyone"/> > <authenticationUser username="admin" password="password" > groups="mcollective,admins,everyone"/> > </users> > </simpleAuthenticationPlugin> > <authorizationPlugin> > <map> > <authorizationMap> > <authorizationEntries> > <authorizationEntry queue=">" write="admins" > read="admins" admin="admins" /> > <authorizationEntry topic=">" write="admins" > read="admins" admin="admins" /> > <authorizationEntry topic="mcollective.>" > write="mcollective" read="mcollective" admin="mcollective" /> > <authorizationEntry queue="mcollective.>" > write="mcollective" read="mcollective" admin="mcollective" /> > <authorizationEntry topic="ActiveMQ.Advisory.>" > read="everyone" write="everyone" admin="everyone"/> > </authorizationEntries> > </authorizationMap> > </map> > </authorizationPlugin> > </plugins> > </broker> > </beans> > keyStores.xml: > <sslContext > keyStore="/etc/activemq/keystore.jks" > keyStorePassword="password" > trustStore="/etc/activemq/truststore.jks" > trustStorePassword="password" /> > mcollective/server.cfg: > connector = activemq > direct_addressing = 1 > plugin.activemq.pool.size = 1 > plugin.activemq.pool.1.host = broker.example.com > plugin.activemq.pool.1.port = 61614 > plugin.activemq.pool.1.user = mcollective > plugin.activemq.pool.1.password = password > plugin.activemq.pool.1.ssl = 1 > plugin.activemq.pool.1.ssl.ca = /etc/puppetlabs/puppet/ssl/certs/ca.pem > plugin.activemq.pool.1.ssl.cert = > /etc/puppetlabs/puppet/ssl/certs/mail.example.com.pem > plugin.activemq.pool.1.ssl.key = > /etc/puppetlabs/puppet/ssl/private_keys/mail.example.com.pem > plugin.activemq.pool.1.ssl.fallback = 0 > securityprovider = ssl > plugin.ssl_client_cert_dir = /etc/puppetlabs/mcollective/clients > plugin.ssl_server_private = > /etc/puppetlabs/mcollective/server_private.pem > plugin.ssl_server_public = /etc/puppetlabs/mcollective/server_public.pem > identity = mail.example.com > factsource = yaml > plugin.yaml = /etc/puppetlabs/mcollective/facts.yaml > classesfile = /var/lib/puppet/state/classes.txt > collectives = mcollective > main_collective = mcollective > registerinterval = 600 > rpcaudit = 1 > rpcauditprovider = logfile > plugin.rpcaudit.logfile = /var/log/mcollective-audit.log > logger_type = file > loglevel = debug > logfile = /var/log/mcollective.log > keeplogs = 5 > max_log_size = 2097152 > logfacility = user > libdir = /usr/share/mcollective/plugins > daemonize = 1 > The keys/certs in keystore.jks and truststore.jks are correct and so is > the password. The shared keys and certs are also available. So lets try > connecting with `openssl`: > root@mail:/etc/puppetlabs/puppet/ssl# openssl s_client -connect > broker.example.com:61614 -CAfile certs/ca.pem -cert > certs/mail.example.com.pem -key private_keys/mail.example.com.pem > CONNECTED(00000003) > write:errno=104 > --- > no peer certificate available > --- > No client certificate CA names sent > --- > SSL handshake has read 0 bytes and written 305 bytes > --- > New, (NONE), Cipher is (NONE) > Secure Renegotiation IS NOT supported > Compression: NONE > Expansion: NONE > No ALPN negotiated > SSL-Session: > Protocol : TLSv1.2 > Cipher : 0000 > Session-ID: > Session-ID-ctx: > Master-Key: > Key-Arg : None > PSK identity: None > PSK identity hint: None > SRP username: None > Start Time: 1485554633 > Timeout : 300 (sec) > Verify return code: 0 (ok) > --- > I'm not exactly sure how to interpret this openssl output, maybe someone > else knows. > Any new good guesses would be helpful, I'm stuck. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "mcollective-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [1][email protected]. > For more options, visit [2]https://groups.google.com/d/optout. > > References > > Visible links > 1. mailto:[email protected] > 2. https://groups.google.com/d/optout -- --- You received this message because you are subscribed to the Google Groups "mcollective-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
