Hi folks,
Trying to debug a tricky problem, and I'm beginning to suspect it's a
bug in the ActiveMQ server and/or client. The client sometimes ends up
using a wrong truststore, in spite of seemingly correct configuration.
Server: 2.31.0
Client: 2.38 (via Spring Boot)
The client is connecting to the server with the following connection
string:
(tcp://server1.example.com:61616,tcp://server1.example.com:61616)?sslEnabled=true&trustStorePath=/path/to/truststore.jks&trustStorePassword=foobar
The client manages to connect to Server 1 and is able to consume and
produce messages, this is fine.
However, the server seems to broadcast a topology update
(ClusterTopologyChangeMessage_V4) containing the following:
?port=61616&keyStorePassword=****&sslEnabled=true&host=server1-example-com&keyStorePath=/path/to/broker-ks&verifyHostName=false
and
?port=61616&keyStorePassword=****&sslEnabled=true&host=server2-example-com&keyStorePath=/path/to/broker-ks&verifyHostName=false
The client then seems to attempt to connect to Server 2 using the path
to the keystore provided in the ClusterTopologyChangeMessage, instead of
the the path I configured in the client connection string ???
I don't understand why this is happening. In fact, I don't even
understand why the server would broadcast the path to its own keystore
(and apparently also its password?) to clients. This information is
useless to the client, and it makes no sense for the client to attempt
to use this nonexistent path as a truststore.
We're using Spring Boot, so I'm not 100% certain that the problem isn't
with Spring, but as far as I can tell the correct connection string is
passed to a connection factory, and then somehow it ends up using a
wrong path to a truststore anyway.
The end result is that while things seem to work fine, the log is full
of errors (java.lang.Exception: Failed to find a store at
/path/to/broker.ks). Full stack trace at the bottom of the message.
Presumably things will stop working when we attempt to fail over to
Server 2.
Could someone be so kind as to point me in the right direction for
getting to the bottom of this?
Thanks,
Elric
Full stacktrace:
java.lang.Exception: Failed to find a store at /path/to/broker.ks
at
org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.validateStoreURL(SSLSupport.java:399)
~[artemis-core-client-2.28.0.jar!/:2.28.0]
at
org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.loadKeystore(SSLSupport.java:339)
~[artemis-core-client-2.28.0.jar!/:2.28.0]
at
org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.loadKeyManagerFactory(SSLSupport.java:375)
~[artemis-core-client-2.28.0.jar!/:2.28.0]
at
org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.loadKeyManagers(SSLSupport.java:355)
~[artemis-core-client-2.28.0.jar!/:2.28.0]
at
org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.createContext(SSLSupport.java:222)
~[artemis-core-client-2.28.0.jar!/:2.28.0]
at
org.apache.activemq.artemis.core.remoting.impl.ssl.DefaultSSLContextFactory.getSSLContext(DefaultSSLContextFactory.java:50)
~[artemis-core-client-2.28.0.jar!/:2.28.0]
at
org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.loadJdkSslEngine(NettyConnector.java:786)
~[artemis-core-client-2.28.0.jar!/:2.28.0]
at
org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$1.initChannel(NettyConnector.java:690)
~[artemis-core-client-2.28.0.jar!/:2.28.0]
at
io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416)
~[netty-transport-classes-epoll-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
~[artemis-commons-2.28.0.jar!/:na]
