I think I've found the way to connect SolrCloud to an external ZooKeeper
ensemble via SSL.
By default, Solr does not use SSL to connect to ZooKeeper. So if the
ZooKeeper configuration requires SSL for client connections, Solr will
complain like this when it tries to connect to ZooKeeper:
--8<---------------cut here---------------start------------->8---
WARN - 2022-03-25 12:34:43.681; org.apache.zookeeper.ClientCnxn; Session 0x0
for sever localhost/127.0.0.1:2182, Closing socket connection. Attempting
reconnect except it is a SessionExpiredException. => EndOfStreamException:
Unable to read additional data from server sessionid 0x0, likely server has
closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional
data from server sessionid 0x0, likely server has closed socket
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
~[zookeeper-3.6.2.jar:3.6.2]
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
~[zookeeper-3.6.2.jar:3.6.2]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1275)
~[zookeeper-3.6.2.jar:3.6.2]
--8<---------------cut here---------------end--------------->8---
On the ZooKeeper side, the corresponding log entry is something like
this:
--8<---------------cut here---------------start------------->8---
2022-03-25 12:34:43,652 [myid:1] - ERROR
[nioEventLoopGroup-4-2:NettyServerCnxnFactory$CertificateVerifier@448] -
Unsuccessful handshake with session 0x0
2022-03-25 12:34:43,682 [myid:1] - WARN
[nioEventLoopGroup-4-2:NettyServerCnxnFactory$CnxnChannelHandler@284] -
Exception caught
io.netty.handler.codec.DecoderException:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
0000002d000000000000000000000000000075300000000000000000000000100000000000000000000000000000000000
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1232)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1300)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
... 17 more
--8<---------------cut here---------------end--------------->8---
This error message indicates that ZooKeeper was expecting an SSL
connection, but the client (i.e. Solr) was connecting without SSL.
The solution is to add the appropriate ZooKeeper Java properties. Notice
that these are exactly the same properties needed by standalone
ZooKeeper's 'zkServer.sh' and 'zkCli.sh' to connect to ZooKeeper via
SSL [1] [2]. Add the following to bin/solr.in.sh:
--8<---------------cut here---------------start------------->8---
SOLR_OPTS="$SOLR_OPTS
-Dzookeeper.client.secure=true
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.ssl.keyStore.location=/path/to/zk-keystore.jks
-Dzookeeper.ssl.keyStore.password=thepassword
-Dzookeeper.ssl.trustStore.location=/path/to/zk-truststore.jks
-Dzookeeper.ssl.trustStore.password=thepassword"
--8<---------------cut here---------------end--------------->8---
[1]: https://stackoverflow.com/questions/43930797/configuring-ssl-in-zookeeper
[2]:
https://cwiki.apache.org/confluence/display/zookeeper/zookeeper+ssl+user+guide
(Note that this ^ webpage says, "There is currently no support for
SSL for the communication between ZooKeeper servers". That statement
is no longer correct. "Quorum TLS" is available from ZooKeeper 3.5.5
onwards).
