sjyang18 opened a new issue #2092:
URL: https://github.com/apache/accumulo/issues/2092
**Describe the bug**
When TLS is enabled & configured in the zookeepers, accumulo monitor still
continuously use normal Transport connections to zookeepers to get zookeeper
status. This generates unsuccessful handshake errors and NotSslRecordException
in zookeepers continuously. Looking at code, TTimeoutTransport class seems to
be used, which fails to do handshaking with TLS-enabled zookeepers.
**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: [e.g. 1.10.0] 2.1.0
- OS: [e.g. CentOS 7.5]
- Others:
**To Reproduce**
Steps to reproduce the behavior (or a link to an example repository that
reproduces the problem):
1. Turn TLS on zookeepers. And, configure Accumulo to use TLS as
documented.
2. Add additional zookeeper client libraries and CLIENT_JVMFLAGS to
accumulo-env.sh
```
add_jar_prefix_to_classpath "${ZOOKEEPER_HOME}/lib/netty-"
CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
\
-Dzookeeper.client.secure=true \
-Dzookeeper.ssl.keyStore.location=/opt/muchos/install/ssl/host-keystore.jks \
-Dzookeeper.ssl.keyStore.password=hadoop \
-Dzookeeper.ssl.trustStore.location=/opt/muchos/install/ssl/truststore.jks
\
-Dzookeeper.ssl.trustStore.password=hadoop"
JAVA_OPTS=("${JAVA_OPTS[@]}" $CLIENT_JVMFLAGS)
```
3. accumulo-cluster start
4. Tail the zookeeper log and observer 'Unsuccessful handshake with session
and io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Additional context**
Accumulo monitor thread who calls ZookeeperStatus.
```
"ZooKeeperStatus" #63 daemon prio=5 os_prio=0 tid=0x000055f643762000
nid=0x4351 waiting on condition [0x00007f3cadb49000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:340)
at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
at
org.apache.accumulo.fate.util.UtilWaitThread.sleepUninterruptibly(UtilWaitThread.java:55)
at
org.apache.accumulo.monitor.ZooKeeperStatus.run(ZooKeeperStatus.java:151)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:748)
```
Sample errors and exceptions in zookeeper logs.
```
2021-05-10 16:30:45,664 [myid:1] - ERROR
[nioEventLoopGroup-4-49:NettyServerCnxnFactory$CertificateVerifier@363] -
Unsuccessful handshake with session 0x0
2021-05-10 16:30:45,665 [myid:1] - WARN
[nioEventLoopGroup-4-49:NettyServerCnxnFactory$CnxnChannelHandler@220] -
Exception caught
io.netty.handler.codec.DecoderException:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 737461740a
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS
record: 737461740a
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
... 17 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
