[
https://issues.apache.org/jira/browse/ZOOKEEPER-4247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300166#comment-17300166
]
Mate Szalay-Beko edited comment on ZOOKEEPER-4247 at 3/12/21, 9:28 AM:
-----------------------------------------------------------------------
thanks [~TheDevarshiShah] for reporting the issue!
looks like the server is trying to process an incomming message before the
zkServer object would be initialized.
[https://github.com/apache/zookeeper/blob/803c7f1a12f85978cb049af5e4ef23bd8b688715/zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxn.java#L518]
Before this line, I think we should check if zkServer is null. If it is null,
then we should either
* ignore the message
* or throw an IOException + close the connection
I'm not sure which of the above solution is the better. Whoever is doing this
fix, she/he should think this through and also write some tests (which can be
tricky, as this can be some race condition triggered for some reason in
kubernetes).
Unfortunately I don't have free cycles at the moment, but I hope someone else
has.
was (Author: symat):
thanks [~TheDevarshiShah] for reporting the issue!
looks like the server is trying to process an incomming message before the
zkServer object would be initialized.
https://github.com/apache/zookeeper/blob/803c7f1a12f85978cb049af5e4ef23bd8b688715/zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxn.java#L518
Before this line, I think we should check if zkServer is null. If it is null,
then we should either
* ignore the message
* or throw an IOException + close the connection
I'm not sure which of the above solution is the better. Whoever is doing this
fix, she/he should think this through and also write some tests (which can be
tricky, as this can some race condition triggered for some reason in
kubernetes).
Unfortunately I don't have free cycles at the moment, but I hope someone else
has.
> org.apache.zookeeper.server.NettyServerCnxn.receiveMessage throws
> NullPointerException
> --------------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-4247
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4247
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.6.2
> Environment: K8S
> Reporter: Devarshi Shah
> Priority: Major
>
> *Problem:*
> While upgrading K8S cluster, container running Zookeeper (during serving it's
> client) will rollover one by one.
> During this rollover, +Null Pointer Exception+ was observed as below.
> After updating to the latest Zookeeper 3.6.2 we still see the problem.
> This is happening on a fresh install (and has all the time).
>
> *Stack-trace**:*
> <from zk-pod-0-log>
> {code:java}
> 2021-02-08T12:42:08.229+0000 [myid:] - ERROR
> [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@329] -
> Unexpected exception in receive
> java.lang.NullPointerException: null
> at
> org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:518)
> ~[zookeeper-3.6.2.jar:3.6.2]
> at
> org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:368)
> ~[zookeeper-3.6.2.jar:3.6.2]
> at
> org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:326)
> [zookeeper-3.6.2.jar:3.6.2]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> [netty-common-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [netty-common-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> [netty-common-4.1.50.Final.jar:4.1.50.Final]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> 2021-02-08T12:42:08.229+0000 [myid:] - WARN
> [nioEventLoopGroup-4-1:NettyServerCnxnFactory$CnxnChannelHandler@273] -
> Exception caught
> java.lang.NullPointerException: null
> at
> org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:518)
> ~[zookeeper-3.6.2.jar:3.6.2]
> at
> org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:368)
> ~[zookeeper-3.6.2.jar:3.6.2]
> at
> org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:326)
> ~[zookeeper-3.6.2.jar:3.6.2]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> [netty-transport-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> [netty-common-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [netty-common-4.1.50.Final.jar:4.1.50.Final]
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> [netty-common-4.1.50.Final.jar:4.1.50.Final]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> {code}
>
>
> *Expectation:*
> This scenario should be handled and application should not throw such Null
> Pointer exception.
> We are kindly requesting Apache Zookeeper team to fix this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
