[
https://issues.apache.org/jira/browse/STORM-3751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17528839#comment-17528839
]
Binh Le edited comment on STORM-3751 at 4/27/22 3:31 PM:
---------------------------------------------------------
We are also running into this same error, with the same stack trace and same
line number. This is causing workers to get restarted. Has anyone been able to
figure out what's going on yet?
I see that STORM-3141 is fixed. However, that has a different stack trace;
error occurred on a different line number. So I believe that original issue was
fixed, but now leads to this new error. So this may be a different issue?
was (Author: JIRAUSER288735):
We are also running into this same error, with the same stack trace and same
line number. Has anyone been able to figure out what's going on yet?
I see that STORM-3141 is fixed. However, that has a different stack trace;
error occurred on a different line number. So I believe that original issue was
fixed, but now leads to this new error. So this may be a different issue?
> NPE in WorkerState.transferLocalBatch
> -------------------------------------
>
> Key: STORM-3751
> URL: https://issues.apache.org/jira/browse/STORM-3751
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-client
> Affects Versions: 2.2.0
> Reporter: Arwin S Tio
> Priority: Major
>
> Hello,
>
> I've recently upgraded to Storm 2.2.0 and have been getting this error:
>
> {code:java}
> 2021-03-07 04:36:51.061 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the
> request
> java.lang.NullPointerException: null
> at
> org.apache.storm.daemon.worker.WorkerState.transferLocalBatch(WorkerState.java:543)
> ~[storm-client-2.2.0.jar:2.2.0]
> at
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:71)
> ~[storm-client-2.2.0.jar:2.2.0]
> at org.apache.storm.messaging.netty.Server.enqueue(Server.java:146)
> ~[storm-client-2.2.0.jar:2.2.0]
> at org.apache.storm.messaging.netty.Server.received(Server.java:264)
> ~[storm-client-2.2.0.jar:2.2.0]
> at
> org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.java:51)
> ~[storm-client-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at
> org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> [storm-shaded-deps-2.2.0.jar:2.2.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]
> 2021-03-07 04:36:51.061 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6700-worker-1 [INFO] Received error in netty thread..
> terminating server... {code}
>
> This issue happens every 20-30 minutes and causes the workers to die/restart.
> It seems related to https://issues.apache.org/jira/browse/STORM-3141 but
> seems to have been fixed in 2.0.
> I am happy to provide more information but at the moment am unsure of what is
> relevant.
> I have a suspicion that this is related to load-aware localOrShuffleGrouping
> ("LoadAwareShuffleGrouping") because this issue seems to have started when I
> switched the Grouping, but again, not sure if it's actually related.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
