[
https://issues.apache.org/jira/browse/CASSANDRA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114130#comment-16114130
]
Stefan Podkowinski commented on CASSANDRA-13649:
------------------------------------------------
Looking at this issue again got me a bit suspicious on how our Initializer is
[handled by
ServerBootstrap|https://github.com/netty/netty/blob/3cc405296310643bccddc8c81998c97f25b3201c/transport/src/main/java/io/netty/bootstrap/ServerBootstrap.java#L169].
It seems that instances added by {{ServerBootstrap.childHandler()}} will not
be added immediately to the pipeline (as opposed to
{{ServerBootstrap.handler()}}, see code comment for further explanation. This
behaviour was changed in netty
[4638df2062|https://github.com/netty/netty/commit/4638df20628a8987c8709f0f8e5f3679a914ce1a]
([5566|https://github.com/netty/netty/issues/5566]) and included by upgrading
netty in CASSANDRA-13114. Did anyone already noticed this issue before February?
I'm not really sure there's a lot we can do on our side. Maybe add our
Initilizer using {{handler()}} instead of {{childHandler()}}? But still, if
there's really a race then {{ServerBootstrapAcceptor}} won't get added and the
pipeline will not work anyways. [~norman], do you have any suggestions?
> Uncaught exceptions in Netty pipeline
> -------------------------------------
>
> Key: CASSANDRA-13649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13649
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging, Testing
> Reporter: Stefan Podkowinski
> Attachments: test_stdout.txt
>
>
> I've noticed some netty related errors in trunk in [some of the dtest
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/106/#showFailuresLink].
> Just want to make sure that we don't have to change anything related to the
> exception handling in our pipeline and that this isn't a netty issue.
> Actually if this causes flakiness but is otherwise harmless, we should do
> something about it, even if it's just on the dtest side.
> {noformat}
> WARN [epollEventLoopGroup-2-9] 2017-06-28 17:23:49,699 Slf4JLogger.java:151
> - An exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
> Connection reset by peer
> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> And again in another test:
> {noformat}
> WARN [epollEventLoopGroup-2-8] 2017-06-29 02:27:31,300 Slf4JLogger.java:151
> - An exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
> Connection reset by peer
> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> This one looks also odd and makes
> upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test fail:
> {noformat}
> WARN [epollEventLoopGroup-2-9] 2017-06-29 02:41:37,125 Slf4JLogger.java:151
> - An exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.
> io.netty.handler.codec.DecoderException:
> org.apache.cassandra.transport.ProtocolException: Invalid or unsupported
> protocol version: 4
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:375)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:342)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:220)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1289)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:893)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:691)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Caused by: org.apache.cassandra.transport.ProtocolException: Invalid or
> unsupported protocol version: 4
> at org.apache.cassandra.transport.Frame$Decoder.decode(Frame.java:186)
> ~[main/:na]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
> ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> ... 16 common frames omitted
> {noformat}
> /cc [~jasobrown], [~norman]
> Edit:
> The {{io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed}} error also causes tests to fail for 3.0 and 3.11.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
