[
https://issues.apache.org/jira/browse/CASSANDRA-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17323814#comment-17323814
]
Ekaterina Dimitrova commented on CASSANDRA-16524:
-------------------------------------------------
Thanks [~gianluca] for all the work and bearing with me in Slack to confirm
details.
I just submitted my pass of review with just small comments. I have only one
bigger question.
Do we really need to use the new capacity function for max?
I want to align this with 3.11 and have the least hurdle. Does 3.11 always use
the Netty
[DEFAULT_MAX_CAPACITY|https://github.com/netty/netty/blob/netty-4.1.58.Final/buffer/src/main/java/io/netty/buffer/AbstractByteBufAllocator.java#L31]?
Also, [~aholmber], as you were working on the upgrade tests, do you think there
is another one we can add to cover this case and show the smooth upgrade after
this patch or it is overkill?
> Upgrading SSL enabled Cassandra cluster from 3.11.10 to 4.0-beta4 failing
> with javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16524
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16524
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
> Reporter: Alaykumar Barochia
> Assignee: Gianluca Righetto
> Priority: Normal
> Fix For: 4.0, 4.0-beta
>
> Attachments: system.log.ssl-error.txt
>
>
> Hi,
> We have SSL enabled cluster running on Apache Cassandra 3.11.10 and we are
> trying to upgrade it to 4.0-beta4 as a part of testing.
> Cluster size is 3x3 and deployed on Azure IaaS.
> {noformat}
> [cassandra@cass-521828978-1-1189299202 ~]$ nodetool status
> Datacenter: southcentral
> ========================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.12.74.31 85.61 KiB 16 32.2%
> 6db7a7ef-3490-4823-9ff3-c60a32165124 2
> UN 10.12.74.42 263.27 KiB 16 27.6%
> 7ad99ecf-7c7d-4780-872b-7c68b6b19849 1
> UN 10.12.74.34 85.61 KiB 16 37.8%
> 41ce16b7-2ab2-44ea-a810-8391f7f3caf2 0
> Datacenter: westus
> ==================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.12.90.11 90.63 KiB 16 38.9%
> 8d4cdb65-ff66-4bcd-8d4b-a4a0e893a728 2
> UN 10.12.90.6 85.61 KiB 16 34.5%
> 4f8007e9-fa3e-4e99-a9f9-f99997bf9625 1
> UN 10.12.89.80 94.1 KiB 16 28.9%
> 11f86cb0-c86b-440e-848f-b160118f43d5 0
> {noformat}
> We placed a new 4.0-beta4 binary on the first seed node (10.12.74.310) and
> starting Cassandra.
> It started throwing the below error:
> {noformat}
> ERROR [Messaging-EventLoop-3-11] 2021-03-15 22:10:05,188
> InboundConnectionInitiator.java:342 - Failed to properly handshake with peer
> /10.12.74.42:52356. Closing the channel.
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException:
> java.lang.IndexOutOfBoundsException: writerIndex(8560) +
> minWritableBytes(1977) exceeds maxCapacity(10240):
> BufferPoolAllocator$Wrapped(ridx: 0, widx: 8560, cap: 10240/10240)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException:
> writerIndex(8560) + minWritableBytes(1977) exceeds maxCapacity(10240):
> BufferPoolAllocator$Wrapped(ridx: 0, widx: 8560, cap: 10240/10240)
> at
> io.netty.handler.ssl.OpenSslKeyMaterialManager.setKeyMaterial(OpenSslKeyMaterialManager.java:115)
> at
> io.netty.handler.ssl.OpenSslKeyMaterialManager.setKeyMaterialServerSide(OpenSslKeyMaterialManager.java:84)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslServerContext$OpenSslServerCertificateCallback.handle(ReferenceCountedOpenSslServerContext.java:229)
> at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:596)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1203)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1325)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1368)
> at
> io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1387)
> at
> io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1294)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1331)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
> ... 15 common frames omitted
> {noformat}
> I have also used the below parameter under {{server_encryption_options}} as
> suggested at :
> [https://cassandra.apache.org/doc/latest/configuration/cass_yaml_file.html#server-encryption-options]
> but still getting the same error.
> {noformat}
> enable_legacy_ssl_storage_port: true
> {noformat}
>
> I am attaching the system.log file here for your review.
> It is working fine with Cassandra 3.11.10 and it looks like some bug in
> 4.0-beta4.
> Let me know if you need any more details.
> Thanks,
> Alaykumar Barochia
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
