[
https://issues.apache.org/jira/browse/CASSANDRA-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17311686#comment-17311686
]
Gianluca Righetto commented on CASSANDRA-16524:
-----------------------------------------------
I can programmatically reproduce the exception reported in this ticket with the
following piece of code:
{code}
public static void main(String[] args) throws
CertificateEncodingException
{
X509Certificate[] chain = new X509Certificate[5];
Random rand = new Random();
DatabaseDescriptor.clientInitialization();
for (int i = 0; i < chain.length; i++) {
byte[] bytes = new byte[i % 2 == 0 ? 1024 : 4096];
rand.nextBytes(bytes);
OpenSslX509Certificate certificate = new
OpenSslX509Certificate(bytes);
chain[i] = certificate;
}
PemX509Certificate.toPEM(GlobalBufferPoolAllocator.instance,
true, chain);
}
{code}
Note: this test code has to live in io.netty.hadler.ssl because `toPEM` is
package-protected.
What it does is creating a chain of certificates (just random bytes for testing
purposes) with different sizes (1024 and 4096 bits).
The issue is that netty allocates a buffer that's proportional in size to the
size of the first certificate in the chain, so if the certificates vary in
length and the first one is shorter than the rest, the buffer size estimate
will be off:
https://github.com/netty/netty/blob/netty-4.1.58.Final/handler/src/main/java/io/netty/handler/ssl/PemX509Certificate.java#L135
In general, that shouldn't be a problem because netty's default allocator,
ByteBufAllocator.DEFAULT, will increase the max buffer capacity. But in C* 4.0,
we can see from the attached stacktrace that it's using BufferPoolAllocator,
which does not increase:
https://github.com/apache/cassandra/blob/cassandra-4.0-beta4/src/java/org/apache/cassandra/net/BufferPoolAllocator.java#L59
In the sample code above, if I switch from
{{GlobalBufferPoolAllocator.instance}} to {{ByteBufAllocator.DEFAULT}}, it
throws no exception.
In C* 3.11.10 as far as I can tell it's using netty's default implementation.
> Upgrading SSL enabled Cassandra cluster from 3.11.10 to 4.0-beta4 failing
> with javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16524
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16524
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
> Reporter: Alaykumar Barochia
> Assignee: Gianluca Righetto
> Priority: Normal
> Fix For: 4.0, 4.0-rc
>
> Attachments: system.log.ssl-error.txt
>
>
> Hi,
> We have SSL enabled cluster running on Apache Cassandra 3.11.10 and we are
> trying to upgrade it to 4.0-beta4 as a part of testing.
> Cluster size is 3x3 and deployed on Azure IaaS.
> {noformat}
> [cassandra@cass-521828978-1-1189299202 ~]$ nodetool status
> Datacenter: southcentral
> ========================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.12.74.31 85.61 KiB 16 32.2%
> 6db7a7ef-3490-4823-9ff3-c60a32165124 2
> UN 10.12.74.42 263.27 KiB 16 27.6%
> 7ad99ecf-7c7d-4780-872b-7c68b6b19849 1
> UN 10.12.74.34 85.61 KiB 16 37.8%
> 41ce16b7-2ab2-44ea-a810-8391f7f3caf2 0
> Datacenter: westus
> ==================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.12.90.11 90.63 KiB 16 38.9%
> 8d4cdb65-ff66-4bcd-8d4b-a4a0e893a728 2
> UN 10.12.90.6 85.61 KiB 16 34.5%
> 4f8007e9-fa3e-4e99-a9f9-f99997bf9625 1
> UN 10.12.89.80 94.1 KiB 16 28.9%
> 11f86cb0-c86b-440e-848f-b160118f43d5 0
> {noformat}
> We placed a new 4.0-beta4 binary on the first seed node (10.12.74.310) and
> starting Cassandra.
> It started throwing the below error:
> {noformat}
> ERROR [Messaging-EventLoop-3-11] 2021-03-15 22:10:05,188
> InboundConnectionInitiator.java:342 - Failed to properly handshake with peer
> /10.12.74.42:52356. Closing the channel.
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException:
> java.lang.IndexOutOfBoundsException: writerIndex(8560) +
> minWritableBytes(1977) exceeds maxCapacity(10240):
> BufferPoolAllocator$Wrapped(ridx: 0, widx: 8560, cap: 10240/10240)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException:
> writerIndex(8560) + minWritableBytes(1977) exceeds maxCapacity(10240):
> BufferPoolAllocator$Wrapped(ridx: 0, widx: 8560, cap: 10240/10240)
> at
> io.netty.handler.ssl.OpenSslKeyMaterialManager.setKeyMaterial(OpenSslKeyMaterialManager.java:115)
> at
> io.netty.handler.ssl.OpenSslKeyMaterialManager.setKeyMaterialServerSide(OpenSslKeyMaterialManager.java:84)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslServerContext$OpenSslServerCertificateCallback.handle(ReferenceCountedOpenSslServerContext.java:229)
> at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:596)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1203)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1325)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1368)
> at
> io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1387)
> at
> io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1294)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1331)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
> ... 15 common frames omitted
> {noformat}
> I have also used the below parameter under {{server_encryption_options}} as
> suggested at :
> [https://cassandra.apache.org/doc/latest/configuration/cass_yaml_file.html#server-encryption-options]
> but still getting the same error.
> {noformat}
> enable_legacy_ssl_storage_port: true
> {noformat}
>
> I am attaching the system.log file here for your review.
> It is working fine with Cassandra 3.11.10 and it looks like some bug in
> 4.0-beta4.
> Let me know if you need any more details.
> Thanks,
> Alaykumar Barochia
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
