[
https://issues.apache.org/jira/browse/CASSANDRA-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326149#comment-17326149
]
Ekaterina Dimitrova edited comment on CASSANDRA-16524 at 4/20/21, 11:08 PM:
----------------------------------------------------------------------------
Jenkins finished with three failures which seem to me unrelated to the patch:
[https://jenkins-cm4.apache.org/job/Cassandra-devbranch/690/testReport/]
Two of the tests seems flaky and have failed with the same issue before:
[https://ci-cassandra.apache.org/job/Cassandra-trunk/446/testReport/junit/org.apache.cassandra.net/AsyncPromiseTest/testFailure_cdc/]
[https://ci-cassandra.apache.org/job/Cassandra-trunk/447/testReport/junit/org.apache.cass[…]ctionStrategyCQLTest/stressTestCompactionStrategyManager/|https://ci-cassandra.apache.org/job/Cassandra-trunk/447/testReport/junit/org.apache.cassandra.db.compaction/LongLeveledCompactionStrategyCQLTest/stressTestCompactionStrategyManager/]
The third one looks to me as a CI env issue, WDYT?:
[https://jenkins-cm4.apache.org/job/Cassandra-devbranch/690/testReport/junit.framework/Te[…]sandra_distributed_test_PreviewRepairCoordinatorFastTest/|https://jenkins-cm4.apache.org/job/Cassandra-devbranch/690/testReport/junit.framework/TestSuite/org_apache_cassandra_distributed_test_PreviewRepairCoordinatorFastTest/]
{code:java}
ERROR 20:56:36 Repair e500b3d0-a21a-11eb-882f-cd9857e77aff failed:
java.lang.IllegalArgumentException: Unknown host specified
thisreally.should.not.exist.apache.org
at
org.apache.cassandra.service.ActiveRepairService.getNeighbors(ActiveRepairService.java:478)
at
org.apache.cassandra.repair.RepairRunnable.getNeighborsAndRanges(RepairRunnable.java:326)
at
org.apache.cassandra.repair.RepairRunnable.runMayThrow(RepairRunnable.java:265)
at org.apache.cassandra.repair.RepairRunnable.run(RepairRunnable.java:241)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException:
thisreally.should.not.exist.apache.org: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at java.net.InetAddress.getByName(InetAddress.java:1077)
at
org.apache.cassandra.locator.InetAddressAndPort.getByNameOverrideDefaults(InetAddressAndPort.java:228)
at
org.apache.cassandra.locator.InetAddressAndPort.getByName(InetAddressAndPort.java:213)
at
org.apache.cassandra.service.ActiveRepairService.getNeighbors(ActiveRepairService.java:472)
... 11 common frames omitted{code}
was (Author: e.dimitrova):
Jenkins finished with three failures which seem unrelated to me:
[https://jenkins-cm4.apache.org/job/Cassandra-devbranch/690/testReport/]
Two of the tests seems flaky and have failed with the same issue before:
[https://ci-cassandra.apache.org/job/Cassandra-trunk/446/testReport/junit/org.apache.cassandra.net/AsyncPromiseTest/testFailure_cdc/]
[https://ci-cassandra.apache.org/job/Cassandra-trunk/447/testReport/junit/org.apache.cass[…]ctionStrategyCQLTest/stressTestCompactionStrategyManager/|https://ci-cassandra.apache.org/job/Cassandra-trunk/447/testReport/junit/org.apache.cassandra.db.compaction/LongLeveledCompactionStrategyCQLTest/stressTestCompactionStrategyManager/]
The third one looks to me as a CI env issue, WDYT?:
[https://jenkins-cm4.apache.org/job/Cassandra-devbranch/690/testReport/junit.framework/Te[…]sandra_distributed_test_PreviewRepairCoordinatorFastTest/|https://jenkins-cm4.apache.org/job/Cassandra-devbranch/690/testReport/junit.framework/TestSuite/org_apache_cassandra_distributed_test_PreviewRepairCoordinatorFastTest/]
{code:java}
ERROR 20:56:36 Repair e500b3d0-a21a-11eb-882f-cd9857e77aff failed:
java.lang.IllegalArgumentException: Unknown host specified
thisreally.should.not.exist.apache.org
at
org.apache.cassandra.service.ActiveRepairService.getNeighbors(ActiveRepairService.java:478)
at
org.apache.cassandra.repair.RepairRunnable.getNeighborsAndRanges(RepairRunnable.java:326)
at
org.apache.cassandra.repair.RepairRunnable.runMayThrow(RepairRunnable.java:265)
at org.apache.cassandra.repair.RepairRunnable.run(RepairRunnable.java:241)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException:
thisreally.should.not.exist.apache.org: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at java.net.InetAddress.getByName(InetAddress.java:1077)
at
org.apache.cassandra.locator.InetAddressAndPort.getByNameOverrideDefaults(InetAddressAndPort.java:228)
at
org.apache.cassandra.locator.InetAddressAndPort.getByName(InetAddressAndPort.java:213)
at
org.apache.cassandra.service.ActiveRepairService.getNeighbors(ActiveRepairService.java:472)
... 11 common frames omitted{code}
> Upgrading SSL enabled Cassandra cluster from 3.11.10 to 4.0-beta4 failing
> with javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-16524
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16524
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Encryption
> Reporter: Alaykumar Barochia
> Assignee: Gianluca Righetto
> Priority: Normal
> Fix For: 4.0, 4.0-beta
>
> Attachments: system.log.ssl-error.txt
>
>
> Hi,
> We have SSL enabled cluster running on Apache Cassandra 3.11.10 and we are
> trying to upgrade it to 4.0-beta4 as a part of testing.
> Cluster size is 3x3 and deployed on Azure IaaS.
> {noformat}
> [cassandra@cass-521828978-1-1189299202 ~]$ nodetool status
> Datacenter: southcentral
> ========================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.12.74.31 85.61 KiB 16 32.2%
> 6db7a7ef-3490-4823-9ff3-c60a32165124 2
> UN 10.12.74.42 263.27 KiB 16 27.6%
> 7ad99ecf-7c7d-4780-872b-7c68b6b19849 1
> UN 10.12.74.34 85.61 KiB 16 37.8%
> 41ce16b7-2ab2-44ea-a810-8391f7f3caf2 0
> Datacenter: westus
> ==================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.12.90.11 90.63 KiB 16 38.9%
> 8d4cdb65-ff66-4bcd-8d4b-a4a0e893a728 2
> UN 10.12.90.6 85.61 KiB 16 34.5%
> 4f8007e9-fa3e-4e99-a9f9-f99997bf9625 1
> UN 10.12.89.80 94.1 KiB 16 28.9%
> 11f86cb0-c86b-440e-848f-b160118f43d5 0
> {noformat}
> We placed a new 4.0-beta4 binary on the first seed node (10.12.74.310) and
> starting Cassandra.
> It started throwing the below error:
> {noformat}
> ERROR [Messaging-EventLoop-3-11] 2021-03-15 22:10:05,188
> InboundConnectionInitiator.java:342 - Failed to properly handshake with peer
> /10.12.74.42:52356. Closing the channel.
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException:
> java.lang.IndexOutOfBoundsException: writerIndex(8560) +
> minWritableBytes(1977) exceeds maxCapacity(10240):
> BufferPoolAllocator$Wrapped(ridx: 0, widx: 8560, cap: 10240/10240)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException:
> writerIndex(8560) + minWritableBytes(1977) exceeds maxCapacity(10240):
> BufferPoolAllocator$Wrapped(ridx: 0, widx: 8560, cap: 10240/10240)
> at
> io.netty.handler.ssl.OpenSslKeyMaterialManager.setKeyMaterial(OpenSslKeyMaterialManager.java:115)
> at
> io.netty.handler.ssl.OpenSslKeyMaterialManager.setKeyMaterialServerSide(OpenSslKeyMaterialManager.java:84)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslServerContext$OpenSslServerCertificateCallback.handle(ReferenceCountedOpenSslServerContext.java:229)
> at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:596)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1203)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1325)
> at
> io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1368)
> at
> io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1387)
> at
> io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1294)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1331)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
> ... 15 common frames omitted
> {noformat}
> I have also used the below parameter under {{server_encryption_options}} as
> suggested at :
> [https://cassandra.apache.org/doc/latest/configuration/cass_yaml_file.html#server-encryption-options]
> but still getting the same error.
> {noformat}
> enable_legacy_ssl_storage_port: true
> {noformat}
>
> I am attaching the system.log file here for your review.
> It is working fine with Cassandra 3.11.10 and it looks like some bug in
> 4.0-beta4.
> Let me know if you need any more details.
> Thanks,
> Alaykumar Barochia
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
