[
https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17625366#comment-17625366
]
Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 10/28/22 1:37 AM:
---------------------------------------------------------------------------
{quote}Getting the full stack trace will help inform next steps.
{quote}
I think I have some preliminary good news. I ran some rough tests today to get
to it. I suspect things might have improved on Netty side recently.
So with the Netty version that was current in March, [current trunk and
JDK17|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=trunk-jdk17-oct]
I can see 55 failing tests and a bunch of SSL tests.
Now looking into the cqlsh tests
[here|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2005/workflows/18b54727-8b82-42cc-9dd2-5a04bfb06e5a/jobs/16218/tests#failed-test-1]
I can also find the issue we discussed, test_tls:
{code:java}
test teardown failure Unexpected error found in node logs (see stdout for full
details). Errors: [[node1] "WARN [nioEventLoopGroup-5-5] 2022-10-27
22:45:13,917 ExceptionHandlers.java:140 - Unknown exception in client
networking\nio.netty.handler.codec.DecoderException:
javax.net.ssl.SSLException: Fail to unwrap network record\n\tat
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833)\nCaused by:
javax.net.ssl.SSLException: Fail to unwrap network record\n\tat
java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)\n\tat
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)\n\tat
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:522)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)\n\tat
java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)\n\tat
io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295)\n\tat
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1341)\n\tat
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)\n\tat
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)\n\t...
17 common frames omitted\nCaused by: java.lang.ClassCastException: class
org.apache.cassandra.utils.memory.BufferPool$Chunk cannot be cast to class
sun.nio.ch.DirectBuffer (org.apache.cassandra.utils.memory.BufferPool$Chunk is
in unnamed module of loader 'app'; sun.nio.ch.DirectBuffer is in module
java.base of loader 'bootstrap')\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode$GCMEngine.overlapDetection(GaloisCounterMode.java:865)\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doFinal(GaloisCounterMode.java:1502)\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode.engineDoFinal(GaloisCounterMode.java:447)\n\tat
java.base/javax.crypto.Cipher.doFinal(Cipher.java:2500)\n\tat
java.base/sun.security.ssl.SSLCipher$T12GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1659)\n\tat
java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239)\n\tat
java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196)\n\tat
java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159)\n\tat
java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)\n\t...
25 common frames omitted"]{code}
More info in the logs under artifacts.
------
I also pushed update to newest, current netty version:
[jdk8|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2006/workflows/23d0921a-e1bb-4efb-93ef-754d357836aa],
[jdk8 unit tests as they failed before due to some CircleCI env
issue|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2006/workflows/5f24e702-d537-40c2-bfeb-5ca88d6c84f0],
[jdk11|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2006/workflows/d676b640-cb60-4cad-8e62-9ed6017cc485]
- We hit an issue with test_schema_changes:
{code:java}
test teardown failure Unexpected error found in node logs (see stdout for full
details). Errors: [[node1] 'WARN [nioEventLoopGroup-5-3] 2022-10-28
00:57:21,672 ExceptionHandlers.java:139 - Unknown exception in client
networking\njava.io.IOException: Connection reset by peer\n\tat
sun.nio.ch.FileDispatcherImpl.read0(Native Method)\n\tat
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)\n\tat
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)\n\tat
sun.nio.ch.IOUtil.read(IOUtil.java:192)\n\tat
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.lang.Thread.run(Thread.java:748)']{code}
And now I do not see anymore with
[JDK17|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=trunk-jdk17-oct]
the test_tls failing, plus those SSL failures in the DTests. The errors were
down to about 30 and I think I can attribute a lot of them to the rebase I did
today as the pom files were recently pulled from build.xml, also wrong CCM and
DTest branches in my CI config. I suspect some updates got lost. I need to
clean my branch, probably tomorrow, to see what is left from CI perspective.
I actually just saw
[here|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2002/workflows/2bbf125a-844d-463e-bbf6-a20be78e13a4/jobs/16199/tests],
test_insert_data_during_replace_different_address failing with:
{code:java}
test teardown failure Unexpected error found in node logs (see stdout for full
details). Errors: [[node2] 'ERROR [Messaging-EventLoop-3-4] 2022-10-27
17:12:25,665 OutboundConnection.java:1061 -
/127.0.0.2:7000->/127.0.0.1:7000-URGENT_MESSAGES-c650c9c5 channel in
potentially inconsistent state after error; closing\njava.net.SocketException:
Connection reset\n\tat
java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)\n\tat
java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833)']{code}
It seems to me, remembering [this
commit|https://github.com/apache/cassandra-dtest/commit/7bc8467233af69391efeaf74ef87a73c0ea9ceb1],
that we might just need to add some additional filtering probably?
----
Oh we now hit
{code:java}
com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred
server side on /127.0.0.1:9042: java.lang.UnsupportedOperationException: can't
get field offset on a hidden class: private final
org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1{code}
but this time not from jamm. Look at the in-jvm tests, for example -
MixedModeFuzzTest (this is failing independently of the netty version):
{code:java}
com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred
server side on /127.0.0.1:9042: java.lang.UnsupportedOperationException: can't
get field offset on a hidden class: private final
org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:62)
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:26) at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:86)
at
org.apache.cassandra.distributed.test.MixedModeFuzzTest.lambda$mixedModeFuzzTest$4(MixedModeFuzzTest.java:285)
at java.base/java.lang.Thread.run(Thread.java:833) Suppressed:
com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred
server side on /127.0.0.2:9042: java.lang.UnsupportedOperationException: can't
get field offset on a hidden class: private final
org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$1443/0x00000008014c4010.arg$1
Caused by: com.datastax.driver.core.exceptions.ServerError: An unexpected
error occurred server side on /127.0.0.2:9042:
java.lang.UnsupportedOperationException: can't get field offset on a hidden
class: private final org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$1443/0x00000008014c4010.arg$1
at com.datastax.driver.core.Responses$Error.asException(Responses.java:152) at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:651)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1290)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1208)
at
com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
at
com.datastax.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:546)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at
com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Suppressed: com.datastax.driver.core.exceptions.ServerError: An unexpected
error occurred server side on /127.0.0.1:9042:
java.lang.UnsupportedOperationException: can't get field offset on a hidden
class: private final org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:62)
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:26) at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:86)
at
org.apache.cassandra.distributed.test.MixedModeFuzzTest.lambda$mixedModeFuzzTest$4(MixedModeFuzzTest.java:304)
Caused by: com.datastax.driver.core.exceptions.ServerError: An unexpected
error occurred server side on /127.0.0.1:9042:
java.lang.UnsupportedOperationException: can't get field offset on a hidden
class: private final org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1
at com.datastax.driver.core.Responses$Error.asException(Responses.java:152) at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:651)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1290)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1208)
at
com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
at
com.datastax.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:546)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at
com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Caused by: com.datastax.driver.core.exceptions.ServerError: An unexpected
error occurred server side on /127.0.0.1:9042:
java.lang.UnsupportedOperationException: can't get field offset on a hidden
class: private final org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1
at com.datastax.driver.core.Responses$Error.asException(Responses.java:152) at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:651)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1290)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1208)
at
com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
at
com.datastax.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:546)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at
com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30){code}
was (Author: e.dimitrova):
{quote}Getting the full stack trace will help inform next steps.
{quote}
I think I have some preliminary good news. I ran some rough tests today to get
to it. I suspect things might have improved on Netty side recently.
So with the version that was current in March, [current trunk and
JDK17|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=trunk-jdk17-oct]
I can see 55 failing tests and a bunch of SSL tests.
Now looking into the cqlsh tests
[here|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2005/workflows/18b54727-8b82-42cc-9dd2-5a04bfb06e5a/jobs/16218/tests#failed-test-1]
I can also find the issue we discussed, test_tls:
{code:java}
test teardown failure Unexpected error found in node logs (see stdout for full
details). Errors: [[node1] "WARN [nioEventLoopGroup-5-5] 2022-10-27
22:45:13,917 ExceptionHandlers.java:140 - Unknown exception in client
networking\nio.netty.handler.codec.DecoderException:
javax.net.ssl.SSLException: Fail to unwrap network record\n\tat
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833)\nCaused by:
javax.net.ssl.SSLException: Fail to unwrap network record\n\tat
java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)\n\tat
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)\n\tat
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:522)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)\n\tat
java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)\n\tat
io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295)\n\tat
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1341)\n\tat
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)\n\tat
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)\n\t...
17 common frames omitted\nCaused by: java.lang.ClassCastException: class
org.apache.cassandra.utils.memory.BufferPool$Chunk cannot be cast to class
sun.nio.ch.DirectBuffer (org.apache.cassandra.utils.memory.BufferPool$Chunk is
in unnamed module of loader 'app'; sun.nio.ch.DirectBuffer is in module
java.base of loader 'bootstrap')\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode$GCMEngine.overlapDetection(GaloisCounterMode.java:865)\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doFinal(GaloisCounterMode.java:1502)\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode.engineDoFinal(GaloisCounterMode.java:447)\n\tat
java.base/javax.crypto.Cipher.doFinal(Cipher.java:2500)\n\tat
java.base/sun.security.ssl.SSLCipher$T12GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1659)\n\tat
java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239)\n\tat
java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196)\n\tat
java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159)\n\tat
java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)\n\t...
25 common frames omitted"]{code}
More info in the logs under artifacts.
------
I also pushed update to newest, current netty version:
[jdk8|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2006/workflows/23d0921a-e1bb-4efb-93ef-754d357836aa],
[jdk8 unit tests as they failed before due to some CircleCI env
issue|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2006/workflows/5f24e702-d537-40c2-bfeb-5ca88d6c84f0],
[jdk11|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2006/workflows/d676b640-cb60-4cad-8e62-9ed6017cc485]
- We hit an issue with test_schema_changes:
{code:java}
test teardown failure Unexpected error found in node logs (see stdout for full
details). Errors: [[node1] 'WARN [nioEventLoopGroup-5-3] 2022-10-28
00:57:21,672 ExceptionHandlers.java:139 - Unknown exception in client
networking\njava.io.IOException: Connection reset by peer\n\tat
sun.nio.ch.FileDispatcherImpl.read0(Native Method)\n\tat
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)\n\tat
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)\n\tat
sun.nio.ch.IOUtil.read(IOUtil.java:192)\n\tat
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.lang.Thread.run(Thread.java:748)']{code}
And now I do not see anymore with
[JDK17|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=trunk-jdk17-oct]
the test_tls failing, plus those SSL failures in the DTests. The errors were
down to about 30 and I think I can attribute a lot of them to the rebase I did
today as the pom files were recently pulled from build.xml, also wrong CCM and
DTest branches in my CI config. I suspect some updates got lost. I need to
clean my branch, probably tomorrow, to see what is left from CI perspective.
I actually just saw
[here|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2002/workflows/2bbf125a-844d-463e-bbf6-a20be78e13a4/jobs/16199/tests],
test_insert_data_during_replace_different_address failing with:
{code:java}
test teardown failure Unexpected error found in node logs (see stdout for full
details). Errors: [[node2] 'ERROR [Messaging-EventLoop-3-4] 2022-10-27
17:12:25,665 OutboundConnection.java:1061 -
/127.0.0.2:7000->/127.0.0.1:7000-URGENT_MESSAGES-c650c9c5 channel in
potentially inconsistent state after error; closing\njava.net.SocketException:
Connection reset\n\tat
java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)\n\tat
java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833)']{code}
It seems to me, remembering [this
commit|https://github.com/apache/cassandra-dtest/commit/7bc8467233af69391efeaf74ef87a73c0ea9ceb1],
that we might just need to add some additional filtering probably?
----
Oh we now hit
{code:java}
com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred
server side on /127.0.0.1:9042: java.lang.UnsupportedOperationException: can't
get field offset on a hidden class: private final
org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1{code}
but this time not from jamm. Look at the in-jvm tests, for example -
MixedModeFuzzTest (this is failing independently of the netty version):
{code:java}
com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred
server side on /127.0.0.1:9042: java.lang.UnsupportedOperationException: can't
get field offset on a hidden class: private final
org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:62)
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:26) at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:86)
at
org.apache.cassandra.distributed.test.MixedModeFuzzTest.lambda$mixedModeFuzzTest$4(MixedModeFuzzTest.java:285)
at java.base/java.lang.Thread.run(Thread.java:833) Suppressed:
com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred
server side on /127.0.0.2:9042: java.lang.UnsupportedOperationException: can't
get field offset on a hidden class: private final
org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$1443/0x00000008014c4010.arg$1
Caused by: com.datastax.driver.core.exceptions.ServerError: An unexpected
error occurred server side on /127.0.0.2:9042:
java.lang.UnsupportedOperationException: can't get field offset on a hidden
class: private final org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$1443/0x00000008014c4010.arg$1
at com.datastax.driver.core.Responses$Error.asException(Responses.java:152) at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:651)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1290)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1208)
at
com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
at
com.datastax.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:546)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at
com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Suppressed: com.datastax.driver.core.exceptions.ServerError: An unexpected
error occurred server side on /127.0.0.1:9042:
java.lang.UnsupportedOperationException: can't get field offset on a hidden
class: private final org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:62)
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:26) at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:86)
at
org.apache.cassandra.distributed.test.MixedModeFuzzTest.lambda$mixedModeFuzzTest$4(MixedModeFuzzTest.java:304)
Caused by: com.datastax.driver.core.exceptions.ServerError: An unexpected
error occurred server side on /127.0.0.1:9042:
java.lang.UnsupportedOperationException: can't get field offset on a hidden
class: private final org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1
at com.datastax.driver.core.Responses$Error.asException(Responses.java:152) at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:651)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1290)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1208)
at
com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
at
com.datastax.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:546)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at
com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Caused by: com.datastax.driver.core.exceptions.ServerError: An unexpected
error occurred server side on /127.0.0.1:9042:
java.lang.UnsupportedOperationException: can't get field offset on a hidden
class: private final org.apache.cassandra.db.ClusteringComparator
org.apache.cassandra.db.ClusteringComparator$$Lambda$513/0x00000008011acc20.arg$1
at com.datastax.driver.core.Responses$Error.asException(Responses.java:152) at
com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:651)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1290)
at
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1208)
at
com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
com.datastax.shaded.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
com.datastax.shaded.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
at
com.datastax.shaded.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:546)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at
com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30){code}
> Upgrade Netty on 4.x(current trunk)
> -----------------------------------
>
> Key: CASSANDRA-17992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17992
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
> Reporter: Ekaterina Dimitrova
> Priority: Low
> Fix For: 4.x
>
>
> I haven't been able to identify from the Netty docs which was the lowest
> version where JDK17 was added but we are about 40 versions behind in netty 4
> so I suspect we better update.
> We need to consider there was an issue with class cast exceptions when
> building with JDK17 with newer versions of netty (the newest available in
> March 2022). For the record, we didn't see those when running CI on JDK8 and
> JDK11. We also need to carefully revise the changes between the netty
> versions.
> Upgrading will cover also a fix in netty that was discussed in
> [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF
> Slack thread.
> CC [~benedict] , [~aleksey]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
