[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651738#comment-17651738
]
Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 12/23/22 11:14 PM:
I am not happy with the below error that pops up in almost every CI run with
random tests with the new Netty version. Not convinced we should silence it
similar to other Connection reset failures already silenced in DTests
[here|https://github.com/apache/cassandra-dtest/blob/trunk/dtest_setup.py#L50].
(CC [~dcapwell] and [~jonmeredith] as they looked into silencing those before
in CASSANDRA-17594) I will look further into it in the new year.
{code:java}
ERROR [Messaging-EventLoop-3-4] 2022-10-27 17:12:25,665
OutboundConnection.java:1061 -
/127.0.0.2:7000->/127.0.0.1:7000-URGENT_MESSAGES-c650c9c5 channel in
potentially inconsistent state after error; closing\njava.net.SocketException:
Connection reset\n\tat
java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)\n\tat
java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833){code}
was (Author: e.dimitrova):
I am not happy with the below error that pops up in almost every CI run with
random tests. Not convinced we should silence it similar to other Connection
reset failures already silenced in DTests
[here|https://github.com/apache/cassandra-dtest/blob/trunk/dtest_setup.py#L50].
(CC [~dcapwell] and [~jonmeredith] as they looked into silencing those before
in CASSANDRA-17594) I will look further into it in the new year.
{code:java}
ERROR [Messaging-EventLoop-3-4] 2022-10-27 17:12:25,665
OutboundConnection.java:1061 -
/127.0.0.2:7000->/127.0.0.1:7000-URGENT_MESSAGES-c650c9c5 channel in
potentially inconsistent state after error; closing\njava.net.SocketException:
Connection reset\n\tat
java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)\n\tat
java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833){code}
> Upgrade Netty on 4.x(current trunk)
> ---
>
> Key: CASSANDRA-17992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17992
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Low
> Fix For: 4.x
>
>
> I haven't been able to identify from the Netty docs which was the lowest
> version where JDK17 was added but we are about 40 versions behind in netty 4
> so I suspect we better update.
> We need to consider there was an issue with class cast exceptions when
> building with JDK17 with newer versions of netty (the newest available in
> March 2022). For the record, we didn't see those when
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651738#comment-17651738
]
Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 12/23/22 11:13 PM:
I am not happy with the below error that pops up in almost every CI run with
random tests. Not convinced we should silence it similar to other Connection
reset failures already silenced in DTests
[here|https://github.com/apache/cassandra-dtest/blob/trunk/dtest_setup.py#L50].
(CC [~dcapwell] and [~jonmeredith] as they looked into silencing those before
in CASSANDRA-17594) I will look further into it in the new year.
{code:java}
ERROR [Messaging-EventLoop-3-4] 2022-10-27 17:12:25,665
OutboundConnection.java:1061 -
/127.0.0.2:7000->/127.0.0.1:7000-URGENT_MESSAGES-c650c9c5 channel in
potentially inconsistent state after error; closing\njava.net.SocketException:
Connection reset\n\tat
java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)\n\tat
java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833){code}
was (Author: e.dimitrova):
I am not happy with the below error that pops up in almost every CI run with
random tests. Not convinced we should silence it similar to other Connection
reset failures. I will look further into it in the new year.
{code:java}
ERROR [Messaging-EventLoop-3-4] 2022-10-27 17:12:25,665
OutboundConnection.java:1061 -
/127.0.0.2:7000->/127.0.0.1:7000-URGENT_MESSAGES-c650c9c5 channel in
potentially inconsistent state after error; closing\njava.net.SocketException:
Connection reset\n\tat
java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)\n\tat
java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833){code}
> Upgrade Netty on 4.x(current trunk)
> ---
>
> Key: CASSANDRA-17992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17992
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Low
> Fix For: 4.x
>
>
> I haven't been able to identify from the Netty docs which was the lowest
> version where JDK17 was added but we are about 40 versions behind in netty 4
> so I suspect we better update.
> We need to consider there was an issue with class cast exceptions when
> building with JDK17 with newer versions of netty (the newest available in
> March 2022). For the record, we didn't see those when running CI on JDK8 and
> JDK11. We also need to carefully revise the changes between the netty
> versions.
> Upgrading will cover also a fix in netty that was discussed in
>
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651738#comment-17651738
]
Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 12/23/22 11:09 PM:
I am not happy with the below error that pops up in almost every CI run with
random tests. Not convinced we should silence it similar to other Connection
reset failures. I will look further into it in the new year.
{code:java}
ERROR [Messaging-EventLoop-3-4] 2022-10-27 17:12:25,665
OutboundConnection.java:1061 -
/127.0.0.2:7000->/127.0.0.1:7000-URGENT_MESSAGES-c650c9c5 channel in
potentially inconsistent state after error; closing\njava.net.SocketException:
Connection reset\n\tat
java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)\n\tat
java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833){code}
was (Author: e.dimitrova):
I am not happy with the bellow error that pops up in almost every CI run with
random tests. Not convinced we should silence it similar to other Connection
reset failures. I will look further into it in the new year.
ERROR [Messaging-EventLoop-3-4] 2022-10-27 17:12:25,665
OutboundConnection.java:1061 -
/127.0.0.2:7000->/127.0.0.1:7000-URGENT_MESSAGES-c650c9c5 channel in
potentially inconsistent state after error; closing\njava.net.SocketException:
Connection reset\n\tat
java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)\n\tat
java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)\n\tat
io.netty.buffer.UnpooledDirectByteBuf.setBytes(UnpooledDirectByteBuf.java:570)\n\tat
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)\n\tat
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833)
> Upgrade Netty on 4.x(current trunk)
> ---
>
> Key: CASSANDRA-17992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17992
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Low
> Fix For: 4.x
>
>
> I haven't been able to identify from the Netty docs which was the lowest
> version where JDK17 was added but we are about 40 versions behind in netty 4
> so I suspect we better update.
> We need to consider there was an issue with class cast exceptions when
> building with JDK17 with newer versions of netty (the newest available in
> March 2022). For the record, we didn't see those when running CI on JDK8 and
> JDK11. We also need to carefully revise the changes between the netty
> versions.
> Upgrading will cover also a fix in netty that was discussed in
> [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF
> Slack thread.
> CC [~benedict] , [~aleksey]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[ https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651001#comment-17651001 ] Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 12/21/22 9:05 PM: --- A few updates after I cleaned my JDK17 WIP branch and tested with different versions etc: * _bulkLoaderSuccessfullyStreamsOverSsl_ always fails with JDK17 - current and newer Netty versions with the classCastException we mentioned * _tls_test_ Python DTest is failing the same way with particular Netty versions. On further logging I can confirm that indeed [~benedict] is right it is submarine :) When using JDK provider and not OpenSSL, the class cast exception is there with JDK17, no matter of Netty Version. So two things: * I think the netty update can be finished without being blocked by the mentioned problem * we can go in a separate ticket addressing the classCastException and considering whether there might be an option without using add-exports. Not that I have a better idea at this very moment I will prepare a patch for the current ticket and also have to finish reviewing the list of changes between current and newest versions to ensure I did not miss anything that CI didn't catch was (Author: e.dimitrova): A few updates after I cleaned try JDK17 WIP branch and tested with different versions etc: * _bulkLoaderSuccessfullyStreamsOverSsl_ always fails with JDK17 - current and newer Netty versions with the classCastException we mentioned * _tls_test_ Python DTest is failing the same way with particular Netty versions. On further logging I can confirm that indeed [~benedict] is right it is submarine :) When using JDK provider and not OpenSSL, the class cast exception is there with JDK17, no matter of Netty Version. So two things: * I think the netty update can be finished without being blocked by the mentioned problem * we can go in a separate ticket addressing the classCastException and considering whether there might be an option without using add-exports. Not that I have a better idea at this very moment I will prepare a patch for the current ticket and also have to finish reviewing the list of changes between current and newest versions to ensure I did not miss anything that CI didn't catch > Upgrade Netty on 4.x(current trunk) > --- > > Key: CASSANDRA-17992 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17992 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Low > Fix For: 4.x > > > I haven't been able to identify from the Netty docs which was the lowest > version where JDK17 was added but we are about 40 versions behind in netty 4 > so I suspect we better update. > We need to consider there was an issue with class cast exceptions when > building with JDK17 with newer versions of netty (the newest available in > March 2022). For the record, we didn't see those when running CI on JDK8 and > JDK11. We also need to carefully revise the changes between the netty > versions. > Upgrading will cover also a fix in netty that was discussed in > [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF > Slack thread. > CC [~benedict] , [~aleksey] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[ https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17651001#comment-17651001 ] Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 12/21/22 9:05 PM: --- A few updates after I cleaned try JDK17 WIP branch and tested with different versions etc: * _bulkLoaderSuccessfullyStreamsOverSsl_ always fails with JDK17 - current and newer Netty versions with the classCastException we mentioned * _tls_test_ Python DTest is failing the same way with particular Netty versions. On further logging I can confirm that indeed [~benedict] is right it is submarine :) When using JDK provider and not OpenSSL, the class cast exception is there with JDK17, no matter of Netty Version. So two things: * I think the netty update can be finished without being blocked by the mentioned problem * we can go in a separate ticket addressing the classCastException and considering whether there might be an option without using add-exports. Not that I have a better idea at this very moment I will prepare a patch for the current ticket and also have to finish reviewing the list of changes between current and newest versions to ensure I did not miss anything that CI didn't catch was (Author: e.dimitrova): A few updates after I cleaned the branch and tested with different versions etc: * _bulkLoaderSuccessfullyStreamsOverSsl_ always fails with JDK17 - current and newer Netty versions with the classCastException we mentioned * _tls_test_ Python DTest is failing the same way with particular Netty versions. On further logging I can confirm that indeed [~benedict] is right it is submarine :) When using JDK provider and not OpenSSL, the class cast exception is there with JDK17, no matter of Netty Version. So two things: * I think the netty update can be finished without being blocked by the mentioned problem * we can go in a separate ticket addressing the classCastException and considering whether there might be an option without using add-exports. Not that I have a better idea at this very moment I will prepare a patch for the current ticket and also have to finish reviewing the list of changes between current and newest versions to ensure I did not miss anything that CI didn't catch > Upgrade Netty on 4.x(current trunk) > --- > > Key: CASSANDRA-17992 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17992 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Low > Fix For: 4.x > > > I haven't been able to identify from the Netty docs which was the lowest > version where JDK17 was added but we are about 40 versions behind in netty 4 > so I suspect we better update. > We need to consider there was an issue with class cast exceptions when > building with JDK17 with newer versions of netty (the newest available in > March 2022). For the record, we didn't see those when running CI on JDK8 and > JDK11. We also need to carefully revise the changes between the netty > versions. > Upgrading will cover also a fix in netty that was discussed in > [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF > Slack thread. > CC [~benedict] , [~aleksey] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[ https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642257#comment-17642257 ] Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 12/2/22 3:44 AM: -- Ok, it seems to me one of the issues we were hitting back in March was this bug - https://github.com/netty/netty/issues/12101. It was presented in 4.1.73, checked that it is fixed with 4.1.75 I easily can see it by running testServerSocketShouldUseKeystore and testClientSocketShouldUseOutboundKeystore The test_tls is probably different though. I have to run some tests tomorrow as some of the CircleCI ones I ran before have their artifacts already expired. was (Author: e.dimitrova): Ok, it seems to me one of the issues we were hitting back in March was this bug - https://github.com/netty/netty/issues/12101. It was presented in 4.1.73, checked that it is fixed with 4.1.75 I easily can see it running testServerSocketShouldUseKeystore and testClientSocketShouldUseOutboundKeystore The test_tls is probably different though. I have to run some tests tomorrow as some of the CircleCI ones I ran before have their artifacts already expired. > Upgrade Netty on 4.x(current trunk) > --- > > Key: CASSANDRA-17992 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17992 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Low > Fix For: 4.x > > > I haven't been able to identify from the Netty docs which was the lowest > version where JDK17 was added but we are about 40 versions behind in netty 4 > so I suspect we better update. > We need to consider there was an issue with class cast exceptions when > building with JDK17 with newer versions of netty (the newest available in > March 2022). For the record, we didn't see those when running CI on JDK8 and > JDK11. We also need to carefully revise the changes between the netty > versions. > Upgrading will cover also a fix in netty that was discussed in > [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF > Slack thread. > CC [~benedict] , [~aleksey] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625366#comment-17625366
]
Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 11/30/22 10:56 PM:
{quote}Getting the full stack trace will help inform next steps.
{quote}
I think I have some preliminary good news. I ran some rough tests today to get
to it. I suspect things might have improved on Netty side recently.
So with the Netty version that was current in March, [current trunk and
JDK17|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2005/workflows/18b54727-8b82-42cc-9dd2-5a04bfb06e5a]
I can see 55 failing tests and a bunch of SSL tests.
Now looking into the cqlsh tests
[here|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2005/workflows/18b54727-8b82-42cc-9dd2-5a04bfb06e5a/jobs/16218/tests#failed-test-1]
I can also find the issue we discussed, test_tls:
{code:java}
test teardown failure Unexpected error found in node logs (see stdout for full
details). Errors: [[node1] "WARN [nioEventLoopGroup-5-5] 2022-10-27
22:45:13,917 ExceptionHandlers.java:140 - Unknown exception in client
networking\nio.netty.handler.codec.DecoderException:
javax.net.ssl.SSLException: Fail to unwrap network record\n\tat
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833)\nCaused by:
javax.net.ssl.SSLException: Fail to unwrap network record\n\tat
java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)\n\tat
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)\n\tat
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:522)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)\n\tat
java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)\n\tat
io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295)\n\tat
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1341)\n\tat
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)\n\tat
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)\n\t...
17 common frames omitted\nCaused by: java.lang.ClassCastException: class
org.apache.cassandra.utils.memory.BufferPool$Chunk cannot be cast to class
sun.nio.ch.DirectBuffer (org.apache.cassandra.utils.memory.BufferPool$Chunk is
in unnamed module of loader 'app'; sun.nio.ch.DirectBuffer is in module
java.base of loader 'bootstrap')\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode$GCMEngine.overlapDetection(GaloisCounterMode.java:865)\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doFinal(GaloisCounterMode.java:1502)\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode.engineDoFinal(GaloisCounterMode.java:447)\n\tat
java.base/javax.crypto.Cipher.doFinal(Cipher.java:2500)\n\tat
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625579#comment-17625579
]
Benedict Elliott Smith edited comment on CASSANDRA-17992 at 10/28/22 10:33 AM:
---
Yes, the {{GaloisCounterMode}} appears to make about {{DirectBuffer}} and their
attachments
[here|https://github.com/openjdk/jdk/blob/jdk-17%2B35/src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java#L866].
Looking again at the rest of the JDK, I'm not actually sure attachments can be
accessed publicly, so this is probably our problem rather than the JDK's.
However, we might be able to fix it by simply having {{Chunk}} implement
{{DirectBuffer}}, and having it return its address.
was (Author: benedict):
Yes, the {{GaloisCounterMode}} appears to make about {{DirectBuffer}} and their
attachments
[here|https://github.com/openjdk/jdk/blob/jdk-17%2B35/src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java#L866].
Looking again at the rest of the JDK, I'm not actually sure attachments can be
accessed publicly, so this is probably our problem rather than the JDK's.
> Upgrade Netty on 4.x(current trunk)
> ---
>
> Key: CASSANDRA-17992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17992
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Ekaterina Dimitrova
>Priority: Low
> Fix For: 4.x
>
>
> I haven't been able to identify from the Netty docs which was the lowest
> version where JDK17 was added but we are about 40 versions behind in netty 4
> so I suspect we better update.
> We need to consider there was an issue with class cast exceptions when
> building with JDK17 with newer versions of netty (the newest available in
> March 2022). For the record, we didn't see those when running CI on JDK8 and
> JDK11. We also need to carefully revise the changes between the netty
> versions.
> Upgrading will cover also a fix in netty that was discussed in
> [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF
> Slack thread.
> CC [~benedict] , [~aleksey]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625579#comment-17625579
]
Benedict Elliott Smith edited comment on CASSANDRA-17992 at 10/28/22 10:32 AM:
---
Yes, the {{GaloisCounterMode}} appears to make about {{DirectBuffer}} and their
attachments
[here|https://github.com/openjdk/jdk/blob/jdk-17%2B35/src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java#L866].
Looking again at the rest of the JDK, I'm not actually sure attachments can be
accessed publicly, so this is probably our problem rather than the JDK's.
was (Author: benedict):
Yes, the {{GaloisCounterMode}} appears to make bad assumptions about
{{DirectBuffer}} and their attachments
[here|https://github.com/openjdk/jdk/blob/jdk-17%2B35/src/java.base/share/classes/com/sun/crypto/provider/GaloisCounterMode.java#L866]
> Upgrade Netty on 4.x(current trunk)
> ---
>
> Key: CASSANDRA-17992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17992
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Ekaterina Dimitrova
>Priority: Low
> Fix For: 4.x
>
>
> I haven't been able to identify from the Netty docs which was the lowest
> version where JDK17 was added but we are about 40 versions behind in netty 4
> so I suspect we better update.
> We need to consider there was an issue with class cast exceptions when
> building with JDK17 with newer versions of netty (the newest available in
> March 2022). For the record, we didn't see those when running CI on JDK8 and
> JDK11. We also need to carefully revise the changes between the netty
> versions.
> Upgrading will cover also a fix in netty that was discussed in
> [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF
> Slack thread.
> CC [~benedict] , [~aleksey]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625366#comment-17625366
]
Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 10/28/22 1:37 AM:
---
{quote}Getting the full stack trace will help inform next steps.
{quote}
I think I have some preliminary good news. I ran some rough tests today to get
to it. I suspect things might have improved on Netty side recently.
So with the Netty version that was current in March, [current trunk and
JDK17|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra?branch=trunk-jdk17-oct]
I can see 55 failing tests and a bunch of SSL tests.
Now looking into the cqlsh tests
[here|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2005/workflows/18b54727-8b82-42cc-9dd2-5a04bfb06e5a/jobs/16218/tests#failed-test-1]
I can also find the issue we discussed, test_tls:
{code:java}
test teardown failure Unexpected error found in node logs (see stdout for full
details). Errors: [[node1] "WARN [nioEventLoopGroup-5-5] 2022-10-27
22:45:13,917 ExceptionHandlers.java:140 - Unknown exception in client
networking\nio.netty.handler.codec.DecoderException:
javax.net.ssl.SSLException: Fail to unwrap network record\n\tat
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)\n\tat
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)\n\tat
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
java.base/java.lang.Thread.run(Thread.java:833)\nCaused by:
javax.net.ssl.SSLException: Fail to unwrap network record\n\tat
java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)\n\tat
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)\n\tat
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:522)\n\tat
java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)\n\tat
java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)\n\tat
io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295)\n\tat
io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1341)\n\tat
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)\n\tat
io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)\n\tat
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)\n\t...
17 common frames omitted\nCaused by: java.lang.ClassCastException: class
org.apache.cassandra.utils.memory.BufferPool$Chunk cannot be cast to class
sun.nio.ch.DirectBuffer (org.apache.cassandra.utils.memory.BufferPool$Chunk is
in unnamed module of loader 'app'; sun.nio.ch.DirectBuffer is in module
java.base of loader 'bootstrap')\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode$GCMEngine.overlapDetection(GaloisCounterMode.java:865)\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doFinal(GaloisCounterMode.java:1502)\n\tat
java.base/com.sun.crypto.provider.GaloisCounterMode.engineDoFinal(GaloisCounterMode.java:447)\n\tat
java.base/javax.crypto.Cipher.doFinal(Cipher.java:2500)\n\tat
java.base/sun.security.ssl.SSLCipher$T12GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1659)\n\tat
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624499#comment-17624499
]
Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 10/26/22 2:21 PM:
---
{quote}Do you have a reference to the exceptions we were seeing?
{quote}
Unfortunately, those CircleCI runs logs have expired. I will try to bump the
version later today and try on a recent rebase with JDK17.
But this is an excerpt I found in my TODO list:
{code:java}
java.lang.ClassCastException: class
org.apache.cassandra.utils.memory.BufferPool$Chunk cannot be cast to class
sun.nio.ch.DirectBuffer{code}
{quote}I vaguely recall that it only supports {{DirectByteBuffer}} now, which
is probably quite straightforward to resolve.
{quote}
Quite possible. I haven't dug into it yet, just literally put it on the list
just as a thing to consider when the time comes. Thank you for the hint.
was (Author: e.dimitrova):
{quote}Do you have a reference to the exceptions we were seeing?
{quote}
Unfortunately, those CircleCI runs logs have expired. I will try to bump the
version later today and try on a recent rebase with JDK17.
But this is an excerpt I found in my TODO list:
java.lang.ClassCastException: class
org.apache.cassandra.utils.memory.BufferPool$Chunk cannot be cast to class
sun.nio.ch.DirectBuffer
{quote}I vaguely recall that it only supports {{DirectByteBuffer}} now, which
is probably quite straightforward to resolve.
{quote}
Quite possible. I haven't dug into it yet, just literally put it on the list
just as a thing to consider when the time comes. Thank you for the hint.
> Upgrade Netty on 4.x(current trunk)
> ---
>
> Key: CASSANDRA-17992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17992
> Project: Cassandra
> Issue Type: Task
> Components: Dependencies
>Reporter: Ekaterina Dimitrova
>Priority: Low
> Fix For: 4.x
>
>
> I haven't been able to identify from the Netty docs which was the lowest
> version where JDK17 was added but we are about 40 versions behind in netty 4
> so I suspect we better update.
> We need to consider there was an issue with class cast exceptions when
> building with JDK17 with newer versions of netty (the newest available in
> March 2022). For the record, we didn't see those when running CI on JDK8 and
> JDK11. We also need to carefully revision the changes between the netty
> versions.
> Upgrading will cover also a fix that was discussed in
> [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF
> Slack thread.
> CC [~benedict] , [~aleksey]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-17992) Upgrade Netty on 4.x(current trunk)
[ https://issues.apache.org/jira/browse/CASSANDRA-17992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624111#comment-17624111 ] Ekaterina Dimitrova edited comment on CASSANDRA-17992 at 10/26/22 1:25 AM: --- [~norman] , do you think you can confirm for us the earliest netty version that supports JDK17? Or even point me in the docs if it is listed somewhere and I am missing it so I know for the next time where to look. Thanks in advance was (Author: e.dimitrova): [~norman] , do you think you can confirm for us the earliest netty version that supports JDK17? Or even point me in the docs if it is listed somewhere and I am missing so I know for the next time. Thanks in advance > Upgrade Netty on 4.x(current trunk) > --- > > Key: CASSANDRA-17992 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17992 > Project: Cassandra > Issue Type: Task > Components: Dependencies >Reporter: Ekaterina Dimitrova >Priority: Low > Fix For: 4.x > > > I haven't been able to identify from the Netty docs which was the lowest > version where JDK17 was added but we are about 40 versions behind in netty 4 > so I suspect we better update. > We need to consider there was an issue with class cast exceptions when > building with JDK17 with newer versions of netty (the newest available in > March 2022). For the record, we didn't see those when running CI on JDK8 and > JDK11. We also need to carefully revision the changes between the netty > versions. > Upgrading will cover also a fix that was discussed in > [this|https://the-asf.slack.com/archives/CK23JSY2K/p1665567660202989] ASF > Slack thread. > CC [~benedict] , [~aleksey] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
