[jira] [Commented] (CASSANDRA-18815) Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
[
https://issues.apache.org/jira/browse/CASSANDRA-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762300#comment-17762300
]
Berenguer Blasi commented on CASSANDRA-18815:
-
LGTM. Are we going to open a ticket for 4.1 to surface these?
> Fix dtests:
> replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
> --
>
> Key: CASSANDRA-18815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18815
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/
> This and other similar tests recently started failing:
> {noformat}
> Unexpected error found in node logs (see stdout for full details). Errors:
> [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47]
> 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream
> #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session
> completion, peer 127.0.0.1:7000 is probably
> down.\njava.nio.channels.ClosedChannelException: null\n\tat
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat
>
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat
>
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat
>
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
> java.base/java.lang.Thread.run(Thread.java:833)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18815) Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
[
https://issues.apache.org/jira/browse/CASSANDRA-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762157#comment-17762157
]
Brandon Williams commented on CASSANDRA-18815:
--
bq. If the test is causing the log error, then adding a suppression for it
seems like the way to go.
That is simple enough, and I think a single dtest-large run will suffice:
||Branch||CI||
|[5.0|https://github.com/driftx/cassandra/tree/CASSANDRA-18815-5.0]|[dtests-large|https://app.circleci.com/pipelines/github/driftx/cassandra/1259/workflows/dca68d84-4cf8-40fe-a660-30feca8edfa3/jobs/50195]|
> Fix dtests:
> replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
> --
>
> Key: CASSANDRA-18815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18815
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/
> This and other similar tests recently started failing:
> {noformat}
> Unexpected error found in node logs (see stdout for full details). Errors:
> [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47]
> 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream
> #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session
> completion, peer 127.0.0.1:7000 is probably
> down.\njava.nio.channels.ClosedChannelException: null\n\tat
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat
>
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat
>
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat
>
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
> java.base/java.lang.Thread.run(Thread.java:833)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18815) Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
[
https://issues.apache.org/jira/browse/CASSANDRA-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761407#comment-17761407
]
Jon Meredith commented on CASSANDRA-18815:
--
The difference in behavior is interesting and I don't have a good explanation
for. The clusters we saw the deadlock on were running 4.1 and the changes in
18733 could definitely affect the order threads see the message.
If the test is causing the log error, then adding a suppression for it seems
like the way to go.
> Fix dtests:
> replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
> --
>
> Key: CASSANDRA-18815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18815
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/
> This and other similar tests recently started failing:
> {noformat}
> Unexpected error found in node logs (see stdout for full details). Errors:
> [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47]
> 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream
> #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session
> completion, peer 127.0.0.1:7000 is probably
> down.\njava.nio.channels.ClosedChannelException: null\n\tat
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat
>
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat
>
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat
>
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
> java.base/java.lang.Thread.run(Thread.java:833)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18815) Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
[
https://issues.apache.org/jira/browse/CASSANDRA-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761404#comment-17761404
]
Brandon Williams commented on CASSANDRA-18815:
--
After digging into this a bit I think this error, though newly appearing, is
actually correct and was missing before: all the dtests that are failing are
resuming or restarting an operation, so they are indeed causing a failure in
the stream session in order to facilitate that. This would mean that
CASSANDRA-18733 actually fixed whatever was suppressing this message before,
and the bug is now in 4.1 where we don't see it. WDYT, [~jmeredithco]?
If this is the case, then in this ticket we should accept this error in the
dtests, and make a new ticket to fix whatever is hiding it in 4.1.
> Fix dtests:
> replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
> --
>
> Key: CASSANDRA-18815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18815
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/
> This and other similar tests recently started failing:
> {noformat}
> Unexpected error found in node logs (see stdout for full details). Errors:
> [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47]
> 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream
> #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session
> completion, peer 127.0.0.1:7000 is probably
> down.\njava.nio.channels.ClosedChannelException: null\n\tat
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat
>
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat
>
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat
>
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
> java.base/java.lang.Thread.run(Thread.java:833)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18815) Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
[
https://issues.apache.org/jira/browse/CASSANDRA-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761066#comment-17761066
]
Ekaterina Dimitrova commented on CASSANDRA-18815:
-
That's what my non-native speaker me was trying to say :D the patch is the
same, but it boils to the rest of the C* code. There are indeed a few
differences. :D
> Fix dtests:
> replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
> --
>
> Key: CASSANDRA-18815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18815
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/
> This and other similar tests recently started failing:
> {noformat}
> Unexpected error found in node logs (see stdout for full details). Errors:
> [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47]
> 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream
> #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session
> completion, peer 127.0.0.1:7000 is probably
> down.\njava.nio.channels.ClosedChannelException: null\n\tat
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat
>
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat
>
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat
>
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
> java.base/java.lang.Thread.run(Thread.java:833)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18815) Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
[
https://issues.apache.org/jira/browse/CASSANDRA-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761063#comment-17761063
]
Brandon Williams commented on CASSANDRA-18815:
--
Your assumption is correct, it's definitely caused by CASSANDRA-18733.
bq. Maybe not from patch perspective, but the rest of the C* code?
I meant the patch. I hope there are differences between 4.1 and 5.0 :)
> Fix dtests:
> replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
> --
>
> Key: CASSANDRA-18815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18815
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/
> This and other similar tests recently started failing:
> {noformat}
> Unexpected error found in node logs (see stdout for full details). Errors:
> [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47]
> 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream
> #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session
> completion, peer 127.0.0.1:7000 is probably
> down.\njava.nio.channels.ClosedChannelException: null\n\tat
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat
>
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat
>
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat
>
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
> java.base/java.lang.Thread.run(Thread.java:833)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18815) Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
[
https://issues.apache.org/jira/browse/CASSANDRA-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761062#comment-17761062
]
Ekaterina Dimitrova commented on CASSANDRA-18815:
-
Maybe not from patch perspective, but the rest of the C* code? As it seems to
fail consistently, I guess we should just bisect it to confirm. I can do it
later or next week. My assumption was based on Butler, Jenkins and Jira :)
> Fix dtests:
> replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
> --
>
> Key: CASSANDRA-18815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18815
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/
> This and other similar tests recently started failing:
> {noformat}
> Unexpected error found in node logs (see stdout for full details). Errors:
> [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47]
> 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream
> #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session
> completion, peer 127.0.0.1:7000 is probably
> down.\njava.nio.channels.ClosedChannelException: null\n\tat
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat
>
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat
>
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat
>
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
> java.base/java.lang.Thread.run(Thread.java:833)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18815) Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
[
https://issues.apache.org/jira/browse/CASSANDRA-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761061#comment-17761061
]
Brandon Williams commented on CASSANDRA-18815:
--
bq. I guess there are some differences
Heh, there actually are not any, it was a clean merge from 4.1 to 5.0.
> Fix dtests:
> replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
> --
>
> Key: CASSANDRA-18815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18815
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/
> This and other similar tests recently started failing:
> {noformat}
> Unexpected error found in node logs (see stdout for full details). Errors:
> [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47]
> 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream
> #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session
> completion, peer 127.0.0.1:7000 is probably
> down.\njava.nio.channels.ClosedChannelException: null\n\tat
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat
>
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat
>
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat
>
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
> java.base/java.lang.Thread.run(Thread.java:833)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18815) Fix dtests: replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
[
https://issues.apache.org/jira/browse/CASSANDRA-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17761048#comment-17761048
]
Ekaterina Dimitrova commented on CASSANDRA-18815:
-
I also noticed today that TestReplaceAddress started consistently failing in
5.0 and trunk, probably after
CASSANDRA-18733
It makes sense, as pre-commit, that we do not have the large tests as a
requirement to run. They are optional in CircleCI. Though those are not failing
on 4.1 so I guess there are some differences. I haven't checked the details, it
was on my list for later but you were faster [~brandon.williams] :D
> Fix dtests:
> replace_address_test.TestReplaceAddress.test_restart_failed_replace and others
> --
>
> Key: CASSANDRA-18815
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18815
> Project: Cassandra
> Issue Type: Bug
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Normal
> Fix For: 5.0.x, 5.x
>
>
> https://ci-cassandra.apache.org/job/Cassandra-5.0/18/testReport/dtest-large.replace_address_test/TestReplaceAddress/test_restart_failed_replace/
> This and other similar tests recently started failing:
> {noformat}
> Unexpected error found in node logs (see stdout for full details). Errors:
> [[replacement] 'ERROR [Stream-Deserializer-/127.0.0.1:7000-91782e47]
> 2023-08-29 23:05:51,677 StreamSession.java:700 - [Stream
> #990152d0-46c0-11ee-9290-158c46e94542] Socket closed before session
> completion, peer 127.0.0.1:7000 is probably
> down.\njava.nio.channels.ClosedChannelException: null\n\tat
> org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)\n\tat
>
> org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)\n\tat
>
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)\n\tat
>
> org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)\n\tat
>
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat
> java.base/java.lang.Thread.run(Thread.java:833)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
