The only difference I found is that the new nodes have been installed with Debian-12 and OpenJDK-17 while the rest of the cluster is running on Debian-11 with OpenJDK-11.
> On 9. May 2025, at 07:34, Tibor Répási <tibor.rep...@anzix.org> wrote: > > Rechecked that, the upgrade was complete, all nodes are running 5.0.4 and the > new once trying to join have the same version installed. > >> On 8. May 2025, at 16:53, C. Scott Andreas <sc...@paradoxica.net> wrote: >> >> Hi Tibor, >> >> Can you double check the versions in use by all streaming peers? >> >> I see the "app" module issue you mention and the ClassCastException – but >> notice the attempt to cast Result.INCOMPATIBLE to Result.SUCCESS. If >> Result.INCOMPATIBLE is the node's determination, this looks like an attempt >> to stream across incompatible versions of the database. >> >> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundConnectionInitiator.java#L451-L467 >> >> – Scott >> >>> On May 8, 2025, at 5:18 AM, Tibor Répási <tibor.rep...@anzix.org> wrote: >>> >>> >>> Hi all, >>> >>> Recently I face some issues when a node is joining an already existing >>> cluster, the bootstrap ends up incomplete and in failed state due to some >>> streams have failed. >>> Using Cassandra 5.0.3 running on OpenJDK 17.0.15: >>> >>> Logs show: >>> >>> WARN [NonPeriodicTasks:1] 2025-05-08 13:23:26,145 >>> StreamResultFuture.java:250 - [Stream >>> #e8dc7090-2bf8-11f0-8e15-57fc8d65eca3] Stream failed: >>> Session peer /172.20.145.155:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> Session peer /172.20.145.29:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> Session peer /172.20.145.207:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> Session peer /172.20.145.209:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> ERROR [main] 2025-05-08 13:23:26,149 StorageService.java:2216 - Error while >>> waiting on bootstrap to complete. Bootstrap will have to >>> be restarted. >>> java.util.concurrent.ExecutionException: >>> org.apache.cassandra.streaming.StreamException: Stream failed: >>> Session peer /172.20.145.155:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> Session peer /172.20.145.29:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> Session peer /172.20.145.207:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> Session peer /172.20.145.209:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> at >>> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239) >>> at >>> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246) >>> at >>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2209) >>> at >>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1292) >>> at >>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1252) >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:1031) >>> at >>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:947) >>> at >>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:383) >>> at >>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:727) >>> at >>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:865) >>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed: >>> Session peer /172.20.145.155:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> Session peer /172.20.145.29:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> Session peer /172.20.145.207:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> Session peer /172.20.145.209:7000 Failed because of an unknown exception >>> java.lang.ClassCastException: class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >>> cannot be cast to class >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >>> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >>> compatible and >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader 'app') >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >>> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >>> >>> at >>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:251) >>> at >>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205) >>> at >>> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:552) >>> at org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:124) >>> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) >>> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) >>> at >>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) >>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >>> at >>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>> at >>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >>> at java.base/java.lang.Thread.run(Thread.java:840) >>> INFO [main] 2025-05-08 13:23:26,149 StorageService.java:2042 - >>> JOINING_FAILED >>> WARN [main] 2025-05-08 13:23:26,180 StorageService.java:1328 - Some data >>> streaming failed. Use nodetool to check bootstrap state an >>> d resume. For more, see `nodetool help bootstrap`. IN_PROGRESS >>> INFO [main] 2025-05-08 13:23:26,189 Gossiper.java:2449 - Waiting for gossip >>> to settle... >>> INFO [main] 2025-05-08 13:23:34,190 Gossiper.java:2480 - No gossip backlog; >>> proceeding >>> >>> Any clue what’s going wrong here? The part with >>> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >>> unnamed module of loader ‘app’ looks very strange. >> >> >
