Rechecked that, the upgrade was complete, all nodes are running 5.0.4 and the new once trying to join have the same version installed.
> On 8. May 2025, at 16:53, C. Scott Andreas <sc...@paradoxica.net> wrote: > > Hi Tibor, > > Can you double check the versions in use by all streaming peers? > > I see the "app" module issue you mention and the ClassCastException – but > notice the attempt to cast Result.INCOMPATIBLE to Result.SUCCESS. If > Result.INCOMPATIBLE is the node's determination, this looks like an attempt > to stream across incompatible versions of the database. > > https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundConnectionInitiator.java#L451-L467 > > – Scott > >> On May 8, 2025, at 5:18 AM, Tibor Répási <tibor.rep...@anzix.org> wrote: >> >> >> Hi all, >> >> Recently I face some issues when a node is joining an already existing >> cluster, the bootstrap ends up incomplete and in failed state due to some >> streams have failed. >> Using Cassandra 5.0.3 running on OpenJDK 17.0.15: >> >> Logs show: >> >> WARN [NonPeriodicTasks:1] 2025-05-08 13:23:26,145 >> StreamResultFuture.java:250 - [Stream #e8dc7090-2bf8-11f0-8e15-57fc8d65eca3] >> Stream failed: >> Session peer /172.20.145.155:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> Session peer /172.20.145.29:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> Session peer /172.20.145.207:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> Session peer /172.20.145.209:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> ERROR [main] 2025-05-08 13:23:26,149 StorageService.java:2216 - Error while >> waiting on bootstrap to complete. Bootstrap will have to >> be restarted. >> java.util.concurrent.ExecutionException: >> org.apache.cassandra.streaming.StreamException: Stream failed: >> Session peer /172.20.145.155:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> Session peer /172.20.145.29:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> Session peer /172.20.145.207:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> Session peer /172.20.145.209:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> at >> org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239) >> at >> org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246) >> at >> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2209) >> at >> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1292) >> at >> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1252) >> at >> org.apache.cassandra.service.StorageService.initServer(StorageService.java:1031) >> at >> org.apache.cassandra.service.StorageService.initServer(StorageService.java:947) >> at >> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:383) >> at >> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:727) >> at >> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:865) >> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed: >> Session peer /172.20.145.155:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> Session peer /172.20.145.29:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> Session peer /172.20.145.207:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> Session peer /172.20.145.209:7000 Failed because of an unknown exception >> java.lang.ClassCastException: class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible >> cannot be cast to class >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success >> (org.apache.cassandra.net.OutboundConnectionInitiator$Result$In >> compatible and >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader 'app') >> org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520) >> org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68) >> >> at >> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:251) >> at >> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205) >> at >> org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:552) >> at org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:124) >> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) >> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) >> at >> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) >> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) >> at >> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >> at >> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) >> at java.base/java.lang.Thread.run(Thread.java:840) >> INFO [main] 2025-05-08 13:23:26,149 StorageService.java:2042 - JOINING_FAILED >> WARN [main] 2025-05-08 13:23:26,180 StorageService.java:1328 - Some data >> streaming failed. Use nodetool to check bootstrap state an >> d resume. For more, see `nodetool help bootstrap`. IN_PROGRESS >> INFO [main] 2025-05-08 13:23:26,189 Gossiper.java:2449 - Waiting for gossip >> to settle... >> INFO [main] 2025-05-08 13:23:34,190 Gossiper.java:2480 - No gossip backlog; >> proceeding >> >> Any clue what’s going wrong here? The part with >> org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in >> unnamed module of loader ‘app’ looks very strange. > >
