Hi Tibor, Can you double check the versions in use by all streaming peers? I see the
"app" module issue you mention and the ClassCastException – but notice the attempt to
cast Result.INCOMPATIBLE to Result.SUCCESS. If Result.INCOMPATIBLE is the node's determination,
this looks like an attempt to stream across incompatible versions of the database.
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/net/OutboundConnectionInitiator.java#L451-L467
– Scott On May 8, 2025, at 5:18 AM, Tibor Répási <tibor.rep...@anzix.org> wrote: Hi all,
Recently I face some issues when a node is joining an already existing cluster, the bootstrap
ends up incomplete and in failed state due to some streams have failed. Using Cassandra 5.0.3
running on OpenJDK 17.0.15: Logs show: WARN [NonPeriodicTasks:1] 2025-05-08 13:23:26,145
StreamResultFuture.java:250 - [Stream #e8dc7090-2bf8-11f0-8e15-57fc8d65eca3] Stream failed:
Session peer /172.20.145.155:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.29:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.207:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.209:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
ERROR [main] 2025-05-08 13:23:26,149 StorageService.java:2216 - Error while waiting on
bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream
failed: Session peer /172.20.145.155:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.29:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.207:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.209:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
at org.apache.cassandra.utils.concurrent.AbstractFuture.getWhenDone(AbstractFuture.java:239)
at org.apache.cassandra.utils.concurrent.AbstractFuture.get(AbstractFuture.java:246) at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:2209) at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1292) at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1252) at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:1031) at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:947) at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:383) at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:727) at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:865) Caused by:
org.apache.cassandra.streaming.StreamException: Stream failed: Session peer
/172.20.145.155:7000 Failed because of an unknown exception java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.29:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.207:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
Session peer /172.20.145.209:7000 Failed because of an unknown exception
java.lang.ClassCastException: class
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Incompatible cannot be cast to
class org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success
(org.apache.cassandra.net.OutboundConnectionInitiator$Result$In compatible and
org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in unnamed module of
loader 'app')
org.apache.cassandra.net.OutboundConnectionInitiator$Result.success(OutboundConnectionInitiator.java:520)
org.apache.cassandra.streaming.async.NettyStreamingConnectionFactory.connect(NettyStreamingConnectionFactory.java:68)
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:251) at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:205)
at org.apache.cassandra.streaming.StreamSession.lambda$closeSession$2(StreamSession.java:552)
at org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:124) at
org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61) at
org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71) at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at
java.base/java.lang.Thread.run(Thread.java:840) INFO [main] 2025-05-08 13:23:26,149
StorageService.java:2042 - JOINING_FAILED WARN [main] 2025-05-08 13:23:26,180
StorageService.java:1328 - Some data streaming failed. Use nodetool to check bootstrap state an
d resume. For more, see `nodetool help bootstrap`. IN_PROGRESS INFO [main] 2025-05-08
13:23:26,189 Gossiper.java:2449 - Waiting for gossip to settle... INFO [main] 2025-05-08
13:23:34,190 Gossiper.java:2480 - No gossip backlog; proceeding Any clue what’s going wrong
here? The part with org.apache.cassandra.net.OutboundConnectionInitiator$Result$Success are in
unnamed module of loader ‘app’ looks very strange.
