You have a storage layer problem rather than a Cassandra problem. Probably a failing drive, but there's a smaller chance it could be something like bad or loose cabling, bad controller, or bad RAM.
As a side note: I understand the challenges and risks of major version upgrades in production all too well, but I'd definitely recommend at least going to the latest 3.11 (3.11.19). Most of the fixes since 3.11.13 are relatively minor, but there's a few significant ones in there. On Tue, Oct 21, 2025 at 8:23 PM Suresh Koya via user < [email protected]> wrote: > Hello > I am seeing the error below in my cassandra server logs. I am running > cassandra 3.11.13. I understand the version is old. I know a lot of people > run old cassandra versions. Has anyone seen this error in the past. > > What concerns me is that, after this error it appears the server appeared > to silently restart. I see the startup time messages > > INFO [main] 2025-10-21 19:06:55,713 Gossiper.java:1900 - No gossip > backlog; proceeding > INFO [main] 2025-10-21 19:06:56,622 NativeTransportService.java:68 - > Netty using native Epoll event loop > > Is this how it is supposed to work? > > INFO [Service Thread] 2025-10-21 18:56:14,400 GCInspector.java:285 - G1 > Young Generation GC in 289ms. G1 Eden Space: 1451229184 -> 0; G1 Old Gen: > 7839154160 -> 7436500976; > ERROR [PerDiskMemtableFlushWriter_0:1] 2025-10-21 18:56:55,885 > CassandraDaemon.java:244 - Exception in thread > Thread[PerDiskMemtableFlushWriter_0:1,5,main] > java.lang.ClassCastException: <unknown> cannot be cast to > org.apache.cassandra.db.rows.ColumnData > at > org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:681) > at > org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:724) > at > org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:699) > at > org.apache.cassandra.db.rows.BTreeRow.transformAndFilter(BTreeRow.java:430) > at org.apache.cassandra.db.rows.BTreeRow.filter(BTreeRow.java:298) > at > org.apache.cassandra.db.rows.RowAndDeletionMergeIterator.consumeNextRow(RowAndDeletionMergeIterator.java:178) > at > org.apache.cassandra.db.rows.RowAndDeletionMergeIterator.computeNextInternal(RowAndDeletionMergeIterator.java:97) > at > org.apache.cassandra.db.rows.RowAndDeletionMergeIterator.computeNext(RowAndDeletionMergeIterator.java:144) > at > org.apache.cassandra.db.rows.RowAndDeletionMergeIterator.computeNext(RowAndDeletionMergeIterator.java:35) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) > at > org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:110) > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173) > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48) > at > org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:482) > at > org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:517) > at > org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:404) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) > at java.lang.Thread.run(Thread.java:750) > ERROR [MemtablePostFlush:1] 2025-10-21 18:56:55,951 > CassandraDaemon.java:244 - Exception in thread > Thread[MemtablePostFlush:1,5,main] > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.ClassCastException: <unknown> cannot be cast to > org.apache.cassandra.db.rows.ColumnData > at > org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:62) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:430) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:393) > at > org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1202) > at > org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1143) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) > at java.lang.Thread.run(Thread.java:750) > Caused by: java.util.concurrent.ExecutionException: > java.lang.ClassCastException: <unknown> cannot be cast to > org.apache.cassandra.db.rows.ColumnData > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:417) > ... 7 common frames omitted > Caused by: java.lang.ClassCastException: <unknown> cannot be cast to > org.apache.cassandra.db.rows.ColumnData > at > org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:681) > at > org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:724) > at > org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:699) > at > org.apache.cassandra.db.rows.BTreeRow.transformAndFilter(BTreeRow.java:430) > at org.apache.cassandra.db.rows.BTreeRow.filter(BTreeRow.java:298) > at > org.apache.cassandra.db.rows.RowAndDeletionMergeIterator.consumeNextRow(RowAndDeletionMergeIterator.java:178) > at > org.apache.cassandra.db.rows.RowAndDeletionMergeIterator.computeNextInternal(RowAndDeletionMergeIterator.java:97) > at > org.apache.cassandra.db.rows.RowAndDeletionMergeIterator.computeNext(RowAndDeletionMergeIterator.java:144) > at > org.apache.cassandra.db.rows.RowAndDeletionMergeIterator.computeNext(RowAndDeletionMergeIterator.java:35) > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) > at > org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:110) > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173) > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48) > at > org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:482) > at > org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:517) > at > org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:404) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ... 4 common frames omitted > > I checked dmesg output > > [Tue Oct 21 19:00:18 2025] INFO: task java:24431 blocked for more than 122 > seconds. > [Tue Oct 21 19:00:18 2025] Not tainted 6.8.0-1031-aws > #33~22.04.1-Ubuntu > [Tue Oct 21 19:00:18 2025] "echo 0 > > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [Tue Oct 21 19:00:18 2025] task:java state:D stack:0 > pid:24431 tgid:24431 ppid:1 flags:0x00004002 > [Tue Oct 21 19:00:18 2025] Call Trace: > [Tue Oct 21 19:00:18 2025] <TASK> > [Tue Oct 21 19:00:18 2025] __schedule+0x27c/0x680 > [Tue Oct 21 19:00:18 2025] schedule+0x2c/0xf0 > [Tue Oct 21 19:00:18 2025] do_exit+0x117/0x510 > [Tue Oct 21 19:00:18 2025] do_group_exit+0x35/0x90 > [Tue Oct 21 19:00:18 2025] get_signal+0x963/0x970 > [Tue Oct 21 19:00:18 2025] arch_do_signal_or_restart+0x39/0x120 > [Tue Oct 21 19:00:18 2025] syscall_exit_to_user_mode+0x206/0x260 > [Tue Oct 21 19:00:18 2025] do_syscall_64+0x8d/0x170 > [Tue Oct 21 19:00:18 2025] ? __update_load_avg_cfs_rq+0x212/0x2a0 > [Tue Oct 21 19:00:18 2025] ? place_entity+0xf0/0x100 > [Tue Oct 21 19:00:18 2025] ? __enqueue_entity+0x109/0x150 > [Tue Oct 21 19:00:18 2025] ? enqueue_entity+0xfc/0x570 > [Tue Oct 21 19:00:18 2025] ? enqueue_task_fair+0x1b7/0x360 > [Tue Oct 21 19:00:18 2025] ? x2apic_send_IPI+0x5d/0x90 > [Tue Oct 21 19:00:18 2025] ? native_smp_send_reschedule+0x1f/0x50 > [Tue Oct 21 19:00:18 2025] ? resched_curr+0x5e/0x140 > [Tue Oct 21 19:00:18 2025] ? _raw_spin_unlock_irqrestore+0x11/0x40 > [Tue Oct 21 19:00:18 2025] ? wake_up_new_task+0x22b/0x300 > [Tue Oct 21 19:00:18 2025] ? __raw_spin_unlock_irq+0x9/0x20 > [Tue Oct 21 19:00:18 2025] ? sigprocmask+0xb4/0xe0 > [Tue Oct 21 19:00:18 2025] ? __x64_sys_rt_sigprocmask+0x7f/0xe0 > [Tue Oct 21 19:00:18 2025] ? syscall_exit_to_user_mode+0x86/0x260 > [Tue Oct 21 19:00:18 2025] ? do_syscall_64+0x8d/0x170 > [Tue Oct 21 19:00:18 2025] ? __do_sys_clone3+0xf6/0x130 > [Tue Oct 21 19:00:18 2025] ? syscall_exit_to_user_mode+0x86/0x260 > [Tue Oct 21 19:00:18 2025] ? do_syscall_64+0x8d/0x170 > [Tue Oct 21 19:00:18 2025] ? irqentry_exit_to_user_mode+0x7b/0x260 > [Tue Oct 21 19:00:18 2025] ? irqentry_exit+0x21/0x40 > [Tue Oct 21 19:00:18 2025] ? exc_page_fault+0x94/0x190 > [Tue Oct 21 19:00:18 2025] entry_SYSCALL_64_after_hwframe+0x78/0x80 > [Tue Oct 21 19:00:18 2025] RIP: 0033:0x7405dd291117 > [Tue Oct 21 19:00:18 2025] RSP: 002b:00007ffea0bf25a0 EFLAGS: 00000246 > ORIG_RAX: 00000000000000ca > [Tue Oct 21 19:00:18 2025] RAX: fffffffffffffe00 RBX: 00007405dd4b4640 > RCX: 00007405dd291117 > [Tue Oct 21 19:00:18 2025] RDX: 0000000000005f70 RSI: 0000000000000109 > RDI: 00007405dd4b4910 > [Tue Oct 21 19:00:18 2025] RBP: 00007ffea0bf2668 R08: 0000000000000000 > R09: 00000000ffffffff > [Tue Oct 21 19:00:18 2025] R10: 0000000000000000 R11: 0000000000000246 > R12: 0000000000000000 > [Tue Oct 21 19:00:18 2025] R13: 0000000000005f70 R14: 0000000000000000 > R15: 0000000000000000 > > Juniper Business Use Only > -- This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and is solely for the review and use of the intended recipient(s). If you have received this email in error, please notify the sender and permanently delete this email, its content, and any attachment(s). Any disclosure, copying, or taking of any action in reliance on an email received in error is strictly prohibited.
