[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15493711#comment-15493711
]
Sam Tunnicliffe commented on CASSANDRA-12590:
---------------------------------------------
I spent some time hacking on a unit test using byteman to inject
synchronisation points so that we can pause the flush/reclaim threads at just
the right points. Doing so, I was able to effectively stop the flush process at
the point that the base table has been flushed/reclaimed, but the index table
is still in the flushing state.
Accessing the contents of the index memtable at this point does not seem to
cause a problem, whilst reading from the (now reclaimed) base memtable results
in a segfault as expected.
One piece of good news for this is that the switching and reclaiming have been
re-ordered in trunk by CASSANDRA-12358, such that no reclamation is performed
until all memtables have been switched.
[~cam1982], as we're not any closer to a repro/test right now, do you have a
dev/test cluster where you can try running your workload on latest trunk?
> Segfault reading secondary index
> --------------------------------
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
> Reporter: Cameron Zemek
> Assignee: Sam Tunnicliffe
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x00007fd7354749b7 [0x00007fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x00007fd7351b35b8 [0x00007fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x00007fd736404dd8 [0x00007fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x00007fd736e09638 [0x00007fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x00007fd736e01a48 [0x00007fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x00007fd736417c1c [0x00007fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x00007fd736541e6c [0x00007fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x00007fd7357957b4 [0x00007fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x00007fd735938d8c [0x00007fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
