[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488091#comment-15488091
]
Sam Tunnicliffe commented on CASSANDRA-12590:
---------------------------------------------
I think your analysis is on the right path, but so far I'm unable to square the
circle to figure out exactly what's causing the issue. It's true that
LocalToken references the ByteBuffer of the indexed Cell from the base table,
which is more than likely problematic in itself. On top of that, during a
flush, each memtable is released/discarded individually. Because that
ByteBuffer was allocated by the base table's memtable, it may be freed before
the index memtable (which is still holding a reference to it).
However, Regions (i.e. slabs) created by the allocator are not recycled, so I
don't think there's anything explicity reclaiming/re-using buffers obtained
from those. However, memory allocated offheap is freed by the OS, so
theoretically it could be being re-used by subsequent allocations and
corrupting the (still referenced) contents.
A couple of questions:
What {{memtable_allocation_type}} are you setting in cassandra.yaml?
Is this easily reproducible for you, if so what does the workload look like?
The explanation seems plausible, but it would feel much happier if I could
repro or preferably come up with a unit or dtest.
Lastly, regarding the proposed solution, it's probably not necessary to clone
for every instance of LocalToken. Aside from internal indexes, system tables
also use LT, so this may have a negative impact in terms of generating
unnecessary garbage. It seems to me that it's the fact that those buffers are
referenced from
multiple Memtables which is the real issue. So in the first instance, I propose
cloning in the index instead.
{code:title=CassandraIndex.java}
private DecoratedKey getIndexKeyFor(ByteBuffer value)
{
return indexCfs.decorateKey(ByteBufferUtil.clone(value));
}
{code}
CassandraIndexTest & SecondaryIndexTest pass for me with that modification, but
that really says nothing about whether it fixes the actual problem, so a solid
repro or unit test would be very useful.
> Segfault reading secondary index
> --------------------------------
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
> Reporter: Cameron Zemek
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x00007fd7354749b7 [0x00007fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x00007fd7351b35b8 [0x00007fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x00007fd736404dd8 [0x00007fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x00007fd736e09638 [0x00007fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x00007fd736e01a48 [0x00007fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x00007fd736417c1c [0x00007fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x00007fd736541e6c [0x00007fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x00007fd7357957b4 [0x00007fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x00007fd735938d8c [0x00007fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
