[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15701348#comment-15701348
]
Kurt Greaves commented on CASSANDRA-12590:
--
I haven't spent too much time on it but yes, haven't gotten any further yet.
Seems like this may as well be resolved by CASSANDRA-12651 now though.
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>Assignee: Sam Tunnicliffe
>
> Getting segfaults when reading secondary index as follows:
> {code}
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> {code}
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695571#comment-15695571
]
Sam Tunnicliffe commented on CASSANDRA-12590:
-
[~cam1982], I was hoping we'd be able to repro before committing, if not with
a test at least with some manually reproducible steps. I spoke offline with
[~KurtG] a week or 2 back & shared my attempts using byteman, but I guess he's
not been able to get any further than I did. Obviously, if we can't repro but
empirically the fix looks good, then we shouldn't block it for lack of a unit
test.
In the meantime though, take a look at [this
comment|https://issues.apache.org/jira/browse/CASSANDRA-12651?focusedCommentId=15674407=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15674407]
on CASSANDRA-12651. Basically, I think we'll end up committing the fix for
this there (but actually your original suggestion, with the fix in
{{LocalToken}}). That ticket is just waiting review, and I expect to commit it
soon.
The only question is to which version. CASSANDRA-12651 relates to one of the
failing tests that drew a -1 on the most recent 3.10 vote, so I haven't pushed
harder to commit it until we make a call on the 3.10 release. Hopefully, that
should get resolved in the next few days.
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>Assignee: Sam Tunnicliffe
>
> Getting segfaults when reading secondary index as follows:
> {code}
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> {code}
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15695541#comment-15695541
]
Cameron Zemek commented on CASSANDRA-12590:
---
Is there any reason to not accept the following as the patch?
{code:title=CassandraIndex.java|borderStyle=solid}
private DecoratedKey getIndexKeyFor(ByteBuffer value)
{
return indexCfs.decorateKey(ByteBufferUtil.clone(value));
}
{code}
Or do we need a test that demonstrates the issue to get this patch accepted?
Just wondering what todo to get a fix into Cassandra.
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>Assignee: Sam Tunnicliffe
>
> Getting segfaults when reading secondary index as follows:
> {code}
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> {code}
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536091#comment-15536091
]
Kurt Greaves commented on CASSANDRA-12590:
--
No row caching options set in the cassandra.yaml. Caching keys only for the
table.
Cameron's on leave at the moment and not really sure what he was getting at
regarding the workload however I can tell you we are updating. We're also
frequently updating the connectionid field, which is the field that causes the
segfault.
[~ifesdjeen] If by sstable flush threshold you mean based off the memtable
cleanup threshold it should be 444MB.
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>Assignee: Sam Tunnicliffe
>
> Getting segfaults when reading secondary index as follows:
> {code}
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> {code}
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517027#comment-15517027
]
Sam Tunnicliffe commented on CASSANDRA-12590:
-
bq.Deployed trunk to one of the test environments and it segfaulted:
Well, that's disappointing.
I'm still trying to figure out a repro case, it's probably not relevant but
what row caching options do you have set in cassandra.yaml and for that
specific table?
When describing the workload, you said:
bq. The timing is generally write, ~10ms, read, ~40ms, then repeat with
slightly different data
How different is the data? Is it an append only workload, just with similar
looking rows, or are you actually updating stuff?
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>Assignee: Sam Tunnicliffe
>
> Getting segfaults when reading secondary index as follows:
> {code}
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> {code}
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514736#comment-15514736
]
Cameron Zemek commented on CASSANDRA-12590:
---
Hey [~beobal],
Deployed trunk to one of the test environments and it segfaulted:
{noformat}
Sep 22 11:14:16 ip-10-222-104-29.ec2.internal cassandra[5296]: # A fatal error
has been detected by the Java Runtime Environment:
Sep 22 11:14:16 ip-10-222-104-29.ec2.internal cassandra[5296]: #
Sep 22 11:14:16 ip-10-222-104-29.ec2.internal cassandra[5296]: # SIGSEGV (0xb)
at pc=0x7fba85c4a3b5, pid=1, tid=140438743848704
Sep 22 11:14:16 ip-10-222-104-29.ec2.internal cassandra[5296]: #
Sep 22 11:14:16 ip-10-222-104-29.ec2.internal cassandra[5296]: # JRE version:
Java(TM) SE Runtime Environment (8.0_91-b14) (build 1.8.0_91-b14)
Sep 22 11:14:16 ip-10-222-104-29.ec2.internal cassandra[5296]: # Java VM: Java
HotSpot(TM) 64-Bit Server VM (25.91-b14 mixed mode linux-amd64 compressed oops)
Sep 22 11:14:16 ip-10-222-104-29.ec2.internal cassandra[5296]: # Problematic
frame:
Sep 22 11:14:16 ip-10-222-104-29.ec2.internal cassandra[5296]: # J 14723 C2
org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
(53 bytes) @ 0x7fba85c4a3b5 [0x7fba85c4a280+0x135]
{noformat}
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>Assignee: Sam Tunnicliffe
>
> Getting segfaults when reading secondary index as follows:
> {code}
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> {code}
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501742#comment-15501742
]
Cameron Zemek commented on CASSANDRA-12590:
---
[~beobal] I will see what I can do.
[~ifesdjeen] Not sure exactly what you mean by sstable flush threshold. Here
related settings from cassandra.yaml:
{noformat}
memtable_allocation_type: offheap_objects
concurrent_writes: 2
key_cache_size_in_mb: 0
memtable_flush_writers: 1
concurrent_compactors: 1
concurrent_reads: 2
commitlog_total_space_in_mb: 1024
file_cache_size_in_mb: '1'
{noformat}
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>Assignee: Sam Tunnicliffe
>
> Getting segfaults when reading secondary index as follows:
> {code}
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> {code}
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495713#comment-15495713
]
Alex Petrov commented on CASSANDRA-12590:
-
Question: what's the sstable flush threshold? I'm trying to evaluate a
hypothesis if this can be related to memtable switching.
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>Assignee: Sam Tunnicliffe
>
> Getting segfaults when reading secondary index as follows:
> {code}
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> {code}
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[ https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15493711#comment-15493711 ] Sam Tunnicliffe commented on CASSANDRA-12590: - I spent some time hacking on a unit test using byteman to inject synchronisation points so that we can pause the flush/reclaim threads at just the right points. Doing so, I was able to effectively stop the flush process at the point that the base table has been flushed/reclaimed, but the index table is still in the flushing state. Accessing the contents of the index memtable at this point does not seem to cause a problem, whilst reading from the (now reclaimed) base memtable results in a segfault as expected. One piece of good news for this is that the switching and reclaiming have been re-ordered in trunk by CASSANDRA-12358, such that no reclamation is performed until all memtables have been switched. [~cam1982], as we're not any closer to a repro/test right now, do you have a dev/test cluster where you can try running your workload on latest trunk? > Segfault reading secondary index > > > Key: CASSANDRA-12590 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12590 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Occurs on Cassandra 3.5 and 3.7 >Reporter: Cameron Zemek >Assignee: Sam Tunnicliffe > > Getting segfaults when reading secondary index as follows: > J 9272 C2 > org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I > (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177] > J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I > (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178] > J 14205 C2 > java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object; > (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118] > J 17764 C2 > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; > (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18] > J 17808 C2 > org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; > (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8] > J 14217 C2 > org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; > (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c] > J 14585 C2 > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V > (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c] > J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V > (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54] > J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @ > 0x7fd735938d8c [0x7fd7359356e0+0x36ac] > Which I have translated to the codepath: > org.apache.cassandra.dht.LocalPartitioner (Line 139) > org.apache.cassandra.db.DecoratedKey (Line 85) > java.util.concurrent.ConcurrentSkipListMap (Line 794) > org.apache.cassandra.db.SinglePartitionReadCommand (Line 498) > org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60) > org.apache.cassandra.db.ReadCommand (Line 367) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489990#comment-15489990
]
Cameron Zemek commented on CASSANDRA-12590:
---
Hey Sam.
It was previously on 3.5 then upgraded to 3.7. So don't have anything on
earlier versions of 3.x
{quote}
How feasible is it for you to deploy a patched version and monitor for
re-occurrences?
{quote}
Yeah I already deployed patched version with my suggested fix some days ago and
since then have had 0 Last written key exceptions and 0 segfaults.
In another environment I have deployed your suggested change (since it
generates less garbage) and so far no occurrences either.
I definitely think we on the right track here. Do you have any ideas on how we
could make a test that reproduces the issue? Do you think the segfault is when
it reads a byte buffer that points to memory that Cassandra no longer owns
cause it freed it?
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[ https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489892#comment-15489892 ] Sam Tunnicliffe commented on CASSANDRA-12590: - bq. memtable_allocation_type: offheap_objects So using direct, off heap memory as opposed to slabs. This feature was dropped from early 3.x versions and only re-introduced in 3.4 by CASSANDRA-9472. Did the cluster(s) exhibiting this problem ever run a 3.x < 3.4 and if so, do you know if the issue manifested then also? bq. unfortunately it is not easy to reproduce :) pretty much what I was expecting. How feasible is it for you to deploy a patched version and monitor for re-occurrences? > Segfault reading secondary index > > > Key: CASSANDRA-12590 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12590 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Occurs on Cassandra 3.5 and 3.7 >Reporter: Cameron Zemek > > Getting segfaults when reading secondary index as follows: > J 9272 C2 > org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I > (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177] > J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I > (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178] > J 14205 C2 > java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object; > (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118] > J 17764 C2 > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; > (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18] > J 17808 C2 > org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; > (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8] > J 14217 C2 > org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; > (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c] > J 14585 C2 > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V > (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c] > J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V > (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54] > J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @ > 0x7fd735938d8c [0x7fd7359356e0+0x36ac] > Which I have translated to the codepath: > org.apache.cassandra.dht.LocalPartitioner (Line 139) > org.apache.cassandra.db.DecoratedKey (Line 85) > java.util.concurrent.ConcurrentSkipListMap (Line 794) > org.apache.cassandra.db.SinglePartitionReadCommand (Line 498) > org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60) > org.apache.cassandra.db.ReadCommand (Line 367) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488957#comment-15488957
]
Cameron Zemek commented on CASSANDRA-12590:
---
{quote}
What memtable_allocation_type are you setting in cassandra.yaml?
{quote}
memtable_allocation_type: offheap_objects
{quote}
Is this easily reproducible for you, if so what does the workload look like?
The explanation seems plausible, but it would feel much happier if I could
repro or preferably come up with a unit or dtest.
{quote}
No unfortunately it is not easy to reproduce. I would love to come up with unit
or dtest if I could figure out how to trigger it on consistent basis.
The workload is about 9 write-then-read cycles per second, each blocking on the
client. The timing is generally write, ~10ms, read, ~40ms, then repeat with
slightly different data.
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488091#comment-15488091
]
Sam Tunnicliffe commented on CASSANDRA-12590:
-
I think your analysis is on the right path, but so far I'm unable to square the
circle to figure out exactly what's causing the issue. It's true that
LocalToken references the ByteBuffer of the indexed Cell from the base table,
which is more than likely problematic in itself. On top of that, during a
flush, each memtable is released/discarded individually. Because that
ByteBuffer was allocated by the base table's memtable, it may be freed before
the index memtable (which is still holding a reference to it).
However, Regions (i.e. slabs) created by the allocator are not recycled, so I
don't think there's anything explicity reclaiming/re-using buffers obtained
from those. However, memory allocated offheap is freed by the OS, so
theoretically it could be being re-used by subsequent allocations and
corrupting the (still referenced) contents.
A couple of questions:
What {{memtable_allocation_type}} are you setting in cassandra.yaml?
Is this easily reproducible for you, if so what does the workload look like?
The explanation seems plausible, but it would feel much happier if I could
repro or preferably come up with a unit or dtest.
Lastly, regarding the proposed solution, it's probably not necessary to clone
for every instance of LocalToken. Aside from internal indexes, system tables
also use LT, so this may have a negative impact in terms of generating
unnecessary garbage. It seems to me that it's the fact that those buffers are
referenced from
multiple Memtables which is the real issue. So in the first instance, I propose
cloning in the index instead.
{code:title=CassandraIndex.java}
private DecoratedKey getIndexKeyFor(ByteBuffer value)
{
return indexCfs.decorateKey(ByteBufferUtil.clone(value));
}
{code}
CassandraIndexTest & SecondaryIndexTest pass for me with that modification, but
that really says nothing about whether it fixes the actual problem, so a solid
repro or unit test would be very useful.
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484088#comment-15484088
]
Cameron Zemek commented on CASSANDRA-12590:
---
Found more occurrences of the Last written key exceptions.
{noformat}
Sep 06 18:26:12 ip-10-222-104-36.ec2.internal cassandra[27225]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(,
336a4d744647435a375035456c55754a4e786e4158366b783236656768386d337576796d61477946)
Sep 07 11:14:23 ip-10-222-104-36.ec2.internal cassandra[27225]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(,
334f67714e65494a6545736b674b6f41613156304d4a747a52314f70376b74534548647333584944)
Sep 08 05:21:44 ip-10-222-104-36.ec2.internal cassandra[27225]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(,
303634483949417853652d4e67594e6577384d3976454451424a53614a754845304a7766536b6d56)
Sep 08 05:32:04 ip-10-222-104-36.ec2.internal cassandra[27225]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(0034435f4200e490f1302e0284527f70070484527fd02fa9,
4d34435f427762664e476f5a324e32474a41544776734247736a3562412d70584749585f74693876)
Sep 08 19:01:57 ip-10-222-104-36.ec2.internal cassandra[13874]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(0059715f696347654b714f53474137356b54504a5039654643656c6c574e,
6b456d354830534278464d59715f696347654b714f53474137356b54504a5039654643656c6c574e)
Sep 08 19:08:33 ip-10-222-104-36.ec2.internal cassandra[13874]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(,
2d4332766e7161713748577259634833707634614831624f3031394f67384d5971334e4e38466e54)
Sep 09 06:09:10 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00785a375f39746153694b41676b376e425834354f473358504437765a61,
34505339496f345a714b51785a375f39746153694b41676b376e425834354f473358504437765a61)
Sep 09 06:30:32 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00564f76537a4e78705267697875395a616767776f785549455769704b4f65485f64594f4d52,
4c4867564f76537a4e78705267697875395a616767776f785549455769704b4f65485f64594f4d52)
Sep 10 05:41:22 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(,
3139374e61423059564e7646736f524e3949667632754c74526d48596943594a7373457263796a32)
Sep 10 05:52:01 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(,
2d417a4b35354d6d5a794f41347062786431556d513753324534414662746d7236732d336b6f)
Sep 10 17:59:23 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(,
4557694571644d2d5a684669352d524152675a45642d645951316c7056415861444139576161345f)
Sep 10 17:59:23 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(,
5a424776594e454137756c5a4741634e44646b325449684a4a424677665053575357443875507868)
Sep 11 20:00:09 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(,
3559367076494531586a3853366d42527a6b6351422d62544c4b4f715a71427049574a6869565148)
{noformat}
So I think the issue is LocalToken uses the ByteBuffer of the Cell which gets
reclaimed/recycled. And its only an issue for secondary indexes since
LocalToken is the only Token that uses an allocated data structure. When
inserted into memtable:
{code:title=Memtable.java|borderStyle=solid}
long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group
opGroup)
{
// .. omitted for brevity
final DecoratedKey cloneKey = allocator.clone(update.partitionKey(), opGroup);
{code}
which results in calling
{code:title=MemtableBufferAllocator.java|borderStyle=solid}
public DecoratedKey clone(DecoratedKey key, OpOrder.Group writeOp)
{
return new
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476024#comment-15476024
]
Cameron Zemek commented on CASSANDRA-12590:
---
With added extra logging
{code:title=BigTableWriter.java|borderStyle=solid}
protected long beforeAppend(DecoratedKey decoratedKey)
{
assert decoratedKey != null : "Keys must not be null"; // empty keys
ARE allowed b/c of indexed column values
if (lastWrittenKey != null && lastWrittenKey.compareTo(decoratedKey) >=
0) {
ByteBuffer key = decoratedKey.getKey();
if (decoratedKey.getToken() instanceof LocalPartitioner.LocalToken)
{
LocalPartitioner.LocalToken token =
(LocalPartitioner.LocalToken) decoratedKey.getToken();
logger.warn("BigTableWriter::beforeAppend (" + (token == null ?
"null" : ByteBufferUtil.bytesToHex(token.getTokenValue())) + ", " + (key ==
null ? "null" : ByteBufferUtil.bytesToHex(key)) + ")");
} else {
logger.warn("BigTableWriter::beforeAppend " + (key == null ?
"null" : ByteBufferUtil.bytesToHex(key)));
}
throw new RuntimeException("Writing into " + getFilename() + " Last
written key " + lastWrittenKey + " >= current key " + decoratedKey);
}
return (lastWrittenKey == null) ? 0 : dataFile.position();
}
{code}
got the following
{noformat}
Sep 06 16:58:54 ip-10-222-128-182.eu-west-1.compute.internal cassandra[7338]:
WARN o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(007657683449634a374136764138703051562d6a48697165793065506354,
7a433144785f615a6e36417657683449634a374136764138703051562d6a48697165793065506354)
Sep 06 18:26:05 ip-10-222-128-139.eu-west-1.compute.internal cassandra[18536]:
WARN o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00608fa5273578666a493271484e48505f514c574b544b486a454c555a6661316164,
4335485f61584e2d4c62593578666a493271484e48505f514c574b544b486a454c555a6661316164)
Sep 08 13:24:01 ip-10-222-128-139.eu-west-1.compute.internal cassandra[18536]:
WARN o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00494b324c5268554f4c546852554143795f31634d724e385f4975797873,
634c5155767a77494a5f32494b324c5268554f4c546852554143795f31634d724e385f4975797873)
{noformat}
So first 11 bytes of the LocalToken ByteBuffer have been corrupted.
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
>
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457248#comment-15457248
]
Kurt Greaves commented on CASSANDRA-12590:
--
Jeff,
Schema is the following, there are also some materialised views I haven't
included but we're fairly sure they are irrelevant. As mentioned by Cameron,
the segfault occurs when reading the deviceconnectionididx secondary index.
From what we've seen the query to the secondary index is fine (i.e we've seen
it crash when querying using '') so we believe it's something to do
with the rows that are returned from the secondary index query.
We're in process of writing a patch that logs all writes to the table so we can
find out what is contained within the row that causes the crash.
CREATE TABLE service.device (
mac text PRIMARY KEY,
bootsequenceid bigint,
connected timeuuid,
connectionid text,
coordinator boolean,
created timeuuid,
disconnected timeuuid,
eventsequence bigint,
externalip text,
groupid text,
householdid text,
householdlocationid text,
hwversion text,
internalip text,
servicehost text,
servicehostuuid timeuuid,
model text,
playerid text,
port text,
roomname text,
satellite boolean,
secure boolean,
sessionid text,
userid text,
swversion text,
swversionmajor int,
swversionminor int,
visible boolean,
websocketcontrolurl text
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
CREATE INDEX deviceconnectionididx ON service.device (connectionid);
> Segfault reading secondary index
>
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
>Reporter: Cameron Zemek
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x7fd735938d8c [0x7fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[ https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456610#comment-15456610 ] Jeff Jirsa commented on CASSANDRA-12590: Can you paste the schema of the table/index? > Segfault reading secondary index > > > Key: CASSANDRA-12590 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12590 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Occurs on Cassandra 3.5 and 3.7 >Reporter: Cameron Zemek > > Getting segfaults when reading secondary index as follows: > J 9272 C2 > org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I > (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177] > J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I > (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178] > J 14205 C2 > java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object; > (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118] > J 17764 C2 > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; > (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18] > J 17808 C2 > org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; > (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8] > J 14217 C2 > org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; > (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c] > J 14585 C2 > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V > (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c] > J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V > (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54] > J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @ > 0x7fd735938d8c [0x7fd7359356e0+0x36ac] > Which I have translated to the codepath: > org.apache.cassandra.dht.LocalPartitioner (Line 139) > org.apache.cassandra.db.DecoratedKey (Line 85) > java.util.concurrent.ConcurrentSkipListMap (Line 794) > org.apache.cassandra.db.SinglePartitionReadCommand (Line 498) > org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60) > org.apache.cassandra.db.ReadCommand (Line 367) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12590) Segfault reading secondary index
[ https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456103#comment-15456103 ] Cameron Zemek commented on CASSANDRA-12590: --- We also experiencing this issue https://issues.apache.org/jira/browse/CASSANDRA-12125 So I believe the issue is in the write path to secondary index memtable. > Segfault reading secondary index > > > Key: CASSANDRA-12590 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12590 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Occurs on Cassandra 3.5 and 3.7 >Reporter: Cameron Zemek > > Getting segfaults when reading secondary index as follows: > J 9272 C2 > org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I > (53 bytes) @ 0x7fd7354749b7 [0x7fd735474840+0x177] > J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I > (9 bytes) @ 0x7fd7351b35b8 [0x7fd7351b3440+0x178] > J 14205 C2 > java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object; > (142 bytes) @ 0x7fd736404dd8 [0x7fd736404cc0+0x118] > J 17764 C2 > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator; > (635 bytes) @ 0x7fd736e09638 [0x7fd736e08720+0xf18] > J 17808 C2 > org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; > (68 bytes) @ 0x7fd736e01a48 [0x7fd736e012a0+0x7a8] > J 14217 C2 > org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator; > (219 bytes) @ 0x7fd736417c1c [0x7fd736416fa0+0xc7c] > J 14585 C2 > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V > (337 bytes) @ 0x7fd736541e6c [0x7fd736541d60+0x10c] > J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V > (48 bytes) @ 0x7fd7357957b4 [0x7fd735795760+0x54] > J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @ > 0x7fd735938d8c [0x7fd7359356e0+0x36ac] > Which I have translated to the codepath: > org.apache.cassandra.dht.LocalPartitioner (Line 139) > org.apache.cassandra.db.DecoratedKey (Line 85) > java.util.concurrent.ConcurrentSkipListMap (Line 794) > org.apache.cassandra.db.SinglePartitionReadCommand (Line 498) > org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60) > org.apache.cassandra.db.ReadCommand (Line 367) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
