[
https://issues.apache.org/jira/browse/CASSANDRA-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15484088#comment-15484088
]
Cameron Zemek commented on CASSANDRA-12590:
-------------------------------------------
Found more occurrences of the Last written key exceptions.
{noformat}
Sep 06 18:26:12 ip-10-222-104-36.ec2.internal cassandra[27225]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000000000000000000000000000000000000000000000000000000000000000,
336a4d744647435a375035456c55754a4e786e4158366b783236656768386d337576796d61477946)
Sep 07 11:14:23 ip-10-222-104-36.ec2.internal cassandra[27225]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000000000000000000000000000000000000000000000000000000000000000,
334f67714e65494a6545736b674b6f41613156304d4a747a52314f70376b74534548647333584944)
Sep 08 05:21:44 ip-10-222-104-36.ec2.internal cassandra[27225]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000000000000000000000000000000000000000000000000000000000000000,
303634483949417853652d4e67594e6577384d3976454451424a53614a754845304a7766536b6d56)
Sep 08 05:32:04 ip-10-222-104-36.ec2.internal cassandra[27225]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(0034435f42000000e400000000000090f100000000302e0284527f000070070484527f0000d02fa9,
4d34435f427762664e476f5a324e32474a41544776734247736a3562412d70584749585f74693876)
Sep 08 19:01:57 ip-10-222-104-36.ec2.internal cassandra[13874]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(000000000000000000000059715f696347654b714f53474137356b54504a5039654643656c6c574e,
6b456d354830534278464d59715f696347654b714f53474137356b54504a5039654643656c6c574e)
Sep 08 19:08:33 ip-10-222-104-36.ec2.internal cassandra[13874]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000000000000000000000000000000000000000000000000000000000000000,
2d4332766e7161713748577259634833707634614831624f3031394f67384d5971334e4e38466e54)
Sep 09 06:09:10 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(0000000000000000000000785a375f39746153694b41676b376e425834354f473358504437765a61,
34505339496f345a714b51785a375f39746153694b41676b376e425834354f473358504437765a61)
Sep 09 06:30:32 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(000000564f76537a4e78705267697875395a616767776f785549455769704b4f65485f64594f4d52,
4c4867564f76537a4e78705267697875395a616767776f785549455769704b4f65485f64594f4d52)
Sep 10 05:41:22 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000000000000000000000000000000000000000000000000000000000000000,
3139374e61423059564e7646736f524e3949667632754c74526d48596943594a7373457263796a32)
Sep 10 05:52:01 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000000000000000000000000000000000000000000000000000000000000000,
2d417a4b35354d6d5a794f41347062786431556d513753324534414662746d7236732d336b64444f)
Sep 10 17:59:23 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000000000000000000000000000000000000000000000000000000000000000,
4557694571644d2d5a684669352d524152675a45642d645951316c7056415861444139576161345f)
Sep 10 17:59:23 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000000000000000000000000000000000000000000000000000000000000000,
5a424776594e454137756c5a4741634e44646b325449684a4a424677665053575357443875507868)
Sep 11 20:00:09 ip-10-222-104-36.ec2.internal cassandra[21970]: WARN
o.a.c.i.s.format.big.BigTableWriter BigTableWriter::beforeAppend
(00000000000000000000000000000000000000000000000000000000000000000000000000000000,
3559367076494531586a3853366d42527a6b6351422d62544c4b4f715a71427049574a6869565148)
{noformat}
So I think the issue is LocalToken uses the ByteBuffer of the Cell which gets
reclaimed/recycled. And its only an issue for secondary indexes since
LocalToken is the only Token that uses an allocated data structure. When
inserted into memtable:
{code:title=Memtable.java|borderStyle=solid}
long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group
opGroup)
{
// .. omitted for brevity
final DecoratedKey cloneKey = allocator.clone(update.partitionKey(), opGroup);
{code}
which results in calling
{code:title=MemtableBufferAllocator.java|borderStyle=solid}
public DecoratedKey clone(DecoratedKey key, OpOrder.Group writeOp)
{
return new BufferDecoratedKey(key.getToken(),
allocator(writeOp).clone(key.getKey()));
}
{code}
So the partition key is cloned when added to memtable but the token is not.
Possible fix would be have LocalPartitioner to allocate its own storage for the
token? eg.
{noformat}
diff --git a/src/java/org/apache/cassandra/dht/LocalPartitioner.java
b/src/java/org/apache/cassandra/dht/LocalPartitioner.java
index f9421c5..10a806b 100644
--- a/src/java/org/apache/cassandra/dht/LocalPartitioner.java
+++ b/src/java/org/apache/cassandra/dht/LocalPartitioner.java
@@ -56,7 +56,7 @@ public class LocalPartitioner implements IPartitioner
public LocalToken getToken(ByteBuffer key)
{
- return new LocalToken(key);
+ return new LocalToken(ByteBufferUtil.clone(key));
}
public LocalToken getRandomToken()
{noformat}
> Segfault reading secondary index
> --------------------------------
>
> Key: CASSANDRA-12590
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12590
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: Occurs on Cassandra 3.5 and 3.7
> Reporter: Cameron Zemek
>
> Getting segfaults when reading secondary index as follows:
> J 9272 C2
> org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(Lorg/apache/cassandra/dht/Token;)I
> (53 bytes) @ 0x00007fd7354749b7 [0x00007fd735474840+0x177]
> J 5661 C2 org.apache.cassandra.db.DecoratedKey.compareTo(Ljava/lang/Object;)I
> (9 bytes) @ 0x00007fd7351b35b8 [0x00007fd7351b3440+0x178]
> J 14205 C2
> java.util.concurrent.ConcurrentSkipListMap.doGet(Ljava/lang/Object;)Ljava/lang/Object;
> (142 bytes) @ 0x00007fd736404dd8 [0x00007fd736404cc0+0x118]
> J 17764 C2
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(Lorg/apache/cassandra/db/ColumnFamilyStore;)Lorg/apache/cassandra/db/rows/UnfilteredRowIterator;
> (635 bytes) @ 0x00007fd736e09638 [0x00007fd736e08720+0xf18]
> J 17808 C2
> org.apache.cassandra.index.internal.CassandraIndexSearcher.search(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (68 bytes) @ 0x00007fd736e01a48 [0x00007fd736e012a0+0x7a8]
> J 14217 C2
> org.apache.cassandra.db.ReadCommand.executeLocally(Lorg/apache/cassandra/db/ReadExecutionController;)Lorg/apache/cassandra/db/partitions/UnfilteredPartitionIterator;
> (219 bytes) @ 0x00007fd736417c1c [0x00007fd736416fa0+0xc7c]
> J 14585 C2
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow()V
> (337 bytes) @ 0x00007fd736541e6c [0x00007fd736541d60+0x10c]
> J 14584 C2 org.apache.cassandra.service.StorageProxy$DroppableRunnable.run()V
> (48 bytes) @ 0x00007fd7357957b4 [0x00007fd735795760+0x54]
> J 9648% C2 org.apache.cassandra.concurrent.SEPWorker.run()V (253 bytes) @
> 0x00007fd735938d8c [0x00007fd7359356e0+0x36ac]
> Which I have translated to the codepath:
> org.apache.cassandra.dht.LocalPartitioner (Line 139)
> org.apache.cassandra.db.DecoratedKey (Line 85)
> java.util.concurrent.ConcurrentSkipListMap (Line 794)
> org.apache.cassandra.db.SinglePartitionReadCommand (Line 498)
> org.apache.cassandra.index.internal.CassandraIndexSearcher (Line 60)
> org.apache.cassandra.db.ReadCommand (Line 367)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
