Wondering if anyone has encountered the same...
Full story and stacktraces below, short version is that creating a SASI
index fails for me when running a 3.10-SNAPSHOT build. One caveat: creating
the index on an empty table doesn't fail; however, soon after I start
pumping data into the table similar problems occur.
I created CASSANDRA-12877 for this, but am beginning to suspect it might be
related to CASSANDRA-11990. The thing that's throwing me is that I can't
seem to duplicate this with a simple test table.
Background:
Ended up building/loading a 3.10-SNAPSHOT to try to get past
CASSANDRA-11670, CASSANDRA-12223, and CASSANDRA-12689.
1) built/installed 3.10-SNAPSHOT from git branch cassandra-3.X
2) created keyspace (SimpleStrategy, RF 1)
3) created table: (simplified version below, many more valX columns present)
CREATE TABLE test_table (
id1 text,
id2 text,
id3 date,
id4 timestamp,
id5 text,
val1 text,
val2 text,
val3 text,
task_id text,
val4 text,
val5 text,
PRIMARY KEY ((id1, id2), id3, id4, id5)
) WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id5 ASC)
4) created materialized view:
CREATE MATERIALIZED VIEW test_table_by_task_id AS
SELECT *
FROM test_table
WHERE id1 IS NOT NULL AND id2 IS NOT NULL AND id3 IS NOT NULL AND id4
IS NOT NULL AND id5 IS NOT NULL AND task_id IS NOT NULL
PRIMARY KEY (task_id, id3, id4, id1, id2, id5)
WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id1 ASC, id2 ASC, id5 ASC)
5) inserted 27 million "rows" (i.e., unique values for id5)
6) create index attempt
create custom index idx_test_table_val5 on test_table(val5) using
'org.apache.cassandra.index.sasi.SASIIndex';
7) no error in cqlsh, but system.log shows many of the following:
INFO [SASI-General:1] 2016-11-04 13:46:47,578
PerSSTableIndexWriter.java:277 - Flushed index segment
/mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-149-big-SI_idx_test_table_val5.db_0,
took 869 ms.
ERROR [SASI-General:1] 2016-11-04 13:46:47,584 CassandraDaemon.java:229 -
Exception in thread Thread[SASI-General:1,5,main]
java.lang.AssertionError: cannot have more than 8 overflow collisions per
leaf, but had: 12
at
org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.createOverflowEntry(AbstractTokenTreeBuilder.java:357)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.createEntry(AbstractTokenTreeBuilder.java:346)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder$DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder$Leaf.serialize(AbstractTokenTreeBuilder.java:306)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder.write(AbstractTokenTreeBuilder.java:90)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableLevel.flush(OnDiskIndexBuilder.java:446)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$MutableLevel.add(OnDiskIndexBuilder.java:433)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.addTerm(OnDiskIndexBuilder.java:207)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:293)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:258)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.finish(OnDiskIndexBuilder.java:241)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
As well as some of these:
ERROR [CompactionExecutor:3] 2016-11-04 13:49:13,142 DataTracker.java:168 -
Can't open index file at
/mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06f0/mc-300-big-SI_idx_test_table_val5.db,
skipping.
java.lang.IllegalArgumentException: position: 3472329188772431788, limit:
8180
at
org.apache.cassandra.index.sasi.utils.MappedBuffer.position(MappedBuffer.java:106)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.conf.DataTracker.getIndexes(DataTracker.java:150)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.SASIIndexBuilder.completeSSTable(SASIIndexBuilder.java:156)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.index.sasi.SASIIndexBuilder.build(SASIIndexBuilder.java:125)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1583)
[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_101]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
