[
https://issues.apache.org/jira/browse/CASSANDRA-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954682#comment-15954682
]
Igor Novgorodov commented on CASSANDRA-13403:
---------------------------------------------
> if you rebuild the index, does it cause the same result?
Rebuilding (nodetool rebuild_index) does not help.
Dropping and recreating index, however, fixes the issue right away - the index
by query returns all the required rows.
> Also, does the problem persist or if you wait for some time, it goes away?
It persists. I even tried rolling restart of all nodes in cluster one by one -
it does not help. Only index drop\create.
> Could you also find log entries where it says the index got opened when you
> were rebuilding it
Here are the entries from debug.log, but as i said the rebuild does not help:
{code}
INFO [RMI TCP Connection(35)-127.0.0.1] 2017-04-04 09:54:07,270
ColumnFamilyStore.java:806 - User Requested secondary index re-build for
cservice_novg/bulks_recipients indexes: bulk_recipients_bulk_id
DEBUG [RMI TCP Connection(35)-127.0.0.1] 2017-04-04 09:54:07,272
ColumnFamilyStore.java:899 - Enqueuing flush of IndexInfo: 0.406KiB (0%)
on-heap, 0.000KiB (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:25] 2017-04-04 09:54:07,279
Memtable.java:461 - Writing Memtable-IndexInfo@825120901(0.066KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:25] 2017-04-04 09:54:07,279
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-241-big-Data.db
(0.058KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=903461)
DEBUG [MemtableFlushWriter:25] 2017-04-04 09:54:07,299
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-241-big-Data.db')]
(1 sstables, 4.925KiB), biggest 4.925KiB, smallest 4.925KiB
DEBUG [CompactionExecutor:1885] 2017-04-04 09:54:07,299 CompactionTask.java:155
- Compacting (7f833130-1903-11e7-b846-a7a5e88b576b)
[/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-240-big-Data.db:level=0,
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-241-big-Data.db:level=0,
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-239-big-Data.db:level=0,
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-238-big-Data.db:level=0,
]
INFO [RMI TCP Connection(35)-127.0.0.1] 2017-04-04 09:54:07,299
SecondaryIndexManager.java:365 - Submitting index build of
bulk_recipients_bulk_id for data in
BigTableReader(path='/var/lib/cassandra/data/cservice_novg/bulks_recipients-d289270016cd11e7a152a5458c06987d/mc-12-big-Data.db')
INFO [CompactionExecutor:1869] 2017-04-04 09:54:07,300
PerSSTableIndexWriter.java:279 - Scheduling index flush to
/var/lib/cassandra/data/cservice_novg/bulks_recipients-d289270016cd11e7a152a5458c06987d/mc-12-big-SI_bulk_recipients_bulk_id.db
INFO [SASI-General:4] 2017-04-04 09:54:07,305 PerSSTableIndexWriter.java:330 -
Index flush to
/var/lib/cassandra/data/cservice_novg/bulks_recipients-d289270016cd11e7a152a5458c06987d/mc-12-big-SI_bulk_recipients_bulk_id.db
took 4 ms.
INFO [CompactionExecutor:1869] 2017-04-04 09:54:07,305 DataTracker.java:152 -
SSTableIndex.open(column: bulk_id, minTerm:
69135a30-531d-43b0-a93a-f751875fc14d, maxTerm:
69135a30-531d-43b0-a93a-f751875fc14d, minKey: 73533731612, maxKey: 78225387251,
sstable:
BigTableReader(path='/var/lib/cassandra/data/cservice_novg/bulks_recipients-d289270016cd11e7a152a5458c06987d/mc-12-big-Data.db'))
INFO [RMI TCP Connection(35)-127.0.0.1] 2017-04-04 09:54:07,306
SecondaryIndexManager.java:385 - Index build of bulk_recipients_bulk_id complete
DEBUG [RMI TCP Connection(35)-127.0.0.1] 2017-04-04 09:54:07,307
ColumnFamilyStore.java:899 - Enqueuing flush of IndexInfo: 0.406KiB (0%)
on-heap, 0.000KiB (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:26] 2017-04-04 09:54:07,315
Memtable.java:461 - Writing Memtable-IndexInfo@723938295(0.066KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:26] 2017-04-04 09:54:07,315
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-242-big-Data.db
(0.057KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=903551)
DEBUG [MemtableFlushWriter:26] 2017-04-04 09:54:07,342
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-242-big-Data.db')]
(1 sstables, 4.907KiB), biggest 4.907KiB, smallest 4.907KiB
DEBUG [CompactionExecutor:1885] 2017-04-04 09:54:07,369 CompactionTask.java:255
- Compacted (7f833130-1903-11e7-b846-a7a5e88b576b) 4 sstables to
[/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-243-big,]
to level=0. 0.366KiB to 0.177KiB (~48% of original) in 69ms. Read Throughput
= 5.302KiB/s, Write Throughput = 2.559KiB/s, Row Throughput = ~10/s. 7 total
partitions merged to 4. Partition merge counts were {1:3, 4:1, }
{code}
Here are the logs when i drop\create index:
{code}
INFO [Native-Transport-Requests-1] 2017-04-04 09:56:39,383
MigrationManager.java:414 - Update table 'cservice_novg/bulks_recipients' From
org.apache.cassandra.config.CFMetaData@6811c286[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[org.apache.cassandra.schema.IndexMetadata@42a655b4[id=2b973602-6729-3181-9bf5-0d3e05130669,name=bulk_recipients_bulk_id,kind=CUSTOM,options={class_name=org.apache.cassandra.index.sasi.SASIIndex,
target=bulk_id}]]] To
org.apache.cassandra.config.CFMetaData@31086dc4[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[]]
DEBUG [MigrationStage:1] 2017-04-04 09:56:39,388 ColumnFamilyStore.java:899 -
Enqueuing flush of keyspaces: 0.503KiB (0%) on-heap, 0.000KiB (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:27] 2017-04-04 09:56:39,398
Memtable.java:461 - Writing Memtable-keyspaces@1596439942(0.140KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:27] 2017-04-04 09:56:39,398
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-351-big-Data.db
(0.114KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=905205)
DEBUG [MemtableFlushWriter:27] 2017-04-04 09:56:39,422
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-351-big-Data.db')]
(1 sstables, 5.062KiB), biggest 5.062KiB, smallest 5.062KiB
DEBUG [MigrationStage:1] 2017-04-04 09:56:39,422 ColumnFamilyStore.java:899 -
Enqueuing flush of tables: 2.192KiB (0%) on-heap, 0.000KiB (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:28] 2017-04-04 09:56:39,430
Memtable.java:461 - Writing Memtable-tables@2010310212(0.665KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:28] 2017-04-04 09:56:39,430
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mc-338-big-Data.db
(0.428KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=905205)
DEBUG [MemtableFlushWriter:28] 2017-04-04 09:56:39,443
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mc-338-big-Data.db')]
(1 sstables, 6.694KiB), biggest 6.694KiB, smallest 6.694KiB
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:39,443 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in columns
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:39,443 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in dropped_columns
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:39,443 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in triggers
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:39,443 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in views
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:39,444 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in types
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:39,444 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in functions
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:39,444 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in aggregates
DEBUG [MigrationStage:1] 2017-04-04 09:56:39,444 ColumnFamilyStore.java:899 -
Enqueuing flush of indexes: 0.383KiB (0%) on-heap, 0.000KiB (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:27] 2017-04-04 09:56:39,449
Memtable.java:461 - Writing Memtable-indexes@1373653010(0.066KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:27] 2017-04-04 09:56:39,449
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-163-big-Data.db
(0.073KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=905205)
DEBUG [MemtableFlushWriter:27] 2017-04-04 09:56:39,462
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-163-big-Data.db')]
(1 sstables, 4.962KiB), biggest 4.962KiB, smallest 4.962KiB
DEBUG [MigrationStage:1] 2017-04-04 09:56:39,470 CFMetaData.java:768 - applying
org.apache.cassandra.config.CFMetaData@2c612c51[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[]] to
org.apache.cassandra.config.CFMetaData@6811c286[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[org.apache.cassandra.schema.IndexMetadata@42a655b4[id=2b973602-6729-3181-9bf5-0d3e05130669,name=bulk_recipients_bulk_id,kind=CUSTOM,options={class_name=org.apache.cassandra.index.sasi.SASIIndex,
target=bulk_id}]]]
DEBUG [MigrationStage:1] 2017-04-04 09:56:39,471 CFMetaData.java:793 -
application result is
org.apache.cassandra.config.CFMetaData@6811c286[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[]]
DEBUG [MigrationStage:1] 2017-04-04 09:56:39,473 ColumnFamilyStore.java:899 -
Enqueuing flush of IndexInfo: 0.406KiB (0%) on-heap, 0.000KiB (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:28] 2017-04-04 09:56:39,480
Memtable.java:461 - Writing Memtable-IndexInfo@2031425230(0.066KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:28] 2017-04-04 09:56:39,480
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-244-big-Data.db
(0.058KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=905299)
DEBUG [MemtableFlushWriter:28] 2017-04-04 09:56:39,494
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-244-big-Data.db')]
(1 sstables, 4.925KiB), biggest 4.925KiB, smallest 4.925KiB
DEBUG [MigrationStage:1] 2017-04-04 09:56:39,495 ColumnFamilyStore.java:899 -
Enqueuing flush of bulks_recipients: 0.000KiB (0%) on-heap, 0.000KiB (0%)
off-heap
DEBUG [MigrationStage:1] 2017-04-04 09:56:39,503 MigrationManager.java:560 -
Gossiping my schema version dd74640a-c175-3e0c-8279-8bebf552377a
DEBUG [GossipStage:1] 2017-04-04 09:56:40,409 MigrationManager.java:95 - Not
pulling schema because versions match or shouldPullSchemaFrom returned false
DEBUG [GossipStage:1] 2017-04-04 09:56:40,409 MigrationManager.java:95 - Not
pulling schema because versions match or shouldPullSchemaFrom returned false
DEBUG [GossipStage:1] 2017-04-04 09:56:40,409 MigrationManager.java:95 - Not
pulling schema because versions match or shouldPullSchemaFrom returned false
DEBUG [GossipStage:1] 2017-04-04 09:56:40,409 MigrationManager.java:95 - Not
pulling schema because versions match or shouldPullSchemaFrom returned false
INFO [Native-Transport-Requests-1] 2017-04-04 09:56:43,310
MigrationManager.java:414 - Update table 'cservice_novg/bulks_recipients' From
org.apache.cassandra.config.CFMetaData@6811c286[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[]] To
org.apache.cassandra.config.CFMetaData@6cbde00f[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[org.apache.cassandra.schema.IndexMetadata@1649f522[id=2b973602-6729-3181-9bf5-0d3e05130669,name=bulk_recipients_bulk_id,kind=CUSTOM,options={class_name=org.apache.cassandra.index.sasi.SASIIndex,
target=bulk_id}]]]
DEBUG [MigrationStage:1] 2017-04-04 09:56:43,313 ColumnFamilyStore.java:899 -
Enqueuing flush of keyspaces: 0.503KiB (0%) on-heap, 0.000KiB (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:27] 2017-04-04 09:56:43,321
Memtable.java:461 - Writing Memtable-keyspaces@1440732918(0.140KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:27] 2017-04-04 09:56:43,322
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-352-big-Data.db
(0.114KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=906786)
DEBUG [MemtableFlushWriter:28] 2017-04-04 09:56:43,349
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-352-big-Data.db')]
(1 sstables, 5.062KiB), biggest 5.062KiB, smallest 5.062KiB
DEBUG [CompactionExecutor:1894] 2017-04-04 09:56:43,350 CompactionTask.java:155
- Compacting (dc86af60-1903-11e7-b846-a7a5e88b576b)
[/var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-351-big-Data.db:level=0,
/var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-349-big-Data.db:level=0,
/var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-350-big-Data.db:level=0,
/var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-352-big-Data.db:level=0,
]
DEBUG [MigrationStage:1] 2017-04-04 09:56:43,357 ColumnFamilyStore.java:899 -
Enqueuing flush of tables: 2.192KiB (0%) on-heap, 0.000KiB (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:28] 2017-04-04 09:56:43,368
Memtable.java:461 - Writing Memtable-tables@2099940660(0.665KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:28] 2017-04-04 09:56:43,368
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mc-339-big-Data.db
(0.428KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=906794)
DEBUG [MemtableFlushWriter:27] 2017-04-04 09:56:43,383
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mc-339-big-Data.db')]
(1 sstables, 6.694KiB), biggest 6.694KiB, smallest 6.694KiB
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:43,383 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in columns
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:43,383 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in dropped_columns
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:43,383 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in triggers
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:43,383 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in views
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:43,384 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in types
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:43,384 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in functions
DEBUG [MemtablePostFlush:14] 2017-04-04 09:56:43,384 ColumnFamilyStore.java:954
- forceFlush requested but everything is clean in aggregates
DEBUG [MigrationStage:1] 2017-04-04 09:56:43,384 ColumnFamilyStore.java:899 -
Enqueuing flush of indexes: 0.650KiB (0%) on-heap, 0.000KiB (0%) off-heap
DEBUG [PerDiskMemtableFlushWriter_0:27] 2017-04-04 09:56:43,390
Memtable.java:461 - Writing Memtable-indexes@282935849(0.186KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:27] 2017-04-04 09:56:43,391
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-164-big-Data.db
(0.164KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=906794)
DEBUG [CompactionExecutor:1894] 2017-04-04 09:56:43,414 CompactionTask.java:255
- Compacted (dc86af60-1903-11e7-b846-a7a5e88b576b) 4 sstables to
[/var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-353-big,]
to level=0. 0.775KiB to 0.409KiB (~52% of original) in 63ms. Read Throughput
= 12.278KiB/s, Write Throughput = 6.479KiB/s, Row Throughput = ~19/s. 13 total
partitions merged to 10. Partition merge counts were {1:9, 4:1, }
DEBUG [MemtableFlushWriter:28] 2017-04-04 09:56:43,432
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-164-big-Data.db')]
(1 sstables, 5.249KiB), biggest 5.249KiB, smallest 5.249KiB
DEBUG [CompactionExecutor:1887] 2017-04-04 09:56:43,433 CompactionTask.java:155
- Compacting (dc933280-1903-11e7-b846-a7a5e88b576b)
[/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-161-big-Data.db:level=0,
/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-164-big-Data.db:level=0,
/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-163-big-Data.db:level=0,
/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-162-big-Data.db:level=0,
]
DEBUG [MigrationStage:1] 2017-04-04 09:56:43,441 CFMetaData.java:768 - applying
org.apache.cassandra.config.CFMetaData@28896a08[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[org.apache.cassandra.schema.IndexMetadata@1e2da7a6[id=2b973602-6729-3181-9bf5-0d3e05130669,name=bulk_recipients_bulk_id,kind=CUSTOM,options={class_name=org.apache.cassandra.index.sasi.SASIIndex,
target=bulk_id}]]] to
org.apache.cassandra.config.CFMetaData@6811c286[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[]]
DEBUG [MigrationStage:1] 2017-04-04 09:56:43,442 CFMetaData.java:793 -
application result is
org.apache.cassandra.config.CFMetaData@6811c286[cfId=d2892700-16cd-11e7-a152-a5458c06987d,ksName=cservice_novg,cfName=bulks_recipients,flags=[COMPOUND],params=TableParams{comment=,
read_repair_chance=0.0, dclocal_read_repair_chance=0.1,
bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000,
default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128,
max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' :
'ALL', 'rows_per_partition' : '100'},
compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,
options={max_threshold=32, min_threshold=4}},
compression=org.apache.cassandra.schema.CompressionParams@f9582726,
extensions={},
cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[]
| [datetime_final datetime_sent request_id
status]],partitionKeyColumns=[recipient],clusteringColumns=[bulk_id],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[recipient,
request_id, bulk_id, status, datetime_final,
datetime_sent],droppedColumns={},triggers=[],indexes=[org.apache.cassandra.schema.IndexMetadata@1e2da7a6[id=2b973602-6729-3181-9bf5-0d3e05130669,name=bulk_recipients_bulk_id,kind=CUSTOM,options={class_name=org.apache.cassandra.index.sasi.SASIIndex,
target=bulk_id}]]]
DEBUG [MigrationStage:1] 2017-04-04 09:56:43,443 ColumnFamilyStore.java:899 -
Enqueuing flush of IndexInfo: 0.406KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO [CompactionExecutor:1895] 2017-04-04 09:56:43,443
PerSSTableIndexWriter.java:279 - Scheduling index flush to
/var/lib/cassandra/data/cservice_novg/bulks_recipients-d289270016cd11e7a152a5458c06987d/mc-12-big-SI_bulk_recipients_bulk_id.db
INFO [SASI-General:5] 2017-04-04 09:56:43,447 PerSSTableIndexWriter.java:330 -
Index flush to
/var/lib/cassandra/data/cservice_novg/bulks_recipients-d289270016cd11e7a152a5458c06987d/mc-12-big-SI_bulk_recipients_bulk_id.db
took 3 ms.
INFO [CompactionExecutor:1895] 2017-04-04 09:56:43,448 DataTracker.java:152 -
SSTableIndex.open(column: bulk_id, minTerm:
69135a30-531d-43b0-a93a-f751875fc14d, maxTerm:
69135a30-531d-43b0-a93a-f751875fc14d, minKey: 73533731612, maxKey: 78225387251,
sstable:
BigTableReader(path='/var/lib/cassandra/data/cservice_novg/bulks_recipients-d289270016cd11e7a152a5458c06987d/mc-12-big-Data.db'))
DEBUG [PerDiskMemtableFlushWriter_0:28] 2017-04-04 09:56:43,451
Memtable.java:461 - Writing Memtable-IndexInfo@942152949(0.066KiB serialized
bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
(min(-9223372036854775808), max(9223372036854775807)]
DEBUG [PerDiskMemtableFlushWriter_0:28] 2017-04-04 09:56:43,451
Memtable.java:490 - Completed flushing
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-245-big-Data.db
(0.057KiB) for commitlog position CommitLogPosition(segmentId=1491255073095,
position=907448)
DEBUG [MemtableFlushWriter:27] 2017-04-04 09:56:43,465
ColumnFamilyStore.java:1197 - Flushed to
[BigTableReader(path='/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-245-big-Data.db')]
(1 sstables, 4.907KiB), biggest 4.907KiB, smallest 4.907KiB
DEBUG [CompactionExecutor:1896] 2017-04-04 09:56:43,465 CompactionTask.java:155
- Compacting (dc983b90-1903-11e7-b846-a7a5e88b576b)
[/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-243-big-Data.db:level=0,
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-242-big-Data.db:level=0,
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-244-big-Data.db:level=0,
/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-245-big-Data.db:level=0,
]
DEBUG [MigrationStage:1] 2017-04-04 09:56:43,465 ColumnFamilyStore.java:899 -
Enqueuing flush of bulks_recipients: 0.000KiB (0%) on-heap, 0.000KiB (0%)
off-heap
DEBUG [MigrationStage:1] 2017-04-04 09:56:43,470 MigrationManager.java:560 -
Gossiping my schema version a3b4014a-a1d2-3aba-a666-ed11072ed3a4
DEBUG [CompactionExecutor:1887] 2017-04-04 09:56:43,492 CompactionTask.java:255
- Compacted (dc933280-1903-11e7-b846-a7a5e88b576b) 4 sstables to
[/var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/mc-165-big,]
to level=0. 0.977KiB to 0.587KiB (~60% of original) in 58ms. Read Throughput
= 16.610KiB/s, Write Throughput = 9.982KiB/s, Row Throughput = ~18/s. 10 total
partitions merged to 7. Partition merge counts were {1:6, 4:1, }
DEBUG [CompactionExecutor:1896] 2017-04-04 09:56:43,527 CompactionTask.java:255
- Compacted (dc983b90-1903-11e7-b846-a7a5e88b576b) 4 sstables to
[/var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-246-big,]
to level=0. 0.369KiB to 0.175KiB (~47% of original) in 60ms. Read Throughput
= 6.064KiB/s, Write Throughput = 2.872KiB/s, Row Throughput = ~10/s. 7 total
partitions merged to 4. Partition merge counts were {1:3, 4:1, }
DEBUG [GossipStage:1] 2017-04-04 09:56:43,772 MigrationManager.java:95 - Not
pulling schema because versions match or shouldPullSchemaFrom returned false
DEBUG [GossipStage:1] 2017-04-04 09:56:43,772 MigrationManager.java:95 - Not
pulling schema because versions match or shouldPullSchemaFrom returned false
DEBUG [GossipStage:1] 2017-04-04 09:56:43,772 MigrationManager.java:95 - Not
pulling schema because versions match or shouldPullSchemaFrom returned false
{code}
> an ls output on data directory?
Here's the tree output of a keyspace when the index is fine:
https://pastebin.com/zAggMJ6i
And here's after the repair when it's broken: https://pastebin.com/bfTcRP02
Thanks!
> nodetool repair breaks SASI index
> ---------------------------------
>
> Key: CASSANDRA-13403
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13403
> Project: Cassandra
> Issue Type: Bug
> Components: sasi
> Environment: 3.10
> Reporter: Igor Novgorodov
>
> I've got table:
> {code}
> CREATE TABLE cservice.bulks_recipients (
> recipient text,
> bulk_id uuid,
> datetime_final timestamp,
> datetime_sent timestamp,
> request_id uuid,
> status int,
> PRIMARY KEY (recipient, bulk_id)
> ) WITH CLUSTERING ORDER BY (bulk_id ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> CREATE CUSTOM INDEX bulk_recipients_bulk_id ON cservice.bulks_recipients
> (bulk_id) USING 'org.apache.cassandra.index.sasi.SASIIndex';
> {code}
> There are 11 rows in it:
> {code}
> > select * from bulks_recipients;
> ...
> (11 rows)
> {code}
> Let's query by index (all rows have the same *bulk_id*):
> {code}
> > select * from bulks_recipients where bulk_id =
> > baa94815-e276-4ca4-adda-5b9734e6c4a5;
> >
> ...
> (11 rows)
> {code}
> Ok, everything is fine.
> Now i'm doing *nodetool repair --partitioner-range --job-threads 4 --full* on
> each node in cluster sequentially.
> After it finished:
> {code}
> > select * from bulks_recipients where bulk_id =
> > baa94815-e276-4ca4-adda-5b9734e6c4a5;
> ...
> (2 rows)
> {code}
> Only two rows.
> While the rows are actually there:
> {code}
> > select * from bulks_recipients;
> ...
> (11 rows)
> {code}
> If i issue an incremental repair on a random node, i can get like 7 rows
> after index query.
> Dropping index and recreating it fixes the issue. Is it a bug or am i doing
> the repair the wrong way?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
