[
https://issues.apache.org/jira/browse/CASSANDRA-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604512#comment-15604512
]
Tom van der Woerdt edited comment on CASSANDRA-12125 at 10/25/16 7:40 AM:
--------------------------------------------------------------------------
This issue still exists in 3.0.9.
{noformat}
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [10..11]: 11
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [12..13]: 38
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [14..15]: 23
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [16..17]: 17
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [18..19]: 12
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [20..21]: 9
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [22..29]: 6
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 Validator.java:235 -
Validated 1868 partitions for bb1f9dd0-98db-11e6-ad5a-89fa523ea316. Partition
sizes are:
INFO [AntiEntropyStage:1] 2016-10-23 04:46:59,703 Validator.java:274 - [repair
#bb1f9dd0-98db-11e6-ad5a-89fa523ea316] Sending completed merkle tree to
/10.0.0.6 for myschema.mytable2
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [17232..387341]: 99
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [387342..1097582]: 16
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [1097583..1807823]: 3
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [1807824..2518064]: 7
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [2518065..3959516]: 3
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
CompactionManager.java:1135 - Validation finished in 793 msec, for [repair
#bb1f9dd0-98db-11e6-ad5a-89fa523ea316 on myschema/mytable2,
[(-6682001123294953015,-6681421039287311765]]]
DEBUG [CompactionExecutor:51358] 2016-10-23 04:47:00,499
CompactionTask.java:218 - Compacted (bb4d3d80-98db-11e6-b4a3-2b509f67851f) 2
sstables to
[/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big,]
to level=0. 31,664,366 bytes to 31,653,685 (~99% of original) in 1,592ms =
18.961876MB/s. 0 total partitions merged to 1,888. Partition merge counts
were {1:2063, }
DEBUG [GossipStage:1] 2016-10-23 04:47:00,705 FailureDetector.java:456 -
Ignoring interval time of 2065449328 for /10.0.0.1
DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 -
Ignoring interval time of 2065577694 for /10.0.0.2
DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 -
Ignoring interval time of 2065549822 for /10.0.0.3
DEBUG [Thread-6] 2016-10-23 04:47:01,201 StorageProxy.java:2246 - Schemas are
in agreement.
DEBUG [GossipStage:1] 2016-10-23 04:47:01,635 FailureDetector.java:456 -
Ignoring interval time of 2543137972 for /10.0.0.4
DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 -
Ignoring interval time of 2218246254 for /10.0.0.4
DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 -
Ignoring interval time of 2218388771 for /10.0.0.5
DEBUG [GossipStage:1] 2016-10-23 04:47:04,549 FailureDetector.java:456 -
Ignoring interval time of 2000269752 for /10.0.0.2
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,281
RepairMessageVerbHandler.java:116 - Validating
ValidationRequest{gcBefore=1474606025}
org.apache.cassandra.repair.messages.ValidationRequest@57e4b3c9
DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,281
StorageService.java:2955 - Forcing flush on keyspace myschema, CF mytable1
DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,282
ColumnFamilyStore.java:850 - Enqueuing flush of mytable1: 28142 (0%) on-heap,
16292220 (0%) off-heap
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,282 Memtable.java:368 -
Writing Memtable-mytable1@1772797855(3.616MiB serialized bytes, 47 ops, 0%/0%
of on/off-heap limit)
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,285
RepairMessageVerbHandler.java:133 - Syncing SyncRequest{initiator=/10.0.0.6,
src=/10.0.0.7, dst=/10.0.0.1,
ranges=[(-6681996591388643318,-6681992059482333621],
(-6681946740419236648,-6681942208512926951],
(-6681933144700307557,-6681928612793997859],
(-6681915017075068768,-6681910485168759070],
(-6681896889449829979,-6681892357543520281],
(-6681887825637210584,-6681883293730900887],
(-6681842506574113612,-6681837974667803914],
(-6681837974667803914,-6681833442761494217],
(-6681815315136255428,-6681810783229945731],
(-6681810783229945731,-6681801719417326336],
(-6681783591792087547,-6681779059885777850],
(-6681769996073158456,-6681765464166848758],
(-6681747336541609969,-6681742804635300272],
(-6681720145103751785,-6681715613197442088],
(-6681634038883867537,-6681629506977557840],
(-6681629506977557840,-6681620443164938445],
(-6681593251727080262,-6681584187914460867],
(-6681570592195531776,-6681566060289222078],
(-6681552464570292987,-6681547932663983289],
(-6681547932663983289,-6681543400757673592],
(-6681520741226125106,-6681516209319815409],
(-6681493549788266922,-6681484485975647528],
(-6681448230725169950,-6681443698818860253]]}
org.apache.cassandra.repair.messages.SyncRequest@5fbf8b2c
INFO [AntiEntropyStage:1] 2016-10-23 04:47:05,286 StreamingRepairTask.java:58
- [streaming task #bb1f9dd0-98db-11e6-ad5a-89fa523ea316] Performing streaming
repair of 23 ranges with /10.0.0.1
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,286 ColumnFamilyStore.java:850 -
Enqueuing flush of mytable2: 1695 (0%) on-heap, 19377 (0%) off-heap
DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,286 Memtable.java:368 -
Writing Memtable-mytable2@1296803980(19.105KiB serialized bytes, 3 ops, 0%/0%
of on/off-heap limit)
DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,322 Memtable.java:401 -
Completed flushing
/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db
(19.019KiB) for commitlog position ReplayPosition(segmentId=1474469138946,
position=165305985)
DEBUG [CompactionExecutor:51352] 2016-10-23 04:47:05,366
CompactionTask.java:146 - Compacting (bf274360-98db-11e6-b4a3-2b509f67851f)
[/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db:level=0,
/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big-Data.db:level=0,
]
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,533 Memtable.java:401 -
Completed flushing
/var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/mc-422108-big-Data.db
(3.615MiB) for commitlog position ReplayPosition(segmentId=1474469138946,
position=165305985)
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,695 Memtable.java:368 -
Writing Memtable-mytable1.mytable1_my_idx@185942079(1.187KiB serialized bytes,
28 ops, 0%/0% of on/off-heap limit)
ERROR [MemtableFlushWriter:5503] 2016-10-23 04:47:05,757
CassandraDaemon.java:205 - Exception in thread
Thread[MemtableFlushWriter:5503,5,main]
java.lang.RuntimeException: Last written key DecoratedKey(1byAcm-002Hsb-EL,
31627941636d2d3030324873622d454c) >= current key
DecoratedKey(^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@,
31627941656e2d3030304171552d554b) writing into
/var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/.mytable1_my_idx/mc-16240-big-Data.db
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:45)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.io.sstable.SSTableTxnWriter.append(SSTableTxnWriter.java:52)
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:394)
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.Memtable.flush(Memtable.java:332)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
{noformat}
After this, the commitlog directory grew a lot (files in the directory weren't
cleaned up anymore, because MemtableFlushWriter died), and the node ran out of
space :)
Again on a table with a secondary index, so that's likely related.
Schema :
{noformat}
CREATE TABLE myschema.mytable1 (
id bigint PRIMARY KEY,
field1 blob,
field2 ascii
) WITH bloom_filter_fp_chance = 0.1
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'chunk_length_in_kb': '1024', 'class':
'org.apache.cassandra.io.compress.DeflateCompressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 2592000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
CREATE INDEX mytable1_my_idx ON myschema.mytable1 (field2);
{noformat}
was (Author: tvdw):
This issue still exists in 3.0.9.
{noformat}
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [10..11]: 11
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [12..13]: 38
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [14..15]: 23
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [16..17]: 17
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [18..19]: 12
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [20..21]: 9
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [22..29]: 6
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704 Validator.java:235 -
Validated 1868 partitions for bb1f9dd0-98db-11e6-ad5a-89fa523ea316. Partition
sizes are:
INFO [AntiEntropyStage:1] 2016-10-23 04:46:59,703 Validator.java:274 - [repair
#bb1f9dd0-98db-11e6-ad5a-89fa523ea316] Sending completed merkle tree to
/10.0.0.6 for myschema.mytable2
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [17232..387341]: 99
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [387342..1097582]: 16
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [1097583..1807823]: 3
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [1807824..2518064]: 7
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
EstimatedHistogram.java:304 - [2518065..3959516]: 3
DEBUG [ValidationExecutor:3656] 2016-10-23 04:46:59,704
CompactionManager.java:1135 - Validation finished in 793 msec, for [repair
#bb1f9dd0-98db-11e6-ad5a-89fa523ea316 on myschema/mytable2,
[(-6682001123294953015,-6681421039287311765]]]
DEBUG [CompactionExecutor:51358] 2016-10-23 04:47:00,499
CompactionTask.java:218 - Compacted (bb4d3d80-98db-11e6-b4a3-2b509f67851f) 2
sstables to
[/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big,]
to level=0. 31,664,366 bytes to 31,653,685 (~99% of original) in 1,592ms =
18.961876MB/s. 0 total partitions merged to 1,888. Partition merge counts
were {1:2063, }
DEBUG [GossipStage:1] 2016-10-23 04:47:00,705 FailureDetector.java:456 -
Ignoring interval time of 2065449328 for /10.0.0.1
DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 -
Ignoring interval time of 2065577694 for /10.0.0.2
DEBUG [GossipStage:1] 2016-10-23 04:47:00,706 FailureDetector.java:456 -
Ignoring interval time of 2065549822 for /10.0.0.3
DEBUG [Thread-6] 2016-10-23 04:47:01,201 StorageProxy.java:2246 - Schemas are
in agreement.
DEBUG [GossipStage:1] 2016-10-23 04:47:01,635 FailureDetector.java:456 -
Ignoring interval time of 2543137972 for /10.0.0.4
DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 -
Ignoring interval time of 2218246254 for /10.0.0.4
DEBUG [GossipStage:1] 2016-10-23 04:47:04,310 FailureDetector.java:456 -
Ignoring interval time of 2218388771 for /10.0.0.5
DEBUG [GossipStage:1] 2016-10-23 04:47:04,549 FailureDetector.java:456 -
Ignoring interval time of 2000269752 for /10.0.0.2
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,281
RepairMessageVerbHandler.java:116 - Validating
ValidationRequest{gcBefore=1474606025}
org.apache.cassandra.repair.messages.ValidationRequest@57e4b3c9
DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,281
StorageService.java:2955 - Forcing flush on keyspace myschema, CF mytable1
DEBUG [ValidationExecutor:3656] 2016-10-23 04:47:05,282
ColumnFamilyStore.java:850 - Enqueuing flush of mytable1: 28142 (0%) on-heap,
16292220 (0%) off-heap
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,282 Memtable.java:368 -
Writing Memtable-mytable1@1772797855(3.616MiB serialized bytes, 47 ops, 0%/0%
of on/off-heap limit)
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,285
RepairMessageVerbHandler.java:133 - Syncing SyncRequest{initiator=/10.0.0.6,
src=/10.0.0.7, dst=/10.0.0.1,
ranges=[(-6681996591388643318,-6681992059482333621],
(-6681946740419236648,-6681942208512926951],
(-6681933144700307557,-6681928612793997859],
(-6681915017075068768,-6681910485168759070],
(-6681896889449829979,-6681892357543520281],
(-6681887825637210584,-6681883293730900887],
(-6681842506574113612,-6681837974667803914],
(-6681837974667803914,-6681833442761494217],
(-6681815315136255428,-6681810783229945731],
(-6681810783229945731,-6681801719417326336],
(-6681783591792087547,-6681779059885777850],
(-6681769996073158456,-6681765464166848758],
(-6681747336541609969,-6681742804635300272],
(-6681720145103751785,-6681715613197442088],
(-6681634038883867537,-6681629506977557840],
(-6681629506977557840,-6681620443164938445],
(-6681593251727080262,-6681584187914460867],
(-6681570592195531776,-6681566060289222078],
(-6681552464570292987,-6681547932663983289],
(-6681547932663983289,-6681543400757673592],
(-6681520741226125106,-6681516209319815409],
(-6681493549788266922,-6681484485975647528],
(-6681448230725169950,-6681443698818860253]]}
org.apache.cassandra.repair.messages.SyncRequest@5fbf8b2c
INFO [AntiEntropyStage:1] 2016-10-23 04:47:05,286 StreamingRepairTask.java:58
- [streaming task #bb1f9dd0-98db-11e6-ad5a-89fa523ea316] Performing streaming
repair of 23 ranges with /10.0.0.1
DEBUG [AntiEntropyStage:1] 2016-10-23 04:47:05,286 ColumnFamilyStore.java:850 -
Enqueuing flush of mytable2: 1695 (0%) on-heap, 19377 (0%) off-heap
DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,286 Memtable.java:368 -
Writing Memtable-mytable2@1296803980(19.105KiB serialized bytes, 3 ops, 0%/0%
of on/off-heap limit)
DEBUG [MemtableFlushWriter:5502] 2016-10-23 04:47:05,322 Memtable.java:401 -
Completed flushing
/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db
(19.019KiB) for commitlog position ReplayPosition(segmentId=1474469138946,
position=165305985)
DEBUG [CompactionExecutor:51352] 2016-10-23 04:47:05,366
CompactionTask.java:146 - Compacting (bf274360-98db-11e6-b4a3-2b509f67851f)
[/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198851-big-Data.db:level=0,
/var/lib/cassandra/data/myschema/mytable2-d66aa7605c6e11e583025dc6433c0832/mc-198850-big-Data.db:level=0,
]
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,533 Memtable.java:401 -
Completed flushing
/var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/mc-422108-big-Data.db
(3.615MiB) for commitlog position ReplayPosition(segmentId=1474469138946,
position=165305985)
DEBUG [MemtableFlushWriter:5503] 2016-10-23 04:47:05,695 Memtable.java:368 -
Writing Memtable-mytable1.mytable1_my_idx@185942079(1.187KiB serialized bytes,
28 ops, 0%/0% of on/off-heap limit)
ERROR [MemtableFlushWriter:5503] 2016-10-23 04:47:05,757
CassandraDaemon.java:205 - Exception in thread
Thread[MemtableFlushWriter:5503,5,main]
java.lang.RuntimeException: Last written key DecoratedKey(1byAcm-002Hsb-EL,
31627941636d2d3030324873622d454c) >= current key
DecoratedKey(^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@,
31627941656e2d3030304171552d554b) writing into
/var/lib/cassandra/data/myschema/mytable1-8057f24050b811e583025dc6433c0832/.mytable1_my_idx/mc-16240-big-Data.db
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.beforeAppend(BigTableWriter.java:106)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:145)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:45)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.io.sstable.SSTableTxnWriter.append(SSTableTxnWriter.java:52)
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:394)
~[apache-cassandra-3.0.9.jar:3.0.9]
at org.apache.cassandra.db.Memtable.flush(Memtable.java:332)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
{noformat}
After this, the commitlog directory grew a lot (files in the directory weren't
cleaned up anymore, because MemtableFlushWriter died), and the node ran out of
space :)
Again on a table with a secondary index, so that's likely related.
Schema :
{noformat}
CREATE TABLE bizmail.pgmailstorage (
id bigint PRIMARY KEY,
field1 blob,
field2 ascii
) WITH bloom_filter_fp_chance = 0.1
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'chunk_length_in_kb': '1024', 'class':
'org.apache.cassandra.io.compress.DeflateCompressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 2592000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
CREATE INDEX mytable1_my_idx ON myschema.mytable1 (field2);
{noformat}
> ERROR [MemtableFlushWriter:4] 2016-07-01 06:20:41,137
> CassandraDaemon.java:185 - Exception in thread
> Thread[MemtableFlushWriter:4,5,main] java.lang.RuntimeException: Last
> written key DecoratedKey(XXXX.XX, XXXXX) >= current key DecoratedKey
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-12125
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12125
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: RHEL-6.5 64-bit Apache Cassandra 2.2.5v
> Reporter: Relish Chackochan
> Fix For: 2.2.x
>
>
> We are running on RHEL-6.5 64-bit with Apache Cassandra 2.2.5v on 4 node
> cluster and getting the following error on multiple node while running the
> repair job and when getting the error repair job is hang.
> Can some one help to identify the issue.
> {code}
> ERROR [MemtableFlushWriter:4] 2016-07-01 06:20:41,137
> CassandraDaemon.java:185 - Exception in thread
> Thread[MemtableFlushWriter:4,5,main]
> java.lang.RuntimeException: Last written key DecoratedKey(1467371986.8870,
> 313436373337313938362e38383730) >= current key DecoratedKey(,
> 313436373337323030312e38383730) writing into
> /opt/cassandra/data/proddb/log_data1-0a5092a0a4fa11e5872fc1ce0a46dc27/.maxdatetimeindex_idx/tmp-la-470-big-Data.db
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
