[jira] [Updated] (CASSANDRA-5957) Cannot drop keyspace Keyspace1 after running cassandra-stress

2013-10-11 Thread Tyler Hobbs (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-5957:
---

Attachment: 5957-1.2-v2.patch

5957-1.2-v2.patch (and 
[branch|https://github.com/thobbs/cassandra/tree/CASSANDRA-5957-v2]) disables 
the was not marked compacted assertion when cleaning up sstables for a 
dropped table and unreferences sstables when compaction tasks finish (or are 
interrupted) if the table has been dropped.

Regarding flushing, a blocking flush is forced after unregistering the CF but 
prior to the CFS being invalidated (which is when sstables get unreferenced), 
so we shouldn't see a similar problem there (and I haven't seen one while 
testing this).

 Cannot drop keyspace Keyspace1 after running cassandra-stress
 -

 Key: CASSANDRA-5957
 URL: https://issues.apache.org/jira/browse/CASSANDRA-5957
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: Cassandra 1.2.9 freshly built from cassandra-1.2 branch 
 (f5b224cf9aa0f319d51078ef4b78d55e36613963)
Reporter: Piotr Kołaczkowski
Assignee: Tyler Hobbs
Priority: Minor
 Fix For: 1.2.11

 Attachments: 5957-1.2-v1.patch, 5957-1.2-v2.patch, system.log


 Steps to reproduce:
 # Set MAX_HEAP=2G, HEAP_NEWSIZE=400M
 # Run ./cassandra-stress -n 5 -c 400 -S 256
 # The test should complete despite several warnings about low heap memory.
 # Try to drop keyspace:
 {noformat}
 cqlsh drop keyspace Keyspace1;
 TSocket read 0 bytes
 {noformat}
 system.log:
 {noformat}
  INFO 15:10:46,516 Enqueuing flush of 
 Memtable-schema_columnfamilies@2127258371(0/0 serialized/live bytes, 1 ops)
  INFO 15:10:46,516 Writing Memtable-schema_columnfamilies@2127258371(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,690 Completed flushing 
 /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-6-Data.db
  (38 bytes) for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794574)
  INFO 15:10:46,692 Enqueuing flush of Memtable-schema_columns@1997964959(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,693 Writing Memtable-schema_columns@1997964959(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,857 Completed flushing 
 /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-6-Data.db
  (38 bytes) for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794574)
  INFO 15:10:46,897 Enqueuing flush of Memtable-local@1366216652(98/98 
 serialized/live bytes, 3 ops)
  INFO 15:10:46,898 Writing Memtable-local@1366216652(98/98 serialized/live 
 bytes, 3 ops)
  INFO 15:10:47,064 Completed flushing 
 /var/lib/cassandra/data/system/local/system-local-ic-12-Data.db (139 bytes) 
 for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794845)
  INFO 15:10:48,956 Enqueuing flush of Memtable-local@432522279(46/46 
 serialized/live bytes, 1 ops)
  INFO 15:10:48,957 Writing Memtable-local@432522279(46/46 serialized/live 
 bytes, 1 ops)
  INFO 15:10:49,132 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 400882073/1094043713)bytes
  INFO 15:10:49,175 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 147514075/645675954)bytes
  INFO 15:10:49,185 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 223249644/609072261)bytes
  INFO 15:10:49,202 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 346471085/990388210)bytes
  INFO 15:10:49,215 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 294748503/2092376617)bytes
  INFO 15:10:49,257 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 692722235/739328646)bytes
  INFO 15:10:49,285 Completed flushing 
 /var/lib/cassandra/data/system/local/system-local-ic-13-Data.db (82 bytes) 
 for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794974)
  INFO 15:10:49,286 Compacting 
 [SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-10-Data.db'),
  
 SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-13-Data.db'),
  
 SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-12-Data.db'),
  
 SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-11-Data.db')]
 ERROR 15:10:49,287 Error occurred during processing of message.
 java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.lang.AssertionError: 
 SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ic-78-Data.db')
  was already marked compacted
   at

[jira] [Updated] (CASSANDRA-5957) Cannot drop keyspace Keyspace1 after running cassandra-stress

2013-10-09 Thread Tyler Hobbs (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-5957:
---

Attachment: 5957-1.2-v1.patch

Quick summary of the problem: when the table is dropped, there are multiple 
ongoing compactions.  When handling the table drop, DataTracker first builds a 
view of non-compacting sstables.  Meanwhile, the compaction tasks get 
cancelled, and as part of the cleanup, it marks the sstables as compacted 
(because the column family is no longer valid).  Then the DataTracker finally 
tries to mark the sstables in its view as compacted, hitting the failing 
assertion.  Additionally, when a compaction task is cancelled, it doesn't 
unreference the sstables involved, and since the DataTracker ignored compacting 
sstables, they never get unreferenced (even if the failing assertion is 
disabled).

5957-1.2-v1.patch (and 
[branch|https://github.com/thobbs/cassandra/tree/CASSANDRA-5957]) uses the 
simplest fix I could think of, which is to block the table drop operation until 
all compactions on that table have stopped (either normally, because of an 
error, or because they saw the stop signal).  This way, when the DataTracker 
cleans up sstables as part of the drop operation, there are no compacting 
sstables.

 Cannot drop keyspace Keyspace1 after running cassandra-stress
 -

 Key: CASSANDRA-5957
 URL: https://issues.apache.org/jira/browse/CASSANDRA-5957
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: Cassandra 1.2.9 freshly built from cassandra-1.2 branch 
 (f5b224cf9aa0f319d51078ef4b78d55e36613963)
Reporter: Piotr Kołaczkowski
Assignee: Tyler Hobbs
Priority: Minor
 Fix For: 1.2.11

 Attachments: 5957-1.2-v1.patch, system.log


 Steps to reproduce:
 # Set MAX_HEAP=2G, HEAP_NEWSIZE=400M
 # Run ./cassandra-stress -n 5 -c 400 -S 256
 # The test should complete despite several warnings about low heap memory.
 # Try to drop keyspace:
 {noformat}
 cqlsh drop keyspace Keyspace1;
 TSocket read 0 bytes
 {noformat}
 system.log:
 {noformat}
  INFO 15:10:46,516 Enqueuing flush of 
 Memtable-schema_columnfamilies@2127258371(0/0 serialized/live bytes, 1 ops)
  INFO 15:10:46,516 Writing Memtable-schema_columnfamilies@2127258371(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,690 Completed flushing 
 /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-6-Data.db
  (38 bytes) for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794574)
  INFO 15:10:46,692 Enqueuing flush of Memtable-schema_columns@1997964959(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,693 Writing Memtable-schema_columns@1997964959(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,857 Completed flushing 
 /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-6-Data.db
  (38 bytes) for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794574)
  INFO 15:10:46,897 Enqueuing flush of Memtable-local@1366216652(98/98 
 serialized/live bytes, 3 ops)
  INFO 15:10:46,898 Writing Memtable-local@1366216652(98/98 serialized/live 
 bytes, 3 ops)
  INFO 15:10:47,064 Completed flushing 
 /var/lib/cassandra/data/system/local/system-local-ic-12-Data.db (139 bytes) 
 for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794845)
  INFO 15:10:48,956 Enqueuing flush of Memtable-local@432522279(46/46 
 serialized/live bytes, 1 ops)
  INFO 15:10:48,957 Writing Memtable-local@432522279(46/46 serialized/live 
 bytes, 1 ops)
  INFO 15:10:49,132 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 400882073/1094043713)bytes
  INFO 15:10:49,175 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 147514075/645675954)bytes
  INFO 15:10:49,185 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 223249644/609072261)bytes
  INFO 15:10:49,202 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 346471085/990388210)bytes
  INFO 15:10:49,215 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 294748503/2092376617)bytes
  INFO 15:10:49,257 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 692722235/739328646)bytes
  INFO 15:10:49,285 Completed flushing 
 /var/lib/cassandra/data/system/local/system-local-ic-13-Data.db (82 bytes) 
 for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794974)
  INFO 15:10:49,286 Compacting 
 [SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-10-Data.db'),

[jira] [Updated] (CASSANDRA-5957) Cannot drop keyspace Keyspace1 after running cassandra-stress

2013-08-31 Thread JIRA 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Kołaczkowski updated CASSANDRA-5957:
--

Attachment: system.log

Attaching system.log from the whole last run.

 Cannot drop keyspace Keyspace1 after running cassandra-stress
 -

 Key: CASSANDRA-5957
 URL: https://issues.apache.org/jira/browse/CASSANDRA-5957
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: Cassandra 1.2.9 freshly built from cassandra-1.2 branch 
 (f5b224cf9aa0f319d51078ef4b78d55e36613963)
Reporter: Piotr Kołaczkowski
Assignee: Aleksey Yeschenko
Priority: Minor
 Fix For: 1.2.10

 Attachments: system.log


 Steps to reproduce:
 # Set MAX_HEAP=2G, HEAP_NEWSIZE=400M
 # Run ./cassandra-stress -n 5 -c 400 -S 256
 # The test should complete despite several warnings about low heap memory.
 # Try to drop keyspace:
 {noformat}
 cqlsh drop keyspace Keyspace1;
 TSocket read 0 bytes
 {noformat}
 system.log:
 {noformat}
  INFO 15:10:46,516 Enqueuing flush of 
 Memtable-schema_columnfamilies@2127258371(0/0 serialized/live bytes, 1 ops)
  INFO 15:10:46,516 Writing Memtable-schema_columnfamilies@2127258371(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,690 Completed flushing 
 /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-6-Data.db
  (38 bytes) for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794574)
  INFO 15:10:46,692 Enqueuing flush of Memtable-schema_columns@1997964959(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,693 Writing Memtable-schema_columns@1997964959(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,857 Completed flushing 
 /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-6-Data.db
  (38 bytes) for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794574)
  INFO 15:10:46,897 Enqueuing flush of Memtable-local@1366216652(98/98 
 serialized/live bytes, 3 ops)
  INFO 15:10:46,898 Writing Memtable-local@1366216652(98/98 serialized/live 
 bytes, 3 ops)
  INFO 15:10:47,064 Completed flushing 
 /var/lib/cassandra/data/system/local/system-local-ic-12-Data.db (139 bytes) 
 for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794845)
  INFO 15:10:48,956 Enqueuing flush of Memtable-local@432522279(46/46 
 serialized/live bytes, 1 ops)
  INFO 15:10:48,957 Writing Memtable-local@432522279(46/46 serialized/live 
 bytes, 1 ops)
  INFO 15:10:49,132 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 400882073/1094043713)bytes
  INFO 15:10:49,175 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 147514075/645675954)bytes
  INFO 15:10:49,185 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 223249644/609072261)bytes
  INFO 15:10:49,202 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 346471085/990388210)bytes
  INFO 15:10:49,215 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 294748503/2092376617)bytes
  INFO 15:10:49,257 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 692722235/739328646)bytes
  INFO 15:10:49,285 Completed flushing 
 /var/lib/cassandra/data/system/local/system-local-ic-13-Data.db (82 bytes) 
 for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794974)
  INFO 15:10:49,286 Compacting 
 [SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-10-Data.db'),
  
 SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-13-Data.db'),
  
 SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-12-Data.db'),
  
 SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-11-Data.db')]
 ERROR 15:10:49,287 Error occurred during processing of message.
 java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.lang.AssertionError: 
 SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ic-78-Data.db')
  was already marked compacted
   at 
 org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:378)
   at 
 org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:281)
   at 
 org.apache.cassandra.service.MigrationManager.announceKeyspaceDrop(MigrationManager.java:262)
   at 
 org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:718)
   at 
 org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:775)
   at

[jira] [Updated] (CASSANDRA-5957) Cannot drop keyspace Keyspace1 after running cassandra-stress

2013-08-30 Thread Jonathan Ellis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-5957:
--

 Reviewer: jbellis
 Priority: Minor  (was: Major)
Fix Version/s: 1.2.10
 Assignee: Aleksey Yeschenko

 Cannot drop keyspace Keyspace1 after running cassandra-stress
 -

 Key: CASSANDRA-5957
 URL: https://issues.apache.org/jira/browse/CASSANDRA-5957
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: Cassandra 1.2.9 freshly built from cassandra-1.2 branch 
 (f5b224cf9aa0f319d51078ef4b78d55e36613963)
Reporter: Piotr Kołaczkowski
Assignee: Aleksey Yeschenko
Priority: Minor
 Fix For: 1.2.10


 Steps to reproduce:
 # Set MAX_HEAP=2G, HEAP_NEWSIZE=400M
 # Run ./cassandra-stress -n 5 -c 400 -S 256
 # The test should complete despite several warnings about low heap memory.
 # Try to drop keyspace:
 {noformat}
 cqlsh drop keyspace Keyspace1;
 TSocket read 0 bytes
 {noformat}
 system.log:
 {noformat}
  INFO 15:10:46,516 Enqueuing flush of 
 Memtable-schema_columnfamilies@2127258371(0/0 serialized/live bytes, 1 ops)
  INFO 15:10:46,516 Writing Memtable-schema_columnfamilies@2127258371(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,690 Completed flushing 
 /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-6-Data.db
  (38 bytes) for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794574)
  INFO 15:10:46,692 Enqueuing flush of Memtable-schema_columns@1997964959(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,693 Writing Memtable-schema_columns@1997964959(0/0 
 serialized/live bytes, 1 ops)
  INFO 15:10:46,857 Completed flushing 
 /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-6-Data.db
  (38 bytes) for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794574)
  INFO 15:10:46,897 Enqueuing flush of Memtable-local@1366216652(98/98 
 serialized/live bytes, 3 ops)
  INFO 15:10:46,898 Writing Memtable-local@1366216652(98/98 serialized/live 
 bytes, 3 ops)
  INFO 15:10:47,064 Completed flushing 
 /var/lib/cassandra/data/system/local/system-local-ic-12-Data.db (139 bytes) 
 for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794845)
  INFO 15:10:48,956 Enqueuing flush of Memtable-local@432522279(46/46 
 serialized/live bytes, 1 ops)
  INFO 15:10:48,957 Writing Memtable-local@432522279(46/46 serialized/live 
 bytes, 1 ops)
  INFO 15:10:49,132 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 400882073/1094043713)bytes
  INFO 15:10:49,175 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 147514075/645675954)bytes
  INFO 15:10:49,185 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 223249644/609072261)bytes
  INFO 15:10:49,202 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 346471085/990388210)bytes
  INFO 15:10:49,215 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 294748503/2092376617)bytes
  INFO 15:10:49,257 Compaction interrupted: 
 Compaction@4d331c44-f018-302b-91c2-2dcf94c4bfad(Keyspace1, Standard1, 
 692722235/739328646)bytes
  INFO 15:10:49,285 Completed flushing 
 /var/lib/cassandra/data/system/local/system-local-ic-13-Data.db (82 bytes) 
 for commitlog position ReplayPosition(segmentId=1377867520699, 
 position=19794974)
  INFO 15:10:49,286 Compacting 
 [SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-10-Data.db'),
  
 SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-13-Data.db'),
  
 SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-12-Data.db'),
  
 SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-11-Data.db')]
 ERROR 15:10:49,287 Error occurred during processing of message.
 java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.lang.AssertionError: 
 SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-ic-78-Data.db')
  was already marked compacted
   at 
 org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:378)
   at 
 org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:281)
   at 
 org.apache.cassandra.service.MigrationManager.announceKeyspaceDrop(MigrationManager.java:262)
   at 
 org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:718)
   at 
 org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:775)
   at