jonathan lacefield created CASSANDRA-6937:
---------------------------------------------
Summary: On large insert, node is being marked down with Commit
Log failure
Key: CASSANDRA-6937
URL: https://issues.apache.org/jira/browse/CASSANDRA-6937
Project: Cassandra
Issue Type: Bug
Environment: 16 vcpu, 480GB SSD storage (used 7GB), 32GB RAM,
Cassandra 2.0.5.24, hsha, datastax Java driver 2.0.1, vnodes
Reporter: jonathan lacefield
Priority: Minor
Attachments: cassandra.yaml copy, system.log
during large insert, will get specs on insert load, nodes in the cluster are
being marked down. creating defect for investigation.
error message in log file of one of the down nodes:
ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:06,666 StorageService.java (line
364) Stopping gossiper
WARN [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:06,666 StorageService.java (line
278) Stopping gossip by operator request
INFO [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:06,666 Gossiper.java (line 1251)
Announcing shutdown
INFO [FlushWriter:421] 2014-03-26 08:28:06,677 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data1/system/compactions_in_progress/system-compactions_in_progress-jb-198-Data.db
(166 bytes) for commitlog position ReplayPosition(segmentId=1395644152684,
position=9577531)
INFO [CompactionExecutor:1880] 2014-03-26 08:28:06,679 CompactionTask.java
(line 115) Compacting
[SSTableReader(path='/var/lib/cassandra/data2/OpsCenter/rollups60/OpsCenter-rollups60-jb-18-Data.db'),
SSTableReader(path='/var/lib/cassandra/data1/OpsCenter/rollups60/OpsCenter-rollups60-jb-19-Data.db'),
SSTableReader(path='/var/lib/cassandra/data2/OpsCenter/rollups60/OpsCenter-rollups60-jb-17-Data.db'),
SSTableReader(path='/var/lib/cassandra/data1/OpsCenter/rollups60/OpsCenter-rollups60-jb-20-Data.db')]
INFO [OptionalTasks:1] 2014-03-26 08:28:07,322 MeteredFlusher.java (line 63)
flushing high-traffic column family CFS(Keyspace='system',
ColumnFamily='batchlog') (estimated 215899526 bytes)
INFO [OptionalTasks:1] 2014-03-26 08:28:07,322 ColumnFamilyStore.java (line
784) Enqueuing flush of Memtable-batchlog@74009414(39242953/215899526
serialized/live bytes, 239 ops)
INFO [FlushWriter:422] 2014-03-26 08:28:07,323 Memtable.java (line 331) Writing
Memtable-batchlog@74009414(39242953/215899526 serialized/live bytes, 239 ops)
INFO [FlushWriter:422] 2014-03-26 08:28:07,370 Memtable.java (line 371)
Completed flushing
/var/lib/cassandra/data1/system/batchlog/system-batchlog-jb-1-Data.db (204617
bytes) for commitlog position ReplayPosition(segmentId=1395644152684,
position=17705519)
INFO [MemoryMeter:1] 2014-03-26 08:28:08,296 Memtable.java (line 449)
CFS(Keyspace='system', ColumnFamily='batchlog') liveRatio is 3.2515027780484544
(just-counted was 1.0013929201634204). calculation took 0ms for 12 cells
ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,667 StorageService.java (line
369) Stopping RPC server
INFO [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,667 ThriftServer.java (line
141) Stop listening to thrift clients
ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,698 StorageService.java (line
374) Stopping native transport
INFO [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,765 Server.java (line 181) Stop
listening for CQL clients
ERROR [COMMIT-LOG-ALLOCATOR] 2014-03-26 08:28:08,766 CommitLog.java (line 379)
Failed to allocate new commit log segments. Commit disk failure policy is stop;
terminating thread
java.lang.AssertionError: attempted to delete non-existing file
CommitLog-3-1395644152652.log
--
This message was sent by Atlassian JIRA
(v6.2#6252)
