Wade Poziombka created CASSANDRA-4314:
-----------------------------------------
Summary: OOM errors on key slice
Key: CASSANDRA-4314
URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.0
Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
Reporter: Wade Poziombka
Attachments: oom.zip
My database (now at 1.0.10) is in a state in which it goes out of memory with
hardly any activity at all. A key slice nothing more.
The logs attached are this including verbose gc in stdout. I started up
cassandra and waited a bit to ensure that it was unperturbed.
Then (about 15:46) I ran this slice (using Pelops), which in this case should
return NO data. My client times out and the database goes OOM.
ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in my
cluster
Selector s = new Selector(this.pool);
List<IndexExpression> indexExpressions = new
ArrayList<IndexExpression>();
IndexExpression e = new IndexExpression(
ByteBuffer.wrap("encryptionSettingsID".getBytes(ASCII)), IndexOperator.EQ,
ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
indexExpressions.add(e);
IndexClause indexClause = new IndexClause(indexExpressions,
ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
SlicePredicate predicate = new SlicePredicate();
predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
{ ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
List<KeySlice> slices = s.getKeySlices(CF_TOKEN, indexClause,
predicate, cl);
Note that “encryptionSettingsID” is an indexed column. When this is executed
there should be no columns with the supplied value.
I suppose I may have some kind of blatant error in this query but it is not
obvious to me. I’m relatively new to cassandra.
My key space is defined as follows:
KsDef(name:TB_UNIT, strategy_class:org.apache.cassandra.locator.SimpleStrategy,
strategy_options:{replication_factor=3},
cf_defs:[
CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
validation_class:UTF8Type, index_type:KEYS, index_name:EncryptionSettingsID)],
caching:keys_only),
CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
column_type:Standard, comparator_type:BytesType, caching:keys_only),
CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75 6E
74, validation_class:CounterColumnType)],
default_validation_class:CounterColumnType, caching:keys_only)
])
tpstats show pending tasks many minutes after time out:
[root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
Pool Name Active Pending Completed Blocked All
time blocked
ReadStage 3 3 107 0
0
RequestResponseStage 0 0 56 0
0
MutationStage 0 0 6 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 2231 0
0
AntiEntropyStage 0 0 0 0
0
MigrationStage 0 0 0 0
0
MemtablePostFlusher 0 0 3 0
0
StreamStage 0 0 0 0
0
FlushWriter 0 0 3 0
0
MiscStage 0 0 0 0
0
InternalResponseStage 0 0 0 0
0
HintedHandoff 0 0 9 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
BINARY 0
READ 0
MUTATION 0
REQUEST_RESPONSE 0
cfstats:
Keyspace: keyspace
Read Count: 118
Read Latency: 0.14722033898305084 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Column Family: token
SSTable count: 7
Space used (live): 4745885584
Space used (total): 4745885584
Number of Keys (estimate): 18626048
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 118
Read Latency: 0.147 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Bloom Filter False Postives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 55058352
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 150
Compacted row maximum size: 258
Compacted row mean size: 201
Column Family: pan_2fef6478b62242dd94aecaa049b9d7bb
SSTable count: 7
Space used (live): 1987147156
Space used (total): 1987147156
Number of Keys (estimate): 14955264
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Bloom Filter False Postives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 28056224
Key cache: disabled
Row cache: disabled
Compacted row minimum size: 104
Compacted row maximum size: 124
Compacted row mean size: 124
Column Family: counters
SSTable count: 11
Space used (live): 3433469364
Space used (total): 3433469364
Number of Keys (estimate): 21475328
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Bloom Filter False Postives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 40271696
Key cache capacity: 4652
Key cache size: 4652
Key cache hit rate: NaN
Row cache: disabled
Compacted row minimum size: 125
Compacted row maximum size: 179
Compacted row mean size: 150
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
