[
https://issues.apache.org/jira/browse/CASSANDRA-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606105#comment-14606105
]
Constance Eustace edited comment on CASSANDRA-9640 at 6/29/15 6:47 PM:
-----------------------------------------------------------------------
Here is the keyspace and the presumed naughty table (entity_etljob)
this has a couple days of untruncated data still in it....
Keyspace: export
Read Count: 299734
Read Latency: 0.45569862945144696 ms.
Write Count: 210654
Write Latency: 0.17873762662944923 ms.
Pending Flushes: 0
Table: blob
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: entity_etljob
SSTable count: 6
Space used (live): 1595396657
Space used (total): 1595396657
Space used by snapshots (total): 1120310109
Off heap memory used (total): 663910
SSTable Compression Ratio: 0.27047144252635896
Number of keys (estimate): 15
Memtable cell count: 277
Memtable data size: 46858
Memtable off heap memory used: 0
Memtable switch count: 1441
Local read count: 251118
Local read latency: 0.387 ms
Local write count: 171758
Local write latency: 0.205 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 896
Bloom filter off heap memory used: 848
Index summary off heap memory used: 318
Compression metadata off heap memory used: 662744
Compacted partition minimum bytes: 1955667
Compacted partition maximum bytes: 1663415872
Compacted partition mean bytes: 400127491
Average live cells per slice (last five minutes):
2.4107152812621955
Maximum live cells per slice (last five minutes): 3.0
Average tombstones per slice (last five minutes):
0.5522065323871646
Maximum tombstones per slice (last five minutes): 12.0
----------------
was (Author: cowardlydragon):
Keyspace: export
Read Count: 299734
Read Latency: 0.45569862945144696 ms.
Write Count: 210654
Write Latency: 0.17873762662944923 ms.
Pending Flushes: 0
Table: blob
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table (index): entity_etljob.export__etljobreferenceididx
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table (index): entity_etljob.export__etljobstatusdetailidx
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table (index): entity_etljob.export__etljobstatusidx
SSTable count: 2
Space used (live): 10391
Space used (total): 10391
Space used by snapshots (total): 15181
Off heap memory used (total): 227
SSTable Compression Ratio: 0.6828158390949088
Number of keys (estimate): 4
Memtable cell count: 2
Memtable data size: 71
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 192
Local write latency: 0.191 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 192
Bloom filter off heap memory used: 176
Index summary off heap memory used: 35
Compression metadata off heap memory used: 16
Compacted partition minimum bytes: 104
Compacted partition maximum bytes: 535
Compacted partition mean bytes: 311
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table (index): entity_etljob.export__etljobuseridx
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: entity_etljob
SSTable count: 6
Space used (live): 1595396657
Space used (total): 1595396657
Space used by snapshots (total): 1120310109
Off heap memory used (total): 663910
SSTable Compression Ratio: 0.27047144252635896
Number of keys (estimate): 15
Memtable cell count: 277
Memtable data size: 46858
Memtable off heap memory used: 0
Memtable switch count: 1441
Local read count: 251118
Local read latency: 0.387 ms
Local write count: 171758
Local write latency: 0.205 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 896
Bloom filter off heap memory used: 848
Index summary off heap memory used: 318
Compression metadata off heap memory used: 662744
Compacted partition minimum bytes: 1955667
Compacted partition maximum bytes: 1663415872
Compacted partition mean bytes: 400127491
Average live cells per slice (last five minutes):
2.4107152812621955
Maximum live cells per slice (last five minutes): 3.0
Average tombstones per slice (last five minutes):
0.5522065323871646
Maximum tombstones per slice (last five minutes): 12.0
Table: entity_etlprocess
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table (index): entity_exportjob.export__exportjobjobidx
SSTable count: 2
Space used (live): 19856516
Space used (total): 19856516
Space used by snapshots (total): 0
Off heap memory used (total): 25219
SSTable Compression Ratio: 0.21122190342005245
Number of keys (estimate): 7865
Memtable cell count: 394
Memtable data size: 55204
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 4826
Local write latency: 0.013 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 9856
Bloom filter off heap memory used: 9840
Index summary off heap memory used: 3339
Compression metadata off heap memory used: 12040
Compacted partition minimum bytes: 125
Compacted partition maximum bytes: 1131752
Compacted partition mean bytes: 13867
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table (index): entity_exportjob.export__exportjobspaceidx
SSTable count: 3
Space used (live): 27029041
Space used (total): 27029041
Space used by snapshots (total): 0
Off heap memory used (total): 12134
SSTable Compression Ratio: 0.2521895897204678
Number of keys (estimate): 31
Memtable cell count: 394
Memtable data size: 57159
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 4826
Local write latency: 0.015 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 224
Bloom filter off heap memory used: 200
Index summary off heap memory used: 78
Compression metadata off heap memory used: 11856
Compacted partition minimum bytes: 104
Compacted partition maximum bytes: 89970660
Compacted partition mean bytes: 3597800
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table (index): entity_exportjob.export__exportstatusidx
SSTable count: 2
Space used (live): 27420825
Space used (total): 27420825
Space used by snapshots (total): 0
Off heap memory used (total): 13188
SSTable Compression Ratio: 0.24913414379401946
Number of keys (estimate): 5
Memtable cell count: 1182
Memtable data size: 129070
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 32581
Local read latency: 1.151 ms
Local write count: 14474
Local write latency: 0.025 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 192
Bloom filter off heap memory used: 176
Index summary off heap memory used: 52
Compression metadata off heap memory used: 12960
Compacted partition minimum bytes: 3974
Compacted partition maximum bytes: 89970660
Compacted partition mean bytes: 22215204
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: entity_exportjob
SSTable count: 8
Space used (live): 1026923378
Space used (total): 1026923378
Space used by snapshots (total): 0
Off heap memory used (total): 3465273
SSTable Compression Ratio: 0.1714111685071024
Number of keys (estimate): 1452928
Memtable cell count: 22064
Memtable data size: 1440533
Memtable off heap memory used: 0
Memtable switch count: 8
Local read count: 15918
Local read latency: 0.131 ms
Local write count: 14480
Local write latency: 0.138 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 2190040
Bloom filter off heap memory used: 2189976
Index summary off heap memory used: 601921
Compression metadata off heap memory used: 673376
Compacted partition minimum bytes: 536
Compacted partition maximum bytes: 4768
Compacted partition mean bytes: 3983
Average live cells per slice (last five minutes):
2.7512878502324414
Maximum live cells per slice (last five minutes): 3.0
Average tombstones per slice (last five minutes):
0.0028269883151149644
Maximum tombstones per slice (last five minutes): 3.0
Table: entity_job
SSTable count: 8
Space used (live): 3334988111
Space used (total): 3334988111
Space used by snapshots (total): 0
Off heap memory used (total): 8698261
SSTable Compression Ratio: 0.1790570118240906
Number of keys (estimate): 3978645
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 4932672
Bloom filter off heap memory used: 4932608
Index summary off heap memory used: 1647717
Compression metadata off heap memory used: 2117936
Compacted partition minimum bytes: 1917
Compacted partition maximum bytes: 5722
Compacted partition mean bytes: 4649
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: entity_synthetic
SSTable count: 2
Space used (live): 11957294
Space used (total): 11957294
Space used by snapshots (total): 0
Off heap memory used (total): 8798
SSTable Compression Ratio: 0.23994047509939614
Number of keys (estimate): 1673
Memtable cell count: 175
Memtable data size: 419769
Memtable off heap memory used: 0
Memtable switch count: 5
Local read count: 20
Local read latency: 0.091 ms
Local write count: 98
Local write latency: 0.026 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 2120
Bloom filter off heap memory used: 2104
Index summary off heap memory used: 742
Compression metadata off heap memory used: 5952
Compacted partition minimum bytes: 771
Compacted partition maximum bytes: 61214
Compacted partition mean bytes: 32337
Average live cells per slice (last five minutes): 3.0
Maximum live cells per slice (last five minutes): 3.0
Average tombstones per slice (last five minutes): 3.0
Maximum tombstones per slice (last five minutes): 3.0
Table: entity_zzzz
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: processing
SSTable count: 0
Space used (live): 0
Space used (total): 0
Space used by snapshots (total): 0
Off heap memory used (total): 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 0
Bloom filter off heap memory used: 0
Index summary off heap memory used: 0
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 0
Compacted partition maximum bytes: 0
Compacted partition mean bytes: 0
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: relation
SSTable count: 1
Space used (live): 13716076
Space used (total): 13716076
Space used by snapshots (total): 0
Off heap memory used (total): 150458
SSTable Compression Ratio: 0.2372980835174207
Number of keys (estimate): 85528
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 97
Local read latency: 0.186 ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 107816
Bloom filter off heap memory used: 107808
Index summary off heap memory used: 38282
Compression metadata off heap memory used: 4368
Compacted partition minimum bytes: 216
Compacted partition maximum bytes: 8239
Compacted partition mean bytes: 468
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: relationbackref
SSTable count: 1
Space used (live): 18520887
Space used (total): 18520887
Space used by snapshots (total): 0
Off heap memory used (total): 270829
SSTable Compression Ratio: 0.32111791108147225
Number of keys (estimate): 147316
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 181768
Bloom filter off heap memory used: 181760
Index summary off heap memory used: 86453
Compression metadata off heap memory used: 2616
Compacted partition minimum bytes: 125
Compacted partition maximum bytes: 179
Compacted partition mean bytes: 152
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
----------------
> Nodetool repair of very wide, large rows causes GC pressure and
> destabilization
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-9640
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9640
> Project: Cassandra
> Issue Type: Bug
> Environment: AWS, ~8GB heap
> Reporter: Constance Eustace
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 2.1.x
>
> Attachments: syslog.zip
>
>
> We've noticed our nodes becoming unstable with large, unrecoverable Old Gen
> GCs until OOM.
> This appears to be around the time of repair, and the specific cause seems to
> be one of our report computation tables that involves possible very wide rows
> with 10GB of data in it. THis is an RF 3 table in a four-node cluster.
> We truncate this occasionally, and we also had disabled this computation
> report for a bit and noticed better node stabiliy.
> I wish I had more specifics. We are switching to an RF 1 table and do more
> proactive truncation of the table.
> When things calm down, we will attempt to replicate the issue and watch GC
> and other logs.
> Any suggestion for things to look for/enable tracing on would be welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
