Re: Any way to control/limit off-heap memory?

2017-03-07 Thread Jeff Jirsa 


On 2017-03-06 07:04 (-0800), "Thakrar, Jayesh"  
wrote: 
> Thanks Hannu - also considered that option.
> However, that's a trial and error and will have to play with the 
> collision/false-positive fraction.
> And each iteration will most likely result in a compaction storm - so I was 
> hoping for a way to throttle/limit the max off-heap size.
> 
> The reason I was thinking of eliminating bloom filters is because due to 
> application design, we search for data using a partial key (prefix columns),
> hence am thinking of completely eliminating the bloom filters as they do not 
> add any value in such a use case.
> 

If you dont want to use the bloom filters, don't set the FP ratio to 0, set it 
to something like 0.1 or 0.5. A fp ratio of 0 says "no false positives", which 
is only possible with HUGE bloom filters. A high FP ratio (since you're not 
using them) basically says "don't try very hard" which corresponds to small 
arrays, which means low accuracy, but low offheap usage. 

We probably shouldn't even allow 0. FP ratio.

Re: Any way to control/limit off-heap memory?

2017-03-06 Thread Thakrar, Jayesh 
Thanks Hannu - also considered that option.
However, that's a trial and error and will have to play with the 
collision/false-positive fraction.
And each iteration will most likely result in a compaction storm - so I was 
hoping for a way to throttle/limit the max off-heap size.

The reason I was thinking of eliminating bloom filters is because due to 
application design, we search for data using a partial key (prefix columns),
hence am thinking of completely eliminating the bloom filters as they do not 
add any value in such a use case.

Is my assumption correct?

From: Hannu Kröger <hkro...@gmail.com>
Date: Sunday, March 5, 2017 at 6:34 AM
To: <user@cassandra.apache.org>
Subject: Re: Any way to control/limit off-heap memory?

If bloom filters are taking too much memory, you can adjust bloom filters:
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tuning_bloom_filters_c.html

Hannu

On 4 Mar 2017, at 22:54, Thakrar, Jayesh 
<jthak...@conversantmedia.com<mailto:jthak...@conversantmedia.com>> wrote:

I have a situation where the off-heap memory is bloating the jvm process 
memory, making it a candidate to be killed by the oom_killer.
My server has 256 GB RAM and Cassandra heap memory of 16 GB

Below is the output of "nodetool info" and nodetool compactionstats for a 
culprit table which causes bloom filter bloat.
Ofcourse one option is to turnoff bloom filter, but I need to look into 
application access pattern, etc.


xss =  -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp 
-javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar 
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M 
-XX:+HeapDumpOnOutOfMemoryError -Xss256k
ID : 2b9b4252-0760-49c1-8d14-544be0183271
Gossip active  : true
Thrift active  : false
Native Transport active: true
Load   : 953.19 GB
Generation No  : 1488641545
Uptime (seconds)   : 15706
Heap Memory (MB)   : 7692.93 / 16309.00
Off Heap Memory (MB)   : 175115.07
Data Center: ord
Rack   : rack3
Exceptions : 0
Key Cache  : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 
requests, NaN recent hit rate, 14400 save period in seconds
Row Cache  : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 
requests, NaN recent hit rate, 0 save period in seconds
Counter Cache  : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 
requests, NaN recent hit rate, 7200 save period in seconds
Token  : (invoke with -T/--tokens to see all 256 tokens)


Table: logs_by_user
SSTable count: 622
SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 313156769247
Space used (total): 313156769247
Space used by snapshots (total): 0
Off heap memory used (total): 180354511884
SSTable Compression Ratio: 0.25016314078395613
Number of keys (estimate): 147261312
Memtable cell count: 44796
Memtable data size: 57578717
Memtable off heap memory used: 0
Memtable switch count: 21
Local read count: 0
Local read latency: NaN ms
Local write count: 1148687
Local write latency: 0.123 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 180269125192
Bloom filter off heap memory used: 180269120216
Index summary off heap memory used: 24335340
Compression metadata off heap memory used: 61056328
Compacted partition minimum bytes: 150
Compacted partition maximum bytes: 668489532
Compacted partition mean bytes: 3539
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0


From: Conversant <jthak<mailto:jthak...@conversantmedia.com>

Re: Any way to control/limit off-heap memory?

2017-03-05 Thread Hannu Kröger 
If bloom filters are taking too much memory, you can adjust bloom filters:
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tuning_bloom_filters_c.html
 


Hannu

> On 4 Mar 2017, at 22:54, Thakrar, Jayesh  wrote:
> 
> I have a situation where the off-heap memory is bloating the jvm process 
> memory, making it a candidate to be killed by the oom_killer.
> My server has 256 GB RAM and Cassandra heap memory of 16 GB
>  
> Below is the output of "nodetool info" and nodetool compactionstats for a 
> culprit table which causes bloom filter bloat.
> Ofcourse one option is to turnoff bloom filter, but I need to look into 
> application access pattern, etc.
>  
>  
> xss =  -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp 
> -javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar 
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G 
> -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
> ID : 2b9b4252-0760-49c1-8d14-544be0183271
> Gossip active  : true
> Thrift active  : false
> Native Transport active: true
> Load   : 953.19 GB
> Generation No  : 1488641545
> Uptime (seconds)   : 15706
> Heap Memory (MB)   : 7692.93 / 16309.00
> Off Heap Memory (MB)   : 175115.07
> Data Center: ord
> Rack   : rack3
> Exceptions : 0
> Key Cache  : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 
> requests, NaN recent hit rate, 14400 save period in seconds
> Row Cache  : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 
> requests, NaN recent hit rate, 0 save period in seconds
> Counter Cache  : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 
> requests, NaN recent hit rate, 7200 save period in seconds
> Token  : (invoke with -T/--tokens to see all 256 tokens)
>  
>  
> Table: logs_by_user
> SSTable count: 622
> SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 313156769247
> Space used (total): 313156769247
> Space used by snapshots (total): 0
> Off heap memory used (total): 180354511884
> SSTable Compression Ratio: 0.25016314078395613
> Number of keys (estimate): 147261312
> Memtable cell count: 44796
> Memtable data size: 57578717
> Memtable off heap memory used: 0
> Memtable switch count: 21
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1148687
> Local write latency: 0.123 ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 180269125192
> Bloom filter off heap memory used: 180269120216
> Index summary off heap memory used: 24335340
> Compression metadata off heap memory used: 61056328
> Compacted partition minimum bytes: 150
> Compacted partition maximum bytes: 668489532
> Compacted partition mean bytes: 3539
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
>  
>  
> From: Conversant mailto:jthak...@conversantmedia.com>

Any way to control/limit off-heap memory?

2017-03-04 Thread Thakrar, Jayesh 
I have a situation where the off-heap memory is bloating the jvm process 
memory, making it a candidate to be killed by the oom_killer.
My server has 256 GB RAM and Cassandra heap memory of 16 GB

Below is the output of "nodetool info" and nodetool compactionstats for a 
culprit table which causes bloom filter bloat.
Ofcourse one option is to turnoff bloom filter, but I need to look into 
application access pattern, etc.


xss =  -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp 
-javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar 
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M 
-XX:+HeapDumpOnOutOfMemoryError -Xss256k
ID : 2b9b4252-0760-49c1-8d14-544be0183271
Gossip active  : true
Thrift active  : false
Native Transport active: true
Load   : 953.19 GB
Generation No  : 1488641545
Uptime (seconds)   : 15706
Heap Memory (MB)   : 7692.93 / 16309.00
Off Heap Memory (MB)   : 175115.07
Data Center: ord
Rack   : rack3
Exceptions : 0
Key Cache  : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 
requests, NaN recent hit rate, 14400 save period in seconds
Row Cache  : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 
requests, NaN recent hit rate, 0 save period in seconds
Counter Cache  : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 
requests, NaN recent hit rate, 7200 save period in seconds
Token  : (invoke with -T/--tokens to see all 256 tokens)


Table: logs_by_user
SSTable count: 622
SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 313156769247
Space used (total): 313156769247
Space used by snapshots (total): 0
Off heap memory used (total): 180354511884
SSTable Compression Ratio: 0.25016314078395613
Number of keys (estimate): 147261312
Memtable cell count: 44796
Memtable data size: 57578717
Memtable off heap memory used: 0
Memtable switch count: 21
Local read count: 0
Local read latency: NaN ms
Local write count: 1148687
Local write latency: 0.123 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 180269125192
Bloom filter off heap memory used: 180269120216
Index summary off heap memory used: 24335340
Compression metadata off heap memory used: 61056328
Compacted partition minimum bytes: 150
Compacted partition maximum bytes: 668489532
Compacted partition mean bytes: 3539
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0


From: Conversant