[
https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13694977#comment-13694977
]
Pavel Yaskevich commented on CASSANDRA-5661:
--------------------------------------------
Well it depends on how do you define equal, having 1TB of data would definitely
require bigger heap and physical memory configuration.
Let's calculate (where each file have one buffer in memory at all times):
5MB files (each 79KB decompression buffer) for 1GB of such files in memory
would be: 204 (num files in 1GB) * 79KB = *16MB* buffers
128MB files (-//-) require 25.5 times less buffers per 1GB than 5MB files: 16MB
(buffers per 1GB in case of 5MB files) / 25.5 = *643KB* buffers
So for 1TB with 5MB files we need 1024 * 16MB = *16GB* of heap and for 128MB
files it's 25.5 times less = *643MB*, if each of the files is going to have at
least 8 caches items in the same time with 128MB files we are going to have
around 5GB of heap but I do think this scenario is a worst case, normal mode
would be 2-3GB. If you go with 14-16GB heap and 1TB of data, 2GB of cache is
the least of your problems as it's around 10% of total heap size which is still
good trade-off to allocation rate if those buffers are allocated per call.
> Discard pooled readers for cold data
> ------------------------------------
>
> Key: CASSANDRA-5661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5661
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.1
> Reporter: Jonathan Ellis
> Assignee: Pavel Yaskevich
> Fix For: 1.2.7
>
> Attachments: DominatorTree.png, Histogram.png
>
>
> Reader pooling was introduced in CASSANDRA-4942 but pooled
> RandomAccessReaders are never cleaned up until the SSTableReader is closed.
> So memory use is "the worst case simultaneous RAR we had open for this file,
> forever."
> We should introduce a global limit on how much memory to use for RAR, and
> evict old ones.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
