[
https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13694164#comment-13694164
]
Pavel Yaskevich commented on CASSANDRA-5661:
--------------------------------------------
bq. While in theory I like the idea of the searchers in most cases expiring
quicker with timers, in practice since these are on the heap, and LCS defaults
to only 5 MB files per levels, you can have A LOT of sstables, 200 per GB...
Which as Jonathan Ellis mentioned makes # of sstables * concurrent readers a
big problem, which was the problem hit above. So we really need to bound memory
usage with a hard cap, not the fuzzy cap of "how many can I open in X seconds"
When I was talking about concurrent readers I actually meant the max number of
threads in READ stage (concurrent_reads in yaml, default 32) that could be
running in the same time, that is the worst case limit of cache queue per
SSTable.
I'm not convinced that LRU on SSTable would work better then [C]RAR instance
expiration for our use case because due to replacement mechanism it creates
jitter in latencies grater than p75 once global threshold is reached, which in
production systems create situation when people can't explain why latencies
suddenly degraded without increasing amount of data.
I'm not sure that I follow what you mean by last part, we are not trying to
limit number of open for period of time but rather expire items that been
returned to the queue after time period. I agree that we can add upper limit on
memory usage just like we did for key/row caches, which in combination with
expiring would yield predictable behavior.
bq. 1k reqs/sec hitting a 4 level deep LCS CF could mean 4k Reader's (~500 MB)
created per second. Now that I say that out loud, I almost think we should go
back to not caching these at all so they always just get recycled in young gen
and never have a chance to hit old gen. I guess it could be a config parameter
which defaults on for STCS and off for LCS. Since STCS work loads have no where
near as many sstables.
It sounds like memory overhead caused by compression for your use-case is too
big with or without caching (it's around 79KB = 64KB default chunk + snappy
overhead) per file, even if you drop caching your allocation rate would cause
[C]RAR buffers to be promoted and cause fragmentation in the old gen which
could cause long "remark" phrase of CMS (can be battled with
CMSScavengeBeforeRemark option) and as CMS is not compacting you can suffer
from FullGC more frequently. That is what we are trying to battle with caching
[C]RAR instances (and their buffers) as if we allocate those close in time and
reuse, it's more luckily that they are going to be laid out closely plus
removes portion of work from young gen collector.
> Discard pooled readers for cold data
> ------------------------------------
>
> Key: CASSANDRA-5661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5661
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.1
> Reporter: Jonathan Ellis
> Assignee: Pavel Yaskevich
> Fix For: 1.2.7
>
>
> Reader pooling was introduced in CASSANDRA-4942 but pooled
> RandomAccessReaders are never cleaned up until the SSTableReader is closed.
> So memory use is "the worst case simultaneous RAR we had open for this file,
> forever."
> We should introduce a global limit on how much memory to use for RAR, and
> evict old ones.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
