[
https://issues.apache.org/jira/browse/CASSANDRA-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13701512#comment-13701512
]
Pavel Yaskevich commented on CASSANDRA-5661:
--------------------------------------------
bq. worst case pool size proportional to sstables * concurrent readers means
that memory usage increases linearly with disk size; increasing sstable size
decreases it linearly, so we expect 100MB sstables [which is on the high side
of what is reasonable IMO] to get us to ~4TB of space managed which is good but
not "problem solved" territory.
Let's clarify that it's a really-really worse case which means that your reads
touch whole dataset all the time and it's a good show case for side affect of
having fixed size files of small size so even if you don't do caching it would
create allocation rate proportional to the dataset size as well as increased
syscall rate to open/seek/close those files so it's frequent GC/FullGC vs OOM
in here (when run without memory cap).
bq. The most common operations in an object pool are the borrow/return. Ben's
multiway pool optimizes for this, with particular attention to possible
contention from multiple reader threads.
If I understand correctly by borrow you mean allocate object in the pool and
give it way to the caller, this is not how I would like it to behave instead we
can allocate object without assigning to the queue when it runs short (as all
of the objects are short lived) and then decide if we want it back when caller
it done with it, disadvantages of allocating to the pool I have already
described in my previous comments.
Bottom line for me being, I'm tired of arguing about this so I will let mighty
people to decide what they see fit as in all other cases (e.g. fadvic'ing whole
file on reads, preheating page cache etc.).
> Discard pooled readers for cold data
> ------------------------------------
>
> Key: CASSANDRA-5661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5661
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.2.1
> Reporter: Jonathan Ellis
> Assignee: Pavel Yaskevich
> Fix For: 1.2.7
>
> Attachments: CASSANDRA-5661.patch, DominatorTree.png, Histogram.png
>
>
> Reader pooling was introduced in CASSANDRA-4942 but pooled
> RandomAccessReaders are never cleaned up until the SSTableReader is closed.
> So memory use is "the worst case simultaneous RAR we had open for this file,
> forever."
> We should introduce a global limit on how much memory to use for RAR, and
> evict old ones.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
