[
https://issues.apache.org/jira/browse/CASSANDRA-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996210#comment-12996210
]
Jonathan Ellis commented on CASSANDRA-1902:
-------------------------------------------
bq. we use the minContiguousPages to filter down the noise of cache fragments
Doesn't this make it basically useless for skinny-row sstables, where rows
don't span multiple fragments and hot rows are scattered randomly throughout a
sstable with many cold rows?
I think we might need a more sophisticated approach to balance this vs the
increased memory usage of tracking individual pages.
Coming back to a pseudo-iterator approach: how about SSTableScanner grabbing
256MB or so of mincore info at a time, lazily during the iteration instead of
at construction time? (The smaller the unit of mincore computation, the less
stale our hotness data will be, which addresses Chris's concern too.) Then we
could track even single-page hotness ranges w/o worrying about memory pressure.
Additional note: not a huge fan of exposing the MAX_BYTES_IN_PAGE_CACHE as a
tunable, let's find out where diminishing returns set in (128MB apparently?)
and hardcode it up. My experience is that super obscure options like this are
not useful to users, it's our job to find a good value.
> Migrate cached pages during compaction
> ---------------------------------------
>
> Key: CASSANDRA-1902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1902
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.1
> Reporter: T Jake Luciani
> Assignee: T Jake Luciani
> Fix For: 0.7.3
>
> Attachments:
> 0001-CASSANDRA-1902-cache-migration-impl-with-config-option.txt
>
> Original Estimate: 32h
> Time Spent: 24h
> Remaining Estimate: 8h
>
> Post CASSANDRA-1470 there is an opportunity to migrate cached pages from a
> pre-compacted CF during the compaction process.
> First, add a method to MmappedSegmentFile: long[] pagesInPageCache() that
> uses the posix mincore() function to detect the offsets of pages for this
> file currently in page cache.
> Then add getActiveKeys() which uses underlying pagesInPageCache() to get the
> keys actually in the page cache.
> use getActiveKeys() to detect which SSTables being compacted are in the os
> cache and make sure the subsequent pages in the new compacted SSTable are
> kept in the page cache for these keys. This will minimize the impact of
> compacting a "hot" SSTable.
> A simpler yet similar approach is described here:
> http://insights.oetiker.ch/linux/fadvise/
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
