[
https://issues.apache.org/jira/browse/CASSANDRA-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-5506:
--------------------------------------
Reviewer: yukim
Priority: Major (was: Minor)
Fix Version/s: 1.2.5
Assignee: Jonathan Ellis
https://github.com/jbellis/cassandra/commits/5506 makes IndexSummary use a
long[] and byte[][] to save memory.
(I'm fairly confident the performance hit for re-decorating during index
lookups will be negligible, since we only have to do the lookup on cache miss.)
> Reduce memory consumption of IndexSummary
> -----------------------------------------
>
> Key: CASSANDRA-5506
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5506
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Nick Puz
> Assignee: Jonathan Ellis
> Fix For: 1.2.5
>
>
> I am evaluating cassandra for a use case with many tiny rows which would
> result in a node with 1-3TB of storage having billions of rows. Before
> loading that much data I am hitting GC issues and when looking at the heap
> dump I noticed that 70+% of the memory was used by IndexSummaries.
> The two major issues seem to be:
> 1) that the positions are stored as an ArrayList<Long> which results in each
> position taking 24 bytes (class + flags + 8 byte long). This might make sense
> when the file is initially written but once it has been serialized it would
> be a lot more memory efficient to just have an long[] (really a int[] would
> be fine unless 2GB sstables are allowed).
> 2) The DecoratedKey for a byte[16] key takes 195 bytes -- this is for the
> overhead of the ByteBuffer in the key and overhead in the token.
> To somewhat "work around" the problem I have increased index_sample but will
> this many rows that didn't really help starts to have diminishing returns.
> NOTE: This heap dump was from linux with a 64bit oracle vm.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
