[
https://issues.apache.org/jira/browse/CASSANDRA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714635#action_12714635
]
Jonathan Ellis commented on CASSANDRA-208:
------------------------------------------
blockindexes_ keeps the indexes in memory as the flush or compaction is
performed, so they can be dumped at the end of the SSTable file.
The old code on code.google dumped each index at the end of its block which
avoided this problem (only one block's index would ever be in memory at once).
I'm not sure why FB changed this behavior. The only reason I can think of is
that reading the index later won't require as many seeks, but this seems like a
bad optimization to make since index reading is done once per sstable.
I couldn't tell at first why this should cause OOM -- in both the google and
apache versions we have indexMetadataMap_ which contains the same block index
info, right? Wrong, the difference is that iMM only contains the first entry
from each block index. So much less memory is used there.
I guess we should switch back to the old, interleaved block indexing method.
> jvm crashes intermittently during compaction
> --------------------------------------------
>
> Key: CASSANDRA-208
> URL: https://issues.apache.org/jira/browse/CASSANDRA-208
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: arch: x86_64
> os: Linux version 2.6.18-92.1.22.el5
> java: nio2-ea-bin-b99-linux-x64-05_feb_2009
> Reporter: Jiansheng Huang
>
> jvm crashes intermittently during compaction. Our test data set is not that
> big, less than 10 GB.
> When jvm is about to crash, we see that it consumes a lot of memory
> (exceeding the max heap size).
> The excessive memory usage during compaction is caused by the maintenance of
> blockIndexes_ in SSTable. this blockIndexes_ was only introduced to the
> apache version.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
