[
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637535#comment-15637535
]
anmols commented on CASSANDRA-12796:
------------------------------------
I am able to reproduce this issue in Apache Cassandra 3.0.8 with a wide
partition and a secondary index defined over it.
The code has changed significantly between the version reported here and 3.0.8
however the characteristics of the failure are fairly similar, i.e. when a
secondary index is rebuild there is a build up of large number of pending
memtable flush runnables and the node gets overwhelmed and crashes due to an
OOM.
Adjusting the granule on which the write barrier applies (taking a pass with
the suggested patch's logic on the 3.0.8 code) does seem to alleviate the
problem and I do not see the memtable flush runnables queue up, however I am
not sure if there are other unintended consequences of tweaking this write
barrier granule which need to be considered.
I would like to know if this patch can be brought into Cassandra 3.0.x or are
there other solutions to deal with large partitions with secondary indexes?
> Heap exhaustion when rebuilding secondary index over a table with wide
> partitions
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Milan Majercik
> Priority: Critical
>
> We have a table with rather wide partition and a secondary index defined over
> it. As soon as we try to rebuild the index we observed exhaustion of Java
> heap and eventual OOM error. After a lengthy investigation we have managed to
> find a culprit which appears to be a wrong granule of barrier issuances in
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set<SecondaryIndex> indexes =
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator<ColumnFamily> pager = QueryPagers.pageRowLocally(cfs,
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table
> which poses a problem for wide partition tables as flush runnable
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed
> with flushing secondary index memtable before completing operations prior
> recent issue of the barrier. In our situation the flush runnable waits until
> whole wide partition gets indexed into the secondary index memtable before
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to
> table partition secondary index (see
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
> rebuild started to work without heap exhaustion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
