[
https://issues.apache.org/jira/browse/CASSANDRA-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725358#comment-15725358
]
Milan Majercik commented on CASSANDRA-12796:
--------------------------------------------
[~beobal], the patch for branch {{3.0}} works fine, however the page size for
single partition pager appears to be calculated incorrectly. My table's average
partition size is around {{7GB}} and yet the page size got calculated as {{1}}.
{code:java}
private int calculateIndexingPageSize()
{
double averageRowSize = baseCfs.getMeanPartitionSize();
if (averageRowSize <= 0)
return DEFAULT_PAGE_SIZE;
return (int) Math.max(1, Math.min(DEFAULT_PAGE_SIZE, 4 * 1024 * 1024 /
averageRowSize));
}
{code}
This rendered index rebuild extremely slow as registering read/write order
group implies significant performance overhead and for this reason the page
size should have reasonable size.
I think there is no harm if we set page size to {{DEFAULT_PAGE_SIZE}} as the
pager doesn't span across different partitions in case the partition is small
([https://github.com/mmajercik/cassandra/commit/3fc016e73d3032f4d04584a45945141151a49213])
[12796-3.0|https://github.com/mmajercik/cassandra/tree/12796-3.0]
> Heap exhaustion when rebuilding secondary index over a table with wide
> partitions
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-12796
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12796
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Milan Majercik
> Priority: Critical
>
> We have a table with rather wide partition and a secondary index defined over
> it. As soon as we try to rebuild the index we observed exhaustion of Java
> heap and eventual OOM error. After a lengthy investigation we have managed to
> find a culprit which appears to be a wrong granule of barrier issuances in
> method {{org.apache.cassandra.db.Keyspace.indexRow}}:
> {code}
> try (OpOrder.Group opGroup = cfs.keyspace.writeOrder.start()){html}
> {
> Set<SecondaryIndex> indexes =
> cfs.indexManager.getIndexesByNames(idxNames);
> Iterator<ColumnFamily> pager = QueryPagers.pageRowLocally(cfs,
> key.getKey(), DEFAULT_PAGE_SIZE);
> while (pager.hasNext())
> {
> ColumnFamily cf = pager.next();
> ColumnFamily cf2 = cf.cloneMeShallow();
> for (Cell cell : cf)
> {
> if (cfs.indexManager.indexes(cell.name(), indexes))
> cf2.addColumn(cell);
> }
> cfs.indexManager.indexRow(key.getKey(), cf2, opGroup);
> }
> }
> {code}
> Please note the operation group granule is a partition of the source table
> which poses a problem for wide partition tables as flush runnable
> ({{org.apache.cassandra.db.ColumnFamilyStore.Flush.run()}}) won't proceed
> with flushing secondary index memtable before completing operations prior
> recent issue of the barrier. In our situation the flush runnable waits until
> whole wide partition gets indexed into the secondary index memtable before
> flushing it. This causes an exhaustion of the heap and eventual OOM error.
> After we changed granule of barrier issue in method
> {{org.apache.cassandra.db.Keyspace.indexRow}} to query page as opposed to
> table partition secondary index (see
> [https://github.com/mmajercik/cassandra/commit/7e10e5aa97f1de483c2a5faf867315ecbf65f3d6?diff=unified]),
> rebuild started to work without heap exhaustion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
