[
https://issues.apache.org/jira/browse/CASSANDRA-12268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Yeksigian updated CASSANDRA-12268:
---------------------------------------
Attachment: 12268.py
I've added just the basics, and it does work for this.
I think we should probably add this fix as is; it will have inefficiencies for
internode communications, but it can handle a large partition. The other
enhancements can come after this. I'm running CI on the branch now.
|[3.0|https://github.com/carlyeks/cassandra/tree/ticket/12268-3.0]|[utest|https://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-12268-3.0-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-12268-3.0-dtest/]|
|[3.9|https://github.com/carlyeks/cassandra/tree/ticket/12268-3.9]|[utest|https://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-12268-3.9-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-12268-3.9-dtest/]|
|[trunk|https://github.com/carlyeks/cassandra/tree/ticket/12268]|[utest|https://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-12268-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/carlyeks/job/carlyeks-ticket-12268-dtest/]|
I'm also attaching the script that I used to test whether this worked or not -
the number of iterations might be able to brought down with a really small heap
size to reproduce the issue, but with a small-ish heap, this crashed without
the fix and ran with it.
> Make MV Index creation robust for wide referent rows
> ----------------------------------------------------
>
> Key: CASSANDRA-12268
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12268
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Jonathan Shook
> Assignee: Carl Yeksigian
> Fix For: 3.0.x, 3.x
>
> Attachments: 12268.py
>
>
> When creating an index for a materialized view for extant data, heap pressure
> is very dependent on the cardinality of of rows associated with each index
> value. With the way that per-index value rows are created within the index,
> this can cause unbounded heap pressure, which can cause OOM. This appears to
> be a side-effect of how each index row is applied atomically as with batches.
> The commit logs can accumulate enough during the process to prevent the node
> from being restarted. Given that this occurs during global index creation,
> this can happen on multiple nodes, making stable recovery of a node set
> difficult, as co-replicas become unavailable to assist in back-filling data
> from commitlogs.
> While it is understandable that you want to avoid having relatively wide rows
> even in materialized views, this represents a particularly difficult
> scenario for triage.
> The basic recommendation for improving this is to sub-group the index
> creation into smaller chunks internally, providing a maximal bound against
> the heap pressure when it is needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
