[
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16152090#comment-16152090
]
ZhaoYang commented on CASSANDRA-12783:
--------------------------------------
bq. separateUpdate
It's for view rebuild.
bq. in the view write path to ensure normal updates only generate a single
mutation to avoid the case of splitting a MV update over multiple batches (I
think?)
For normal base update, existing method is to generate 1 collection of view
mutation and put it into batchlog to ensure eventual consistency that once base
mutation is applied, view mutation should eventually be applied. With the new
"active batches method", we could achieve the same eventual consistency
semantic and reduce memory footprint, eg. 2&3
bq. With the whole "active" batch method could we remove the separateUpdates
case and instead split it over multiple batches, however only set them to be
active once they are all written, and fail the write if any batch store errors?
We still have the problem in #2, all modified base data are held in memory. If
it's partition deletion, entire base partition is held in memory.
Perhaps we could make {{generateViewUpdates}} entirely based on iterator and
split large view mutations into small groups. In the case of view rebuild,
each element from the iterator is marked as active and can be shipped to view
immediately. In the case of normal base update, each element from the iterator
is marked as not active, till all are stored.
> Break up large MV mutations to prevent OOMs
> -------------------------------------------
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
> Reporter: Carl Yeksigian
> Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder
> because otherwise we would break the contract of the batchlog, where some
> mutations may be written and pushed out before the whole batch log has been
> saved.
> We would need to ensure that all of the updates make it to the batchlog
> before allowing the batchlog manager to try to replay them, but also before
> we start pushing out updates to the paired replicas.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
