[
https://issues.apache.org/jira/browse/CASSANDRA-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385657#comment-15385657
]
Sylvain Lebresne commented on CASSANDRA-12245:
----------------------------------------------
bq. just iterate through sstables, not worrying about duplicates, and include
the timestamp of the original write in the MV mutation.
I could misunderstand this but if you talking about dealing with each sstable
completely individually, I don't think we can do that. It's not that we have to
merge sstables to avoid dealing with duplicates, it's we need to current full
version of a row to create the proper MV entry. For instance, say you have a
first insert to a row in one sstable that contains the following columns/values
{{c1=2, c2=3, c3='foo'}}, and a later update to that row in another sstable
containing {{c1=4}}. And further say that {{c1}} is part of your view PK. If we
deal with both sstable individually, in the first case we'll insert an entry
for {{c1=2}}, with the other value, while for the 2nd sstable we'll create a
*different* entry for {{c1=4}} with no other values. Which is doubly wrong as
the {{c1=2}} entry shouldn't exist, and the {{c1=4}} entry should have the
other values for {{c2}} and {{c3}}.
With that said, several weeks to build a view over 3TB is indeed not ideal, and
we can indeed at least do vnodes in parallel.
> initial view build can be parallel
> ----------------------------------
>
> Key: CASSANDRA-12245
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12245
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Tom van der Woerdt
>
> On a node with lots of data (~3TB) building a materialized view takes several
> weeks, which is not ideal. It's doing this in a single thread.
> There are several potential ways this can be optimized :
> * do vnodes in parallel, instead of going through the entire range in one
> thread
> * just iterate through sstables, not worrying about duplicates, and include
> the timestamp of the original write in the MV mutation. since this doesn't
> exclude duplicates it does increase the amount of work and could temporarily
> surface ghost rows (yikes) but I guess that's why they call it eventual
> consistency. doing it this way can avoid holding references to all tables on
> disk, allows parallelization, and removes the need to check other sstables
> for existing data. this is essentially the 'do a full repair' path
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
