[
https://issues.apache.org/jira/browse/IGNITE-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko reassigned IGNITE-18539:
----------------------------------------
Assignee: Kirill Tkalenko
> Implement build procedure for new indexes
> -----------------------------------------
>
> Key: IGNITE-18539
> URL: https://issues.apache.org/jira/browse/IGNITE-18539
> Project: Ignite
> Issue Type: Improvement
> Reporter: Ivan Bessonov
> Assignee: Kirill Tkalenko
> Priority: Major
> Labels: ignite-3
>
> First of all, "build" is the process of moving the index from "WO" to "RW"
> state in schema definitions. Finished build creates a new schema version.
> This means that new transactions, started after a specific timestamp, will be
> able to use this index in their plans.
> Now, when index is created, we also create a table somewhere that looks
> similar to this:
>
> {code:java}
> indexComplation.<indexId>.0 = false
> ...
> indexComplation.<indexId>.N = false {code}
> It is used to track completion of index build for each individual partition.
> Once it's built bor each partition, WO -> RW state transition may occur.
>
> For each individual partition, leader/coordinator/leaseholder/whatever node
> starts iterating all rowIds in the partition and replicates the "index build
> command" for each row (or using batches). This procedure can be described
> with this pseudocode:
> {code:java}
> RowId curRowId = minRowId();
> // Get existing row id higher or equal than the parameter.
> curRowId = partition.findRowId(curRowId);
> while (curRowId != null) {
> // Put that into a state machine.
> replicateIndexRowRebuild(indexId, curRowId);
> // Compute the next row id candidate.
> curRowId = increment(curRowId);
> // Get the real next row id value.
> if (curRowId != null) {
> curRowId = partition.findRowId(curRowId);
> }
> }
> indexBuildCompleted(indexId, partitionId);{code}
> Such replication doesn't require too much traffic, and it guarantees
> consistency between replicas, which is crucial.
> Once index is created and WO, all insertions should update it
> "unconditionally", independently from the build process. This gives us
> cooperation and simplifies the way we guarantee consistency between row store
> and index.
> Easy optimization: during regular load, we can avoid updating WO indexes if
> rowId > lastRowId for that particular index, because it will be updated later
> anyway.
>
> About that: we should store "lastUpdatedRowId" for every index that's not yet
> completed its build. We need it for 2 reasons:
> * leader might be changed
> * cluster may be restarted
> In both cases we simply start where we stopped last time, instead of starting
> from scratch, like it was in Ignite 2.x.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
