[
https://issues.apache.org/jira/browse/IGNITE-21017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-21017:
---------------------------------------
Description:
In IGNITE-21115, it is described how, after index appearance in the Catalog, we
wait for transactions started on schemas that did not yet have that index to be
finished. After this wait completes, we switch the index to the BACKFILLING
state.
The wait described above guarantees that the backfiller does not miss any
update related to a transaction coordinates by alive coordinators. But if some
coordinator leaves before its pre-index transactions finish, it's possible that
even after the index got switched to the BACKFILLING state, there are some
in-flight operations for those transactions from the coordinator-that-left on a
primary that is going to build the index.
This means that a race is possible between those in-flight operations writing
to the partition and the backfiller being started. If it starts before they
write, it might miss the rows they write. Those rows will never be indexed (as
the operations don't know about the index themselves and don't write to the
index). We will get an incomplete index.
To avoid this, we need to start building a secondary index of a specific
partition only *after* all in-flight update operations of RW transactions that
did not see the index have completed their operations.
In this task, we will add tracking of update operations for RW transactions for
partition.
was:
When creating a secondary index, it's possible to miss some rows while building
an index concurrently with a write load; as a result we'll get an incomplete
index.
This can happen due to a race between update operations of RW transactions and
index build start.
The transaction has not yet seen the new index and therefore will not make
updates to it. And the index builder simply skipped them. Thus, we will not
have a consistent index.
To avoid this, we need to start building a secondary index of a specific
partition only when all update operations of RW transactions have completed
their operations strictly after the activation time of the index creation event.
In this task, we will add tracking of update operations for RW transactions for
partition.
> Tracking RW transaction update operations
> -----------------------------------------
>
> Key: IGNITE-21017
> URL: https://issues.apache.org/jira/browse/IGNITE-21017
> Project: Ignite
> Issue Type: Improvement
> Reporter: Kirill Tkalenko
> Assignee: Kirill Tkalenko
> Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In IGNITE-21115, it is described how, after index appearance in the Catalog,
> we wait for transactions started on schemas that did not yet have that index
> to be finished. After this wait completes, we switch the index to the
> BACKFILLING state.
> The wait described above guarantees that the backfiller does not miss any
> update related to a transaction coordinates by alive coordinators. But if
> some coordinator leaves before its pre-index transactions finish, it's
> possible that even after the index got switched to the BACKFILLING state,
> there are some in-flight operations for those transactions from the
> coordinator-that-left on a primary that is going to build the index.
> This means that a race is possible between those in-flight operations writing
> to the partition and the backfiller being started. If it starts before they
> write, it might miss the rows they write. Those rows will never be indexed
> (as the operations don't know about the index themselves and don't write to
> the index). We will get an incomplete index.
> To avoid this, we need to start building a secondary index of a specific
> partition only *after* all in-flight update operations of RW transactions
> that did not see the index have completed their operations.
> In this task, we will add tracking of update operations for RW transactions
> for partition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
