[
https://issues.apache.org/jira/browse/CASSANDRA-10130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034602#comment-16034602
]
Paulo Motta edited comment on CASSANDRA-10130 at 6/2/17 12:38 PM:
------------------------------------------------------------------
bq. I think in such case the new Index initialization task would take care of
indexing, so it doesn't really matter if the new index misses the sstable
notification. So, to cover against the specific race you mentioned, we can just
filter out in SIM#handleNotification(), when receiving the
SSTableAddedNotification, all indexes not marked as building, as we can assume
those missed the first notification because not yet registered (and being just
registered, the initialization task will eventually take care of any initial
indexing).
It's still possible that an index is created after the
{{SSTableAddedNotification}} but before all sstables are added to the tracker,
in this case the initial index rebuild will not index the sstables that were
added and filtering all indexes marked as building will not re-mark the new
index as building since it may be still doing the initial rebuild:
1. {{SSTableBeforeAddNotification}} is received by SIM
3. new index is created (it will not index the new sstables since they are not
in the tracker yet) and mark as building
4. New sstables are added to the tracker
5. {{SSTableAddeNotification}} is received and index rebuild of new sstables is
triggered - the index is not re-marked as building because it's already marked
as building by the index creation
6. original index rebuild is finished and index is marked as built
7. index rebuild of new sstables fail but index is marked as built
This case is extremely unlikely to happen but shows the fragility of this
approach, which may be prone to other races we're not aware of.
bq.I think the race above can be solved easily without adding columns to the
system table. Other than that, let's not forget the pendingBuilds counter was
needed to protect us not just against concurrent building of multiple indexes,
but also against concurrent building of multiple sstables "batches" for the
same index.
My suggestion was not to add new columns to the system table, but rather
overload the existing columns with the following semantics:
- When an index is created for the first time for a table add new entry
{{INSERT INTO system.IndexInfo (table_name, index_name) VALUES ("table",
"table")}}
- When new sstables are received remove this entry from the index while these
sstables are being indexed
- On restart, if this entry is not present it means index rebuild failure for
all indexes which need rebuild
To protect against concurrent building of multiple sstables "batches" for the
same index, we could still have a pendingBuilds counter but a single counter
for all indexes rather than one counter per index. For manual rebuilds we could
simply prevent a new index rebuild if there's one already running, since it
doesn't make much sense to do simultaneous full rebuilds of the same indexes.
In any case this is just a suggestion, I'm fine if we go with other approach
(such as saving which indices were marked on the first notification) or decide
this race is too unlikely to bother doing anything.
was (Author: pauloricardomg):
bq. I think in such case the new Index initialization task would take care of
indexing, so it doesn't really matter if the new index misses the sstable
notification. So, to cover against the specific race you mentioned, we can just
filter out in SIM#handleNotification(), when receiving the
SSTableAddedNotification, all indexes not marked as building, as we can assume
those missed the first notification because not yet registered (and being just
registered, the initialization task will eventually take care of any initial
indexing).
It's still possible that an index is created after the
{{SSTableAddedNotification}} but before all sstables are added to the tracker,
in this case the initial index rebuild will not index the sstables that were
added and filtering all indexes marked as building will not re-mark the new
index as building since it may be still doing the initial rebuild:
1. {{SSTableBeforeAddNotification}} is received by SIM
3. new index is created (it will not index the new sstables since they are not
in the tracker yet) and mark as building
4. New sstables are added to the tracker
5. {{SSTableAddeNotification}} is received and index rebuild of new sstables is
triggered - the index is not re-marked as building because it's already marked
as building by the index creation
6. original index rebuild is finished and index is marked as built
7. index rebuild of new sstables fail but index is marked as built
This case is extremely unlikely to happen but shows the fragility of this
approach, which may be prone to other races we're not aware of.
bq.I think the race above can be solved easily without adding columns to the
system table. Other than that, let's not forget the pendingBuilds counter was
needed to protect us not just against concurrent building of multiple indexes,
but also against concurrent building of multiple sstables "batches" for the
same index.
My suggestion was not to add new columns to the system table, but rather
overload the existing columns with the following semantics:
- When an index is created add new entry {{INSERT INTO system.IndexInfo
(table_name, index_name) VALUES ("table", "table")}}
- When new sstables are received remove this entry from the index while these
sstables are being indexed
- On restart, if this entry is not present it means index rebuild failure for
all indexes which need rebuild
To protect against concurrent building of multiple sstables "batches" for the
same index, we could still have a pendingBuilds counter but a single counter
for all indexes rather than one counter per index. For manual rebuilds we could
simply prevent a new index rebuild if there's one already running, since it
doesn't make much sense to do simultaneous full rebuilds of the same indexes.
In any case this is just a suggestion, I'm fine if we go with other approach
(such as saving which indices were marked on the first notification) or decide
this race is too unlikely to bother doing anything.
> Node failure during 2i update after streaming can have incomplete 2i when
> restarted
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-10130
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10130
> Project: Cassandra
> Issue Type: Bug
> Components: Coordination
> Reporter: Yuki Morishita
> Assignee: Andrés de la Peña
> Priority: Minor
>
> Since MV/2i update happens after SSTables are received, node failure during
> MV/2i update can leave received SSTables live when restarted while MV/2i are
> partially up to date.
> We can add some kind of tracking mechanism to automatically rebuild at the
> startup, or at least warn user when the node restarts.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
