[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125876#comment-17125876
]
Andres de la Peña commented on CASSANDRA-13606:
---
bq. Not sure why index needs to skip writes when its build failed...if no index
uses `LoadType read/no_op`, having `writableIndex` is giving extra overhead..
[~jasonstack] those writes will be replayed during rebuild, so it seems
pointless to process them twice. None current implementation make use of this,
but we might change that in a followup ticket, as an improvement. There's also
the idea of making the entire write fail, but it seems that that would have
more serious implications for availability, especially if the coordinator is
not aware of index availability.
bq. Perhaps we should unify both {{queryableIndex}} and {{writableIndex}} into
{{indexStatus}}, aka. {{Map}}.
Sounds like a good idea.
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125857#comment-17125857
]
ZhaoYang commented on CASSANDRA-13606:
--
bq. I'm not sure about what making the index not available for writes should
mean...
Not sure why index needs to skip writes when its build failed...if no index
uses `LoadType read/no_op`, having `writableIndex` is giving extra overhead..
Perhaps we should unify both {{`queryableIndex`}} and {{`writableIndex`}} into
{{`indexStatus`}}, aka. "Map".
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0-alpha5
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[ https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119513#comment-17119513 ] Andres de la Peña commented on CASSANDRA-13606: --- Committed to master as [595dc61290a9fda15b6765711141039ec9609bb3|https://github.com/apache/cassandra/commit/595dc61290a9fda15b6765711141039ec9609bb3]. Dtests changes committed as [b525ad4129a7963e270b74e8107887ef597cf0d6|https://github.com/apache/cassandra-dtest/commit/b525ad4129a7963e270b74e8107887ef597cf0d6]. > Improve handling of 2i initialization failures > -- > > Key: CASSANDRA-13606 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13606 > Project: Cassandra > Issue Type: Improvement > Components: Feature/2i Index >Reporter: Sergio Bossa >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0-beta > > Time Spent: 8h > Remaining Estimate: 0h > > CASSANDRA-10130 fixes the 2i build management, but initialization failures > are still not properly handled, most notably because: > * Initialization failures make the index non-queryable, but it can still be > written to. > * Initialization failures can be recovered via full rebuilds. > Both points above are probably suboptimal because the initialization logic > could be more complex than just an index build, hence it shouldn't be made > recoverable via a simple rebuild, and could cause the index to be fully > unavailable not just for reads, but for writes as well. > So, we should better handle initialization failures by: > * Allowing the index implementation to specify if unavailable for reads, > writes, or both. > * Providing a proper method to recover, distinct from index rebuilds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[ https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100877#comment-17100877 ] Berenguer Blasi commented on CASSANDRA-13606: - [~adelapena] I think I addressed your comments in the [PR|https://github.com/apache/cassandra/pull/570]. New CI runs are also included there. > Improve handling of 2i initialization failures > -- > > Key: CASSANDRA-13606 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13606 > Project: Cassandra > Issue Type: Improvement > Components: Feature/2i Index >Reporter: Sergio Bossa >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > CASSANDRA-10130 fixes the 2i build management, but initialization failures > are still not properly handled, most notably because: > * Initialization failures make the index non-queryable, but it can still be > written to. > * Initialization failures can be recovered via full rebuilds. > Both points above are probably suboptimal because the initialization logic > could be more complex than just an index build, hence it shouldn't be made > recoverable via a simple rebuild, and could cause the index to be fully > unavailable not just for reads, but for writes as well. > So, we should better handle initialization failures by: > * Allowing the index implementation to specify if unavailable for reads, > writes, or both. > * Providing a proper method to recover, distinct from index rebuilds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099981#comment-17099981
]
Berenguer Blasi commented on CASSANDRA-13606:
-
Adding a new param to {{rebuild_index}} might not be an option. I'll go for
your last suggestion of calling recovery if index failed during init, thx.
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099849#comment-17099849
]
Berenguer Blasi commented on CASSANDRA-13606:
-
[~adelapena]
- If I understood correctly your first point, I'll log writable (and queryable
while I am at it) index set
[here|https://github.com/apache/cassandra/pull/570/files#diff-3f2c8994c4ff8748c3faf7e70958520dR658]
and we silently ignore writes to a failed index. +1 on that.
- Correct, I left the 'recovery path' unused as I saw it more as a future
proofing thing than an actual replacement to the rebuild. I am -1 on adding a
{{recover_index}}. What if we add to {{rebuild_index}} a {{recover}} param that
defaults to false? That would be less 'noisy' and backwards compatible etc.
wdyt?
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099809#comment-17099809
]
Andres de la Peña commented on CASSANDRA-13606:
---
The index initialization failure is already logged, so I'm not sure whether it
would be useful to log every write to the base table that can't go into the
not-writable index. Perhaps we could the {{SIM.writableIndexes}} set instead of
{{SIM#indexes}} when we decide which indexes are going to be affected be a
write transaction, so we would just silently exclude not-writable indexes from
every write until we they have recovered.
As for providing a method to recover distinct from index rebuilds, and not
missing initialization work, I see two problems:
* The new {{Index#getRecoveryTaskSupport}} method is only called from
{{SIM#recoverIndexesBlocking}}, and I think the later is never called.
* Doing a full index rebuild with {{nodetool rebuild_index}} invokes
{{SIM#rebuildIndexesBlocking}}, that makes the index available for both reads
and writes without calling neither {{Index#getInitializationTask}} nor
{{Index#getRecoveryTaskSupport}}.
I can think of two different ways of differentiating the rebuild of a properly
initialized index from recovering from an initialization failure:
- Having a separate nodetool command, for example {{nodetool recover_index}},
that calls {{SIM#recoverIndexesBlocking}}, that calls the new
{{Index#getRecoveryTaskSupport}}.
- Keep the current {{nodetool rebuild_index}} command for both rebuild and
recovery, but use {{Index#getRecoveryTaskSupport}} under the hood if we know
that the index has failed during its initialization.
WDYT?
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[
https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099695#comment-17099695
]
Berenguer Blasi commented on CASSANDRA-13606:
-
[~adelapena] thx for reviewing this. I also was wondering about what a 'failed
built' index should do regarding writes. On reads it simply fails so I did the
same for writes. Yes it is very restrictive.
The alternative would be to ignore those writes as you state. What about
instead calling a {{writeRequestedWhileIndexFailed()}} method on the index
itself? That would allow the implementation to count, noSpamLog them, throw an
exception or whatever. wdyt?
> Improve handling of 2i initialization failures
> --
>
> Key: CASSANDRA-13606
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13606
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/2i Index
>Reporter: Sergio Bossa
>Assignee: Berenguer Blasi
>Priority: Normal
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> CASSANDRA-10130 fixes the 2i build management, but initialization failures
> are still not properly handled, most notably because:
> * Initialization failures make the index non-queryable, but it can still be
> written to.
> * Initialization failures can be recovered via full rebuilds.
> Both points above are probably suboptimal because the initialization logic
> could be more complex than just an index build, hence it shouldn't be made
> recoverable via a simple rebuild, and could cause the index to be fully
> unavailable not just for reads, but for writes as well.
> So, we should better handle initialization failures by:
> * Allowing the index implementation to specify if unavailable for reads,
> writes, or both.
> * Providing a proper method to recover, distinct from index rebuilds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[ https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099127#comment-17099127 ] Andres de la Peña commented on CASSANDRA-13606: --- If I'm understanding the patch, if we create a regular index on a table any write to that base table (including compaction) will fail until the index is completely build. I'm not sure we want this very restrictive behaviour, especially when the index build hasn't even failed. I guess we should have a different behaviour for unfinished and failed index builds, WDYT? I'm not sure about what making the index not available for writes should mean. The patch is making to fail even compactions of the base table, which might be excessive. Perhaps we should just ignore writes to a problematic index, without making the base table operation to fail? > Improve handling of 2i initialization failures > -- > > Key: CASSANDRA-13606 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13606 > Project: Cassandra > Issue Type: Improvement > Components: Feature/2i Index >Reporter: Sergio Bossa >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > CASSANDRA-10130 fixes the 2i build management, but initialization failures > are still not properly handled, most notably because: > * Initialization failures make the index non-queryable, but it can still be > written to. > * Initialization failures can be recovered via full rebuilds. > Both points above are probably suboptimal because the initialization logic > could be more complex than just an index build, hence it shouldn't be made > recoverable via a simple rebuild, and could cause the index to be fully > unavailable not just for reads, but for writes as well. > So, we should better handle initialization failures by: > * Allowing the index implementation to specify if unavailable for reads, > writes, or both. > * Providing a proper method to recover, distinct from index rebuilds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13606) Improve handling of 2i initialization failures
[ https://issues.apache.org/jira/browse/CASSANDRA-13606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094629#comment-17094629 ] Berenguer Blasi commented on CASSANDRA-13606: - PR [here|https://github.com/apache/cassandra/pull/570] > Improve handling of 2i initialization failures > -- > > Key: CASSANDRA-13606 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13606 > Project: Cassandra > Issue Type: Improvement > Components: Feature/2i Index >Reporter: Sergio Bossa >Assignee: Berenguer Blasi >Priority: Normal > Fix For: 4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > CASSANDRA-10130 fixes the 2i build management, but initialization failures > are still not properly handled, most notably because: > * Initialization failures make the index non-queryable, but it can still be > written to. > * Initialization failures can be recovered via full rebuilds. > Both points above are probably suboptimal because the initialization logic > could be more complex than just an index build, hence it shouldn't be made > recoverable via a simple rebuild, and could cause the index to be fully > unavailable not just for reads, but for writes as well. > So, we should better handle initialization failures by: > * Allowing the index implementation to specify if unavailable for reads, > writes, or both. > * Providing a proper method to recover, distinct from index rebuilds. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
