[jira] [Commented] (CASSANDRA-12956) CL is not replayed on custom 2i exception
[ https://issues.apache.org/jira/browse/CASSANDRA-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736553#comment-15736553 ] Jeremiah Jordan commented on CASSANDRA-12956: - What are the fix versions for this? > CL is not replayed on custom 2i exception > - > > Key: CASSANDRA-12956 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12956 > Project: Cassandra > Issue Type: Bug >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Critical > > If during the node shutdown / drain the custom (non-cf) 2i throws an > exception, CommitLog will get correctly preserved (segments won't get > discarded because segment tracking is correct). > However, when it gets replayed on node startup, we're making a decision > whether or not to replay the commit log. CL segment starts getting replayed, > since there are non-discarded segments and during this process we're checking > whether there every [individual > mutation|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L215] > in commit log is already committed or no. Information about the sstables is > taken from [live sstables on > disk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L250-L256]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12956) CL is not replayed on custom 2i exception
[ https://issues.apache.org/jira/browse/CASSANDRA-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725201#comment-15725201 ] Branimir Lambov commented on CASSANDRA-12956: - Thanks, committed as 6f90e55e7e23cbe814a3232c8d1ec67f2ff2a537. > CL is not replayed on custom 2i exception > - > > Key: CASSANDRA-12956 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12956 > Project: Cassandra > Issue Type: Bug >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Critical > > If during the node shutdown / drain the custom (non-cf) 2i throws an > exception, CommitLog will get correctly preserved (segments won't get > discarded because segment tracking is correct). > However, when it gets replayed on node startup, we're making a decision > whether or not to replay the commit log. CL segment starts getting replayed, > since there are non-discarded segments and during this process we're checking > whether there every [individual > mutation|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L215] > in commit log is already committed or no. Information about the sstables is > taken from [live sstables on > disk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L250-L256]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12956) CL is not replayed on custom 2i exception
[
https://issues.apache.org/jira/browse/CASSANDRA-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722609#comment-15722609
]
Alex Petrov commented on CASSANDRA-12956:
-
Great, thank you.
I've removed the duplicate {{await}}, thanks for catching that. {{truncate}}
check for {{false}} is done in [flush
memtable|https://github.com/apache/cassandra/compare/trunk...ifesdjeen:12956-3.X#diff-98f5acb96aa6d684781936c141132e2aR1097].
I've applied the change to all branches and did CI.
> CL is not replayed on custom 2i exception
> -
>
> Key: CASSANDRA-12956
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12956
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Critical
>
> If during the node shutdown / drain the custom (non-cf) 2i throws an
> exception, CommitLog will get correctly preserved (segments won't get
> discarded because segment tracking is correct).
> However, when it gets replayed on node startup, we're making a decision
> whether or not to replay the commit log. CL segment starts getting replayed,
> since there are non-discarded segments and during this process we're checking
> whether there every [individual
> mutation|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L215]
> in commit log is already committed or no. Information about the sstables is
> taken from [live sstables on
> disk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L250-L256].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12956) CL is not replayed on custom 2i exception
[
https://issues.apache.org/jira/browse/CASSANDRA-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15714555#comment-15714555
]
Branimir Lambov commented on CASSANDRA-12956:
-
Thanks, much cleaner and safer indeed. One small issue and a nit:
- The [2i
flush|https://github.com/apache/cassandra/compare/trunk...ifesdjeen:12956-3.X#diff-98f5acb96aa6d684781936c141132e2aR1082]
should only be done if {{truncate}} is false.
- The [barrier
await|https://github.com/apache/cassandra/compare/trunk...ifesdjeen:12956-3.X#diff-98f5acb96aa6d684781936c141132e2aR1130]
is not necessary as the flush does not commence [until that has
happened|https://github.com/apache/cassandra/compare/trunk...ifesdjeen:12956-3.X#diff-98f5acb96aa6d684781936c141132e2aR1071].
> CL is not replayed on custom 2i exception
> -
>
> Key: CASSANDRA-12956
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12956
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Critical
>
> If during the node shutdown / drain the custom (non-cf) 2i throws an
> exception, CommitLog will get correctly preserved (segments won't get
> discarded because segment tracking is correct).
> However, when it gets replayed on node startup, we're making a decision
> whether or not to replay the commit log. CL segment starts getting replayed,
> since there are non-discarded segments and during this process we're checking
> whether there every [individual
> mutation|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L215]
> in commit log is already committed or no. Information about the sstables is
> taken from [live sstables on
> disk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L250-L256].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12956) CL is not replayed on custom 2i exception
[
https://issues.apache.org/jira/browse/CASSANDRA-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712957#comment-15712957
]
Alex Petrov commented on CASSANDRA-12956:
-
Thanks for the review!
I was also skeptical about two threads waiting for one another. Also, tried the
approach you've suggested.
I hesitated mostly because it'd be blocking the flush thread (although you're
right that it's going to be
waiting for flushes anyways) and because {{flushMemtable}} is called from loop,
so I wasn't sure if it's
a good place.
In retrospect, I think your suggestion is much better than the previous
version. I've re-implemented a patch
for 3.0 as well, it got much simpler. Now, we do 2i flush before memtable flush
during flush of "data" memtable
(first one). We could bring back changes that expose sstable writer and run 2i
flush on the other
thread and commit sstable only on successful 2i flush, although since all cf
memtables are flushed sequentially
anyways and it might be a bit out of scope of the bugfix I decided to leave it
this way. Simply running 2i
flush in a different thread is not enough, as we need to ensure it's in sync
with "data" memtable flush.
Order of 2i/memtable flush does not matter, as for 2i it's only important that
data is present either in
sstable or in memtable. We can have the following situations: flush running
(memtable is queried), flush
successfull (sstable is queried), flush unsuccessful (memtable is queried),
flush unsuccessful + node restarted
(CL will replay the data and it'll be available from memtable again). So 3.0
patch relies on this behaviour.
|[3.0-v2|https://github.com/ifesdjeen/cassandra/tree/12956-3.0-v2-v2]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-3.0-v2-v2-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-3.0-v2-v2-dtest/]|
|[3.X|https://github.com/ifesdjeen/cassandra/tree/12956-3.X]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-3.X-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-3.X-dtest/]|
|[trunk|https://github.com/ifesdjeen/cassandra/tree/12956-trunk]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-trunk-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-trunk-dtest/]|
> CL is not replayed on custom 2i exception
> -
>
> Key: CASSANDRA-12956
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12956
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alex Petrov
>Assignee: Alex Petrov
>Priority: Critical
>
> If during the node shutdown / drain the custom (non-cf) 2i throws an
> exception, CommitLog will get correctly preserved (segments won't get
> discarded because segment tracking is correct).
> However, when it gets replayed on node startup, we're making a decision
> whether or not to replay the commit log. CL segment starts getting replayed,
> since there are non-discarded segments and during this process we're checking
> whether there every [individual
> mutation|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L215]
> in commit log is already committed or no. Information about the sstables is
> taken from [live sstables on
> disk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L250-L256].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12956) CL is not replayed on custom 2i exception
[ https://issues.apache.org/jira/browse/CASSANDRA-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711537#comment-15711537 ] Branimir Lambov commented on CASSANDRA-12956: - Looking at the 3.X patch, as flush threads waiting for post-flush waiting for flush threads is a recipe for disaster (poor performance and deadlocks in particular), I would much prefer the 2i flush to be done on a different thread. In particular, as the flush runnables doing the actual work proceed on their per-disk executors, the flush thread itself [here|https://github.com/apache/cassandra/compare/trunk...ifesdjeen:12956-3.X#diff-98f5acb96aa6d684781936c141132e2aR1152] looks like a better candidate. Could the inverse of the current problem also cause issues (i.e. 2i flushing without the data being in sstables)? It appears that the 2i flush also needs to eventually become transactional -- doing the above would make that easier too. > CL is not replayed on custom 2i exception > - > > Key: CASSANDRA-12956 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12956 > Project: Cassandra > Issue Type: Bug >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Critical > > If during the node shutdown / drain the custom (non-cf) 2i throws an > exception, CommitLog will get correctly preserved (segments won't get > discarded because segment tracking is correct). > However, when it gets replayed on node startup, we're making a decision > whether or not to replay the commit log. CL segment starts getting replayed, > since there are non-discarded segments and during this process we're checking > whether there every [individual > mutation|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L215] > in commit log is already committed or no. Information about the sstables is > taken from [live sstables on > disk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L250-L256]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12956) CL is not replayed on custom 2i exception
[ https://issues.apache.org/jira/browse/CASSANDRA-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709291#comment-15709291 ] Joshua McKenzie commented on CASSANDRA-12956: - [~blambov] to review. > CL is not replayed on custom 2i exception > - > > Key: CASSANDRA-12956 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12956 > Project: Cassandra > Issue Type: Bug >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Critical > > If during the node shutdown / drain the custom (non-cf) 2i throws an > exception, CommitLog will get correctly preserved (segments won't get > discarded because segment tracking is correct). > However, when it gets replayed on node startup, we're making a decision > whether or not to replay the commit log. CL segment starts getting replayed, > since there are non-discarded segments and during this process we're checking > whether there every [individual > mutation|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L215] > in commit log is already committed or no. Information about the sstables is > taken from [live sstables on > disk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L250-L256]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12956) CL is not replayed on custom 2i exception
[
https://issues.apache.org/jira/browse/CASSANDRA-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708453#comment-15708453
]
Alex Petrov commented on CASSANDRA-12956:
-
Patch for {{3.0}} is quite different and is much bigger. Main problem is that
there's no transactionality on the same level as in {{3.X}}. {{3.0}} memtables
are flushed and renamed to non-tmp names, readers are returned. We need a bit
better granularity, since after we may have to abort all the flushed sstables
if 2i failed.
Both patches introduce the second latch. I'm usually not the biggest fan of two
threads that have to wait for one another, but here the ordering is an issue.
Problem is that post-flush executor is single-threaded (for ordering), and
flush executor is multi-threaded, so we can't return future backed with that
multi-threaded executor as it will break order. On the other hand, if we move
2i flush to flush executor, we'll have to sequentially wait for 2i, then all
memtables. Current approach allows to keep these actions parallel.
We only need to synchronise the non-cf 2i flush with memtable holding data for
current cf. All the cf-index memtables will be in sync with data one anyways
since they're combined in the transaction.
|[3.X|https://github.com/ifesdjeen/cassandra/tree/12956-3.X]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-3.X-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-3.X-dtest/]|
|[3.0|https://github.com/ifesdjeen/cassandra/tree/12956-3.0]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-3.0-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-3.0-dtest/]|
|[trunk|https://github.com/ifesdjeen/cassandra/tree/12956-trunk]|[utest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-trunk-testall/]|[dtest|https://cassci.datastax.com/view/Dev/view/ifesdjeen/job/ifesdjeen-12956-trunk-dtest/]|
> CL is not replayed on custom 2i exception
> -
>
> Key: CASSANDRA-12956
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12956
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alex Petrov
>Priority: Critical
>
> If during the node shutdown / drain the custom (non-cf) 2i throws an
> exception, CommitLog will get correctly preserved (segments won't get
> discarded because segment tracking is correct).
> However, when it gets replayed on node startup, we're making a decision
> whether or not to replay the commit log. CL segment starts getting replayed,
> since there are non-discarded segments and during this process we're checking
> whether there every [individual
> mutation|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L215]
> in commit log is already committed or no. Information about the sstables is
> taken from [live sstables on
> disk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L250-L256].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12956) CL is not replayed on custom 2i exception
[ https://issues.apache.org/jira/browse/CASSANDRA-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705324#comment-15705324 ] Alex Petrov commented on CASSANDRA-12956: - The problem is only present in Cassandra starting from 3.0. Versions before that will replay commit log despite the exception, possibly generating multiple indentical sstables. > CL is not replayed on custom 2i exception > - > > Key: CASSANDRA-12956 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12956 > Project: Cassandra > Issue Type: Bug >Reporter: Alex Petrov >Priority: Critical > > If during the node shutdown / drain the custom (non-cf) 2i throws an > exception, CommitLog will get correctly preserved (segments won't get > discarded because segment tracking is correct). > However, when it gets replayed on node startup, we're making a decision > whether or not to replay the commit log. CL segment starts getting replayed, > since there are non-discarded segments and during this process we're checking > whether there every [individual > mutation|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L215] > in commit log is already committed or no. Information about the sstables is > taken from [live sstables on > disk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L250-L256]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
