[jira] [Commented] (CASSANDRA-13225) Best Consistency Level
[ https://issues.apache.org/jira/browse/CASSANDRA-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870607#comment-15870607 ] Connor Warrington commented on CASSANDRA-13225: --- I was looking at the statistics from the 9318 JIRA and that's pretty much what I was hoping to accomplish. Less mutations dropped and less hints written. Yes, I get we aren't waiting for all replicas but we kinda are because of the rate limiting feedback loop. > Best Consistency Level > -- > > Key: CASSANDRA-13225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13225 > Project: Cassandra > Issue Type: New Feature >Reporter: Connor Warrington >Priority: Minor > > When writing data into a cluster there are a few consistency levels to choose > from. When choosing the consistency level to write with you are making a > tradeoff between consistency and failover availability. If you choose > consistency level ALL then all replicas have to be up and when a write > succeeds all replicas received the write. If you choose consistency level > QUORUM then a quorum number of replicas have to be up and when a write > succeeds at quorum number of replicas received the write. The tradeoff comes > in when there are more then quorum nodes available for the write. We would > like a write to succeed only when all replicas that are up have received the > write. Hence the suggestion of best as a consistency level. This would be > available for the existing consistency levels. The main idea behind this > feature request is that we are okay with a replica going down (fault > tolerance) but when the cluster is in a good state we don't mind waiting for > all nodes to get the write. This would enable the writer to operate at speed > of the slowest node instead of potentially getting into a state where that > slow node gets even further behind. This would also enable back pressure to > be better propagated through the system as the slowest node likely has back > pressure which is trying to tell the client about but if we don't wait for > that node the writer loses that information. > Example scenarios: > If we have replication factor of 3: > ALL consistency means 3 replicas have to be up and 3 replicas have to > successfully get the write. > QUORUM consistency means 2 replicas have to be up and 2 replicas have to > successfully get the write. > BEST_QUORUM consistency means 2 replicas have be up and all up replicas have > to successfully get the write. > If 3 replicas are up with replication factor of 3: > ALL would succeed as all 3 replicas are up and would return success when all > 3 replicas get the write > QUORUM would succeed as all 3 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as all 3 replicas are up and would return success > when all 3 replicas get the write > If 2 replicas are up with replication factor of 3: > ALL would fail as only 2 replicas are up > QUORUM would succeed as 2 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as 2 replicas are up and would return success when > 2 replicas get the write -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13225) Best Consistency Level
[ https://issues.apache.org/jira/browse/CASSANDRA-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870593#comment-15870593 ] Sergio Bossa commented on CASSANDRA-13225: -- bq. Does the back-pressure strategy set at SLOW and a consistency level of Quorum mean that while we will only wait for 3 replicas out of 5 for the write to succeed we will overall be limiting the number of write requests to all replicas based on the slowest responses we are getting from the cluster? Yes. Specifically, the rate limiting is computed based on the ratio between sent messages and received responses. bq. Does this then mean that a large set of individual writes will be rate limited by the slowest write responses and we will in effect be waiting for all replicas even though each individual write is only waiting for Quorum replicas? Kinda. Keep in mind we're still not actually *waiting* for all replicas, but just rate limiting (so pausing) based on the feedback loop produced by messages sent and responses received. > Best Consistency Level > -- > > Key: CASSANDRA-13225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13225 > Project: Cassandra > Issue Type: New Feature >Reporter: Connor Warrington >Priority: Minor > > When writing data into a cluster there are a few consistency levels to choose > from. When choosing the consistency level to write with you are making a > tradeoff between consistency and failover availability. If you choose > consistency level ALL then all replicas have to be up and when a write > succeeds all replicas received the write. If you choose consistency level > QUORUM then a quorum number of replicas have to be up and when a write > succeeds at quorum number of replicas received the write. The tradeoff comes > in when there are more then quorum nodes available for the write. We would > like a write to succeed only when all replicas that are up have received the > write. Hence the suggestion of best as a consistency level. This would be > available for the existing consistency levels. The main idea behind this > feature request is that we are okay with a replica going down (fault > tolerance) but when the cluster is in a good state we don't mind waiting for > all nodes to get the write. This would enable the writer to operate at speed > of the slowest node instead of potentially getting into a state where that > slow node gets even further behind. This would also enable back pressure to > be better propagated through the system as the slowest node likely has back > pressure which is trying to tell the client about but if we don't wait for > that node the writer loses that information. > Example scenarios: > If we have replication factor of 3: > ALL consistency means 3 replicas have to be up and 3 replicas have to > successfully get the write. > QUORUM consistency means 2 replicas have to be up and 2 replicas have to > successfully get the write. > BEST_QUORUM consistency means 2 replicas have be up and all up replicas have > to successfully get the write. > If 3 replicas are up with replication factor of 3: > ALL would succeed as all 3 replicas are up and would return success when all > 3 replicas get the write > QUORUM would succeed as all 3 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as all 3 replicas are up and would return success > when all 3 replicas get the write > If 2 replicas are up with replication factor of 3: > ALL would fail as only 2 replicas are up > QUORUM would succeed as 2 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as 2 replicas are up and would return success when > 2 replicas get the write -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13225) Best Consistency Level
[ https://issues.apache.org/jira/browse/CASSANDRA-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870573#comment-15870573 ] Connor Warrington commented on CASSANDRA-13225: --- Looks like this JIRA ticket is what I should be looking at: https://issues.apache.org/jira/browse/CASSANDRA-9318 > Best Consistency Level > -- > > Key: CASSANDRA-13225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13225 > Project: Cassandra > Issue Type: New Feature >Reporter: Connor Warrington >Priority: Minor > > When writing data into a cluster there are a few consistency levels to choose > from. When choosing the consistency level to write with you are making a > tradeoff between consistency and failover availability. If you choose > consistency level ALL then all replicas have to be up and when a write > succeeds all replicas received the write. If you choose consistency level > QUORUM then a quorum number of replicas have to be up and when a write > succeeds at quorum number of replicas received the write. The tradeoff comes > in when there are more then quorum nodes available for the write. We would > like a write to succeed only when all replicas that are up have received the > write. Hence the suggestion of best as a consistency level. This would be > available for the existing consistency levels. The main idea behind this > feature request is that we are okay with a replica going down (fault > tolerance) but when the cluster is in a good state we don't mind waiting for > all nodes to get the write. This would enable the writer to operate at speed > of the slowest node instead of potentially getting into a state where that > slow node gets even further behind. This would also enable back pressure to > be better propagated through the system as the slowest node likely has back > pressure which is trying to tell the client about but if we don't wait for > that node the writer loses that information. > Example scenarios: > If we have replication factor of 3: > ALL consistency means 3 replicas have to be up and 3 replicas have to > successfully get the write. > QUORUM consistency means 2 replicas have to be up and 2 replicas have to > successfully get the write. > BEST_QUORUM consistency means 2 replicas have be up and all up replicas have > to successfully get the write. > If 3 replicas are up with replication factor of 3: > ALL would succeed as all 3 replicas are up and would return success when all > 3 replicas get the write > QUORUM would succeed as all 3 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as all 3 replicas are up and would return success > when all 3 replicas get the write > If 2 replicas are up with replication factor of 3: > ALL would fail as only 2 replicas are up > QUORUM would succeed as 2 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as 2 replicas are up and would return success when > 2 replicas get the write -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13225) Best Consistency Level
[ https://issues.apache.org/jira/browse/CASSANDRA-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870527#comment-15870527 ] Connor Warrington commented on CASSANDRA-13225: --- The back-pressure strategy looks like it was added in a newer version of Cassandra then I am on. It does look like it would help keep the cluster operating at it's peak performance but not allow it to go past peak performance. The intent behind the feature request was to limit the dropped mutations, and ultimately limit the hints written. If I modify the scenario as follows: If 4 replicas are up with replication factor of 5: ALL would fail as only 4 replicas are up QUORUM would succeed as 4 replicas are up and would return success when 3 replicas get the write BEST_QUORUM would succeed as 4 replicas are up and would return success when 4 replicas get the write Does the back-pressure strategy set at SLOW and a consistency level of Quorum mean that while we will only wait for 3 replicas out of 5 for the write to succeed we will overall be limiting the number of write requests to all replicas based on the slowest responses we are getting from the cluster? Does this then mean that a large set of individual writes will be rate limited by the slowest write responses and we will in effect be waiting for all replicas even though each individual write is only waiting for Quorum replicas? > Best Consistency Level > -- > > Key: CASSANDRA-13225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13225 > Project: Cassandra > Issue Type: New Feature >Reporter: Connor Warrington >Priority: Minor > > When writing data into a cluster there are a few consistency levels to choose > from. When choosing the consistency level to write with you are making a > tradeoff between consistency and failover availability. If you choose > consistency level ALL then all replicas have to be up and when a write > succeeds all replicas received the write. If you choose consistency level > QUORUM then a quorum number of replicas have to be up and when a write > succeeds at quorum number of replicas received the write. The tradeoff comes > in when there are more then quorum nodes available for the write. We would > like a write to succeed only when all replicas that are up have received the > write. Hence the suggestion of best as a consistency level. This would be > available for the existing consistency levels. The main idea behind this > feature request is that we are okay with a replica going down (fault > tolerance) but when the cluster is in a good state we don't mind waiting for > all nodes to get the write. This would enable the writer to operate at speed > of the slowest node instead of potentially getting into a state where that > slow node gets even further behind. This would also enable back pressure to > be better propagated through the system as the slowest node likely has back > pressure which is trying to tell the client about but if we don't wait for > that node the writer loses that information. > Example scenarios: > If we have replication factor of 3: > ALL consistency means 3 replicas have to be up and 3 replicas have to > successfully get the write. > QUORUM consistency means 2 replicas have to be up and 2 replicas have to > successfully get the write. > BEST_QUORUM consistency means 2 replicas have be up and all up replicas have > to successfully get the write. > If 3 replicas are up with replication factor of 3: > ALL would succeed as all 3 replicas are up and would return success when all > 3 replicas get the write > QUORUM would succeed as all 3 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as all 3 replicas are up and would return success > when all 3 replicas get the write > If 2 replicas are up with replication factor of 3: > ALL would fail as only 2 replicas are up > QUORUM would succeed as 2 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as 2 replicas are up and would return success when > 2 replicas get the write -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13225) Best Consistency Level
[ https://issues.apache.org/jira/browse/CASSANDRA-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15869855#comment-15869855 ] Sergio Bossa commented on CASSANDRA-13225: -- bq. If what you are worried about is overloading nodes and want to get all 3 acks back to slow down the writes, then you might want to take a look at using a back_pressure_strategy, this is exactly the case that was implemented for, reducing the number of dropped mutations when all nodes are up and there is a constant stream of writes coming in. [~Connor Warrington], I fully second [~jjordan]'s suggestion here. The main problem with a "variable" CL is that under failure conditions it doesn't really give you any guarantees, as it will succeed if both quorum or "more than quorum" are met. Hence, it's practically equal to QUORUM in terms of guarantees. Configuring the back-pressure strategy at SLOW will probably be a better way to reduce the number of dropped mutations *and* allow hints to readily fix the ones you'll still get. > Best Consistency Level > -- > > Key: CASSANDRA-13225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13225 > Project: Cassandra > Issue Type: New Feature >Reporter: Connor Warrington >Priority: Minor > > When writing data into a cluster there are a few consistency levels to choose > from. When choosing the consistency level to write with you are making a > tradeoff between consistency and failover availability. If you choose > consistency level ALL then all replicas have to be up and when a write > succeeds all replicas received the write. If you choose consistency level > QUORUM then a quorum number of replicas have to be up and when a write > succeeds at quorum number of replicas received the write. The tradeoff comes > in when there are more then quorum nodes available for the write. We would > like a write to succeed only when all replicas that are up have received the > write. Hence the suggestion of best as a consistency level. This would be > available for the existing consistency levels. The main idea behind this > feature request is that we are okay with a replica going down (fault > tolerance) but when the cluster is in a good state we don't mind waiting for > all nodes to get the write. This would enable the writer to operate at speed > of the slowest node instead of potentially getting into a state where that > slow node gets even further behind. This would also enable back pressure to > be better propagated through the system as the slowest node likely has back > pressure which is trying to tell the client about but if we don't wait for > that node the writer loses that information. > Example scenarios: > If we have replication factor of 3: > ALL consistency means 3 replicas have to be up and 3 replicas have to > successfully get the write. > QUORUM consistency means 2 replicas have to be up and 2 replicas have to > successfully get the write. > BEST_QUORUM consistency means 2 replicas have be up and all up replicas have > to successfully get the write. > If 3 replicas are up with replication factor of 3: > ALL would succeed as all 3 replicas are up and would return success when all > 3 replicas get the write > QUORUM would succeed as all 3 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as all 3 replicas are up and would return success > when all 3 replicas get the write > If 2 replicas are up with replication factor of 3: > ALL would fail as only 2 replicas are up > QUORUM would succeed as 2 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as 2 replicas are up and would return success when > 2 replicas get the write -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13225) Best Consistency Level
[
https://issues.apache.org/jira/browse/CASSANDRA-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868807#comment-15868807
]
Jeff Jirsa commented on CASSANDRA-13225:
I'll add a third "sounds like a bad idea" vote. Downgrading consistency on
retry, back-pressure via app/driver, or
just-use-quorums-and-let-hints-do-their-job all sound better than
{{BEST_QUORUM}} .
> Best Consistency Level
> --
>
> Key: CASSANDRA-13225
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13225
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Connor Warrington
>Priority: Minor
>
> When writing data into a cluster there are a few consistency levels to choose
> from. When choosing the consistency level to write with you are making a
> tradeoff between consistency and failover availability. If you choose
> consistency level ALL then all replicas have to be up and when a write
> succeeds all replicas received the write. If you choose consistency level
> QUORUM then a quorum number of replicas have to be up and when a write
> succeeds at quorum number of replicas received the write. The tradeoff comes
> in when there are more then quorum nodes available for the write. We would
> like a write to succeed only when all replicas that are up have received the
> write. Hence the suggestion of best as a consistency level. This would be
> available for the existing consistency levels. The main idea behind this
> feature request is that we are okay with a replica going down (fault
> tolerance) but when the cluster is in a good state we don't mind waiting for
> all nodes to get the write. This would enable the writer to operate at speed
> of the slowest node instead of potentially getting into a state where that
> slow node gets even further behind. This would also enable back pressure to
> be better propagated through the system as the slowest node likely has back
> pressure which is trying to tell the client about but if we don't wait for
> that node the writer loses that information.
> Example scenarios:
> If we have replication factor of 3:
> ALL consistency means 3 replicas have to be up and 3 replicas have to
> successfully get the write.
> QUORUM consistency means 2 replicas have to be up and 2 replicas have to
> successfully get the write.
> BEST_QUORUM consistency means 2 replicas have be up and all up replicas have
> to successfully get the write.
> If 3 replicas are up with replication factor of 3:
> ALL would succeed as all 3 replicas are up and would return success when all
> 3 replicas get the write
> QUORUM would succeed as all 3 replicas are up and would return success when 2
> replicas get the write
> BEST_QUORUM would succeed as all 3 replicas are up and would return success
> when all 3 replicas get the write
> If 2 replicas are up with replication factor of 3:
> ALL would fail as only 2 replicas are up
> QUORUM would succeed as 2 replicas are up and would return success when 2
> replicas get the write
> BEST_QUORUM would succeed as 2 replicas are up and would return success when
> 2 replicas get the write
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13225) Best Consistency Level
[ https://issues.apache.org/jira/browse/CASSANDRA-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868797#comment-15868797 ] Jeremiah Jordan commented on CASSANDRA-13225: - If you app works correctly when this happens and the 3rd node comes up again: bq. BEST_QUORUM would succeed as 2 replicas are up and would return success when 2 replicas get the write Then you should just use QUORUM. You are getting no advantage from "BEST_QUORUM". Its not like we don't write the data to all 3 nodes when they are all up. If what you are worried about is overloading your nodes and want to get all 3 acks back to slow down your writes, then you might want to take a look at using a back_pressure_strategy. > Best Consistency Level > -- > > Key: CASSANDRA-13225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13225 > Project: Cassandra > Issue Type: New Feature >Reporter: Connor Warrington >Priority: Minor > > When writing data into a cluster there are a few consistency levels to choose > from. When choosing the consistency level to write with you are making a > tradeoff between consistency and failover availability. If you choose > consistency level ALL then all replicas have to be up and when a write > succeeds all replicas received the write. If you choose consistency level > QUORUM then a quorum number of replicas have to be up and when a write > succeeds at quorum number of replicas received the write. The tradeoff comes > in when there are more then quorum nodes available for the write. We would > like a write to succeed only when all replicas that are up have received the > write. Hence the suggestion of best as a consistency level. This would be > available for the existing consistency levels. The main idea behind this > feature request is that we are okay with a replica going down (fault > tolerance) but when the cluster is in a good state we don't mind waiting for > all nodes to get the write. This would enable the writer to operate at speed > of the slowest node instead of potentially getting into a state where that > slow node gets even further behind. This would also enable back pressure to > be better propagated through the system as the slowest node likely has back > pressure which is trying to tell the client about but if we don't wait for > that node the writer loses that information. > Example scenarios: > If we have replication factor of 3: > ALL consistency means 3 replicas have to be up and 3 replicas have to > successfully get the write. > QUORUM consistency means 2 replicas have to be up and 2 replicas have to > successfully get the write. > BEST_QUORUM consistency means 2 replicas have be up and all up replicas have > to successfully get the write. > If 3 replicas are up with replication factor of 3: > ALL would succeed as all 3 replicas are up and would return success when all > 3 replicas get the write > QUORUM would succeed as all 3 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as all 3 replicas are up and would return success > when all 3 replicas get the write > If 2 replicas are up with replication factor of 3: > ALL would fail as only 2 replicas are up > QUORUM would succeed as 2 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as 2 replicas are up and would return success when > 2 replicas get the write -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13225) Best Consistency Level
[ https://issues.apache.org/jira/browse/CASSANDRA-13225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868751#comment-15868751 ] Nate McCall commented on CASSANDRA-13225: - I'm not sure I quite understand the use case for BEST_QUORUM over QUORUM in your examples. Are you saying that you want a CL that defines fallback behavior depending on availability? If that is the case, this type of logic is best handled at the application level. For example, the JavaDriver project can be configured to use a "downgrading consistency" retry policy (http://docs.datastax.com/en/developer/java-driver/3.1/manual/retries/). > Best Consistency Level > -- > > Key: CASSANDRA-13225 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13225 > Project: Cassandra > Issue Type: New Feature >Reporter: Connor Warrington >Priority: Minor > > When writing data into a cluster there are a few consistency levels to choose > from. When choosing the consistency level to write with you are making a > tradeoff between consistency and failover availability. If you choose > consistency level ALL then all replicas have to be up and when a write > succeeds all replicas received the write. If you choose consistency level > QUORUM then a quorum number of replicas have to be up and when a write > succeeds at quorum number of replicas received the write. The tradeoff comes > in when there are more then quorum nodes available for the write. We would > like a write to succeed only when all replicas that are up have received the > write. Hence the suggestion of best as a consistency level. This would be > available for the existing consistency levels. The main idea behind this > feature request is that we are okay with a replica going down (fault > tolerance) but when the cluster is in a good state we don't mind waiting for > all nodes to get the write. This would enable the writer to operate at speed > of the slowest node instead of potentially getting into a state where that > slow node gets even further behind. This would also enable back pressure to > be better propagated through the system as the slowest node likely has back > pressure which is trying to tell the client about but if we don't wait for > that node the writer loses that information. > Example scenarios: > If we have replication factor of 3: > ALL consistency means 3 replicas have to be up and 3 replicas have to > successfully get the write. > QUORUM consistency means 2 replicas have to be up and 2 replicas have to > successfully get the write. > BEST_QUORUM consistency means 2 replicas have be up and all up replicas have > to successfully get the write. > If 3 replicas are up with replication factor of 3: > ALL would succeed as all 3 replicas are up and would return success when all > 3 replicas get the write > QUORUM would succeed as all 3 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as all 3 replicas are up and would return success > when all 3 replicas get the write > If 2 replicas are up with replication factor of 3: > ALL would fail as only 2 replicas are up > QUORUM would succeed as 2 replicas are up and would return success when 2 > replicas get the write > BEST_QUORUM would succeed as 2 replicas are up and would return success when > 2 replicas get the write -- This message was sent by Atlassian JIRA (v6.3.15#6346)
