[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955660#comment-15955660
]
Jason Brown commented on CASSANDRA-13289:
-
[~aweisberg] +1
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954277#comment-15954277
]
Ariel Weisberg commented on CASSANDRA-13289:
I want to add some docs for the ideal consistency level metrics
https://github.com/apache/cassandra/compare/trunk...aweisberg:cassandra-13289-2?expand=1
[~jasobrown] can you take a look?
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949755#comment-15949755
]
Jason Brown commented on CASSANDRA-13289:
-
[~aweisberg] has updated his branch with some improvements and a nice unit
test.
+1, ship it
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947636#comment-15947636
]
Jason Brown commented on CASSANDRA-13289:
-
{{StorageProxyMBean#setIdealConsistencyLevel}} already [returns a
{{String}}|https://github.com/apache/cassandra/compare/trunk...aweisberg:cassandra-13289?expand=1#diff-097eb77c5d9d80a48e7547fbb81822caR70].
The implementation in {{StorageProxy}} simply performs {[return
original.toString();}} as the method return value.
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947370#comment-15947370
]
Ariel Weisberg commented on CASSANDRA-13289:
bq. StorageProxy#setIdealConsistencyLevel returns the previous value of the
ideal_consistency_level. Would that be confusing to operators who execute the
JMX command? maybe return "updating setIdealConsistencyLevel, previous value
was "?
How do I return "updating setIdealConsistencyLevel, previous value was
"? You mean change the return value to a string?
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947063#comment-15947063
]
Jason Brown commented on CASSANDRA-13289:
-
I'm pretty much +1 on the patch, except for the
{{AbstractWriteResponseHandler#responsesAndExpirations}} extra garbage and two
trivial nits, which you can feel free to completely ignore:
- maybe use the javacdoc comment style on {{Config#ideal_consistency_level}}
rather than the line comment style?
- {{StorageProxy#setIdealConsistencyLevel}} returns the previous value of the
{{ideal_consistency_level}}. Would that be confusing to operators who execute
the JMX command? maybe return "updating setIdealConsistencyLevel, previous
value was "?
The last thing is testing: i think it's possible to add unit tests to confirm
most of the behaviors here. Can you take a look into that?
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946991#comment-15946991
]
Jason Brown commented on CASSANDRA-13289:
-
bq. There is a part of me that doesn't like having a null field
Maybe set the type of the field to {{Optional}}, and set it to
{{Optional.empty}} when not being used? Not sure I love this but it does avoid
an NPE.
bq. I don't want to throw an error at the request level. I can't validate the
configuration. And I don't want to silently not increment the counters.
I agree with these points, especially the last. I suspect not counting would be
confusing to an operator who went out of their way to enable this feature, and
are expecting an output of some kind. I don't think logging is necessary (or
helpful).
wrt timeouts, you are correct. I think I was forgetting that this patch is only
affecting the write path, and I must have been thinking the read path was
pulled along, as well. Either way, this concern is not a real issue.
I'll rereview the ticket now with these things in mind.
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933610#comment-15933610
]
Ariel Weisberg commented on CASSANDRA-13289:
bq. maybe only instantiate AbstractWriteResponseHandler#responsesAndExpirations
in #setIdealCLResponseHandler(), and thus only create the AtomicInteger when
you know you are actually going to use it.
Sure. There is a part of me that doesn't like having a null field, but then
again the NPE is asserting something important.
bq. if the ideal CL and the requested CL are the same, should we even bother
capturing metrics about it? I'm kinda mixed on it...
bq. what happens if the user mixes non-CAS consistency levels with CAS
consistency levels (or vice versa)? I think the behavior will be correct (we
won't inadvertantly violate paxos semantics), but the semantic difference
between CAS and non-CAS requests might not be meaningful. So perhaps ignore the
idealCl if the CL types are different? wdyt?
So we could try and block people from doing combinations of things that don't
make sense or aren't useful, but what is the penalty if they do? Their system
will continue running they just won't get anything useful for this metric.
When you think about it for the purpose of what this is measuring CAS and
non-CAS are the same. Only the commit will have a write response handler and
SERIAL == QUORUM and LOCAL_SERIAL == LOCAL_QURUM.
I am not sure there are invalid values other then ideal == current. The problem
is that the error occurs at request time not configuration time so I can't
validate the configuration because I don't know what the CL of subsequent
requests will be. I could not count, but the operator asked me to do something
and even if it looks useless maybe they still expect the metrics to be accurate?
I am in favor of providing mechanism and not policy in these cases. I don't
want to throw an error at the request level. I can't validate the
configuration. And I don't want to silently not increment the counters. The
only other viable alternative is maybe a rate limited log warning.
bq. how will timed out message metrics be affected? We create an entry in
MessagingService#callbacks for each peer contacted for an operation (just
talking reads/mutations right now), and say the request CL is satisfied, but
the idealCL doesn't hear back from some nodes. In that case we'll increment the
timeouts, ConnectionMetrics.totalTimeouts.mark(), even though they weren't
explicitly part of the user's request. It might be confusing to users or
operators. I'm not sure how hard it is to code around that, or if it's
worthwhile. If we feel it's not, perhaps we just document it in the yaml that
"you may see higher than usual timeout counts". Thoughts?
This doesn't impact how timeouts are counted or callbacks are registered. All
this does is hook in and maintain a separate set of metrics for ideal CL. It's
operating within the existing callback that is already being registered and
timed out. It doesn't register additional callbacks. There should be no impact
on existing metrics.
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905541#comment-15905541
]
Jason Brown commented on CASSANDRA-13289:
-
Some thoughts:
- maybe only instantiate
{{AbstractWriteResponseHandler#responsesAndExpirations}} in
{{#setIdealCLResponseHandler()}}, and thus only create the {{AtomicInteger}}
when you know you are actually going to use it.
- if the ideal CL and the requested CL are the same, should we even bother
capturing metrics about it? I'm kinda mixed on it...
- what happens if the user mixes non-CAS consistency levels with CAS
consistency levels (or vice versa)? I think the behavior will be correct (we
won't inadvertantly violate paxos semantics), but the semantic difference
between CAS and non-CAS requests might not be meaningful. So perhaps ignore the
idealCl if the CL types are different? wdyt?
- how will timed out message metrics be affected? We create an entry in
{{MessagingService#callbacks}} for each peer contacted for an operation (just
talking reads/mutations right now), and say the request CL is satisfied, but
the idealCL doesn't hear back from some nodes. In that case we'll increment the
timeouts, {{ConnectionMetrics.totalTimeouts.mark()}}, even though they weren't
explicitly part of the user's request. It might be confusing to users or
operators. I'm not sure how hard it is to code around that, or if it's
worthwhile. If we feel it's not, perhaps we just document it in the yaml that
"you may see higher than usual timeout counts". Thoughts?
- calling it "ideal consistency level" doesn't sound quite right. Maybe
something like "alternative" or "secondary" might work. It might be good to
point out that the emphasis here should be on discovering the latencies a
different CL would bring, and not necessarily the impact on data consistency
itself.
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898858#comment-15898858
]
Romain Hardouin commented on CASSANDRA-13289:
-
> Yes you can set it via JMX.
Great, thanks!
Typo in cassandra.yaml: {{requested by each each write}}
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Fix For: 4.0
>
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898263#comment-15898263
]
Ariel Weisberg commented on CASSANDRA-13289:
It doesn't add much overhead because we already track all the responses until
all of them arrive or the timeout fires. That said I was going to have this off
by default. Yes you can set it via JMX.
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13289) Make it possible to monitor an ideal consistency level separate from actual consistency level
[
https://issues.apache.org/jira/browse/CASSANDRA-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894255#comment-15894255
]
Romain Hardouin commented on CASSANDRA-13289:
-
Very interesting!
Do you have an idea of the overhead that this monitoring layer would add? If
this adds too much overhead we should be able to enable/disable it at runtime
via nodetool.
> Make it possible to monitor an ideal consistency level separate from actual
> consistency level
> -
>
> Key: CASSANDRA-13289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13289
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>
> As an operator there are several issues related to multi-datacenter
> replication and consistency you may want to have more information on from
> your production database.
> For instance. If your application writes at LOCAL_QUORUM how often are those
> writes failing to achieve EACH_QUORUM at other data centers. If you failed
> your application over to one of those data centers roughly how inconsistent
> might it be given the number of writes that didn't propagate since the last
> incremental repair?
> You might also want to know roughly what the latency of writes would be if
> you switched to a different consistency level. For instance you are writing
> at LOCAL_QUORUM and want to know what would happen if you switched to
> EACH_QUORUM.
> The proposed change is to allow an ideal_consistency_level to be specified in
> cassandra.yaml as well as get/set via JMX. If no ideal consistency level is
> specified no additional tracking is done.
> if an ideal consistency level is specified then the
> {{AbstractWriteResponesHandler}} will contain a delegate WriteResponseHandler
> that tracks whether the ideal consistency level is met before a write times
> out. It also tracks the latency for achieving the ideal CL of successful
> writes.
> These two metrics would be reported on a per keyspace basis.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
