Re: ONE has much higher latency than LOCAL_ONE
Thanks for the link, I hadn't seen that before.
It's unfortunate that they don't explain what they mean by "closest replica".
The nodes in the remote DC should not be regarded as "closest". Also, it's not
clear what the black arrows mean… the coordinator sends the read to all three
replicas, but only one of them responds?
Reading further (assuming this article from 2012 is still accurate for
Cassandra 3.0
http://www.datastax.com/dev/blog/dynamic-snitching-in-cassandra-past-present-and-future),
it appears that by "closest replica" what they really mean is the replica
chosen by the "dynamic snitch". The structure of the documentation
https://docs.datastax.com/en/cassandra/3.0/cassandra/architecture/archSnitchesAbout.html
is misleading in this regard: it puts the "Dynamic snitching" section side by
side with the other snitch implementations, implying that it's one of the
choices you can configure as a snitch, which is why I hadn't read that section
(I didn't want a snitch that "monitors the performance of reads"). Instead, the
info about the dynamic snitch should be in the top-level page. In any case, the
dynamic snitch is apparently governed by read latency, the node's state, and
whether the node is doing a compaction ("severity"). So why is it routing
requests to nodes with latency that's ~20 times larger? I don't know, but I
wish it wasn't.
I guess it's important to differentiate between that and the load balancing
policy called LatencyAwarePolicy… even if you're not using the
LatencyAwarePolicy, internally the snitch is still doing stuff based on latency.
This is also unfortunate because it makes the DCAwareRoundRobinPolicy useless
for anything but local consistency levels, and (if you read it at face value)
contradicts the description in the documentation that "This policy queries
nodes of the local data-center in a round-robin fashion; optionally, it can
also try a configurable number of hosts in remote data centers if all local
hosts failed."
Also, if you're right that the requests are getting routed to the remote DC,
then those requests aren't showing up in my graph of read request rate… which
is problematic because I'm not getting an accurate representation of what's
actually happening. I can't find any other metric beyond
org.apache.cassandra.metrics.ClientRequest.* which might include these internal
read requests.
I am wondering if perhaps there's a mistake with the way that the dynamic
snitch measures latency… if it's only measuring requests coming from clients,
then if a remote node happens to win the dynamic snitch's favor momentarily,
the latency of the local node will increase (because it's querying the remote
node), and then the dynamic snitch will see that the local node is performing
poorly, and will continue directing traffic to the remote cluster. Or, perhaps
they're measuring the latency of each node based not on how long the client
request takes but based on how long the internal request takes… which, again,
could mislead the snitch into thinking that the remote host is providing a
better deal to the client than it really is. It seems like a mistake that the
dynamic switch would think that a remote node will be faster or less work than
the local node which actually contains a copy of the data being queried.
Looks like I'm not the only one who's run into this:
https://issues.apache.org/jira/browse/CASSANDRA-6908
I think I'm going to try setting the system property
"cassandra.ignore_dynamic_snitch_severity" to "true" and see what happens. That
or "dynamic_snitch: false"… it's not documented in
https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/configCassandra_yaml.html
but it appears to be a valid config option.
From: Eric Plowe >
Reply-To: "user@cassandra.apache.org"
>
Date: Wednesday, March 22, 2017 at 11:44 AM
To: "user@cassandra.apache.org"
>
Subject: Re: ONE has much higher latency than LOCAL_ONE
Yes, your request from the client is going to the LocalDC that you've defined
for the data center aware load balancing policy, but with a consistency level
of ONE, there is a chance for the coordinator (the node your client has
connected to) to route the request across DC's.
Please see:
https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dmlClientRequestsRead.html#dmlClientRequestsRead__two-dc-one
"A two datacenter cluster with a consistency level of ONE
"In a multiple datacenter cluster with a replication factor of 3, and a read
consistency of ONE, the closest replica for the given row, regardless of
datacenter, is contacted to fulfill the read request. In the background a read
repair is potentially initiated, based on the read_repair_chance setting of the
table,
Re: ONE has much higher latency than LOCAL_ONE
I understand all that, but it doesn't explain why the latency increases. The requests are not going to a remote DC. I know this because currently all requests are going to the client in one particular DC. The read request rate of the Cassandra nodes in the other DC remained flat (near zero) the whole time, compared to ~200read/s on the Cassandra nodes in the DC local to the client doing the reads. This is expected, because the DCAwareRoundRobinPolicy will cause local nodes to be used preferentially whenever possible. What's not expected is the dramatic latency increase. Btw this client is read-only: no writes. From: Eric Plowe <eric.pl...@gmail.com<mailto:eric.pl...@gmail.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Tuesday, March 21, 2017 at 7:23 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: ONE has much higher latency than LOCAL_ONE ONE means at least one replica node to ack the write, but doesn't require that the coordinator route the request to a node in the local data center. LOCAL_ONE was introduced to handle the case of when you have multiple data centers and cross data center traffic is not desirable. In multiple datacenter clusters, a consistency level of ONE is often desirable, but cross-DC traffic is not. LOCAL_ONEaccomplishes this. For security and quality reasons, you can use this consistency level in an offline datacenter to prevent automatic connection to online nodes in other datacenters if an offline node goes down. From: https://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dml_config_consistency_c.html Regards, Eric On Tue, Mar 21, 2017 at 7:49 PM Shannon Carey <sca...@expedia.com<mailto:sca...@expedia.com>> wrote: The cluster is in two DCs, and yes the client is deployed locally to each DC. From: Matija Gobec <matija0...@gmail.com<mailto:matija0...@gmail.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Tuesday, March 21, 2017 at 2:56 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: ONE has much higher latency than LOCAL_ONE Are you running a multi DC cluster? If yes do you have application in both data centers/regions ? On Tue, Mar 21, 2017 at 8:07 PM, Shannon Carey <sca...@expedia.com<mailto:sca...@expedia.com>> wrote: I am seeing unexpected behavior: consistency level ONE increases read latency 99th percentile to ~108ms (95th percentile to 5ms-90ms) up from ~5ms (99th percentile) when using LOCAL_ONE. I am using DSE 5.0 with Datastax client 3.0.0. The client is configured with a TokenAwarePolicy wrapping a DCAwareRoundRobinPolicy with usedHostsPerRemoteDc set to a very high number. Cassandra cluster has two datacenters. I would expect that when the cluster is operating normally (all local nodes reachable), ONE would behave the same as LOCAL_ONE. The Does anyone know why this is not the case?
Re: ONE has much higher latency than LOCAL_ONE
Yes, as I mentioned in my other thread, LOCAL_ONE does not allow the retry
policy to take action if all local nodes are down.
Yes, I am using withLocalDc(); Here's the code (Scala):
def getClusterBuilder: Builder = {
val pool = new PoolingOptions
pool.setConnectionsPerHost(HostDistance.LOCAL,
config.coreConnectionsPerHost, config.maxConnectionsPerHost)
val codecRegistry: CodecRegistry = new CodecRegistry()
.register(InstantCodec.instance)
.register(SimpleTimestampCodec.instance)
// By specifying MaxValue here, we allow any & all hosts in remote DCs to
be used by queries when necessary. That
// allows TokenAwarePolicy to choose the appropriate nodes in remote DCs.
val maxHostsToUsePerRemoteDc = Int.MaxValue
// We have nodes from the remote DC in the initial list (so that we can
tolerate DC failover), so we have to
// specify the local DC explicitly.
val dcAwarePolicy = DCAwareRoundRobinPolicy.builder()
.withLocalDc(config.localDc)
.withUsedHostsPerRemoteDc(maxHostsToUsePerRemoteDc)
.build()
val shuffleReplicas = true
val builder = Cluster.builder()
.withClusterName(config.clusterName)
.withPoolingOptions(pool)
.withLoadBalancingPolicy(new TokenAwarePolicy(dcAwarePolicy,
shuffleReplicas))
.withRetryPolicy(DowngradingConsistencyRetryPolicy.INSTANCE)
.withCodecRegistry(codecRegistry)
val contactPoints = config.contactPointsProvider.getContactPoints.asScala
contactPoints.foreach(builder.addContactPoint)
builder
}
I will have to turn up the logging in order to see that log message you refer
to. But it seems to me that the DC config is the same regardless of whether I
use ONE or LOCAL_ONE so I don't think it would make a difference. From what
I've seen, I'd expect all the non-local nodes to be listed in that message. But
I'll see what I can.
Thanks for your responses! I posted to the other list as you suggested:
https://groups.google.com/a/lists.datastax.com/forum/#!topic/java-driver-user/o0GVBjFCHCA
From: Nate McCall <n...@thelastpickle.com<mailto:n...@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Tuesday, March 21, 2017 at 7:16 PM
To: Cassandra Users
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: ONE has much higher latency than LOCAL_ONE
On Wed, Mar 22, 2017 at 1:11 PM, Nate McCall
<n...@thelastpickle.com<mailto:n...@thelastpickle.com>> wrote:
On Wed, Mar 22, 2017 at 12:48 PM, Shannon Carey
<sca...@expedia.com<mailto:sca...@expedia.com>> wrote:
>
> The cluster is in two DCs, and yes the client is deployed locally to each DC.
First off, what is the goal of using ONE instead of LOCAL_ONE? If it's
failover, this could be addressed with a RetryPolicy starting wth LOCAL_ONE and
falling back to ONE.
Just read your previous thread about this. That's pretty un-intuitive and
counter to the way I remember that working (though admittedly, it's been a
while).
Do please open a thread on the driver mailing list, i'm curious about the
response.
Re: ONE has much higher latency than LOCAL_ONE
The cluster is in two DCs, and yes the client is deployed locally to each DC. From: Matija Gobec <matija0...@gmail.com<mailto:matija0...@gmail.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Tuesday, March 21, 2017 at 2:56 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: ONE has much higher latency than LOCAL_ONE Are you running a multi DC cluster? If yes do you have application in both data centers/regions ? On Tue, Mar 21, 2017 at 8:07 PM, Shannon Carey <sca...@expedia.com<mailto:sca...@expedia.com>> wrote: I am seeing unexpected behavior: consistency level ONE increases read latency 99th percentile to ~108ms (95th percentile to 5ms-90ms) up from ~5ms (99th percentile) when using LOCAL_ONE. I am using DSE 5.0 with Datastax client 3.0.0. The client is configured with a TokenAwarePolicy wrapping a DCAwareRoundRobinPolicy with usedHostsPerRemoteDc set to a very high number. Cassandra cluster has two datacenters. I would expect that when the cluster is operating normally (all local nodes reachable), ONE would behave the same as LOCAL_ONE. The Does anyone know why this is not the case?
ONE has much higher latency than LOCAL_ONE
I am seeing unexpected behavior: consistency level ONE increases read latency 99th percentile to ~108ms (95th percentile to 5ms-90ms) up from ~5ms (99th percentile) when using LOCAL_ONE. I am using DSE 5.0 with Datastax client 3.0.0. The client is configured with a TokenAwarePolicy wrapping a DCAwareRoundRobinPolicy with usedHostsPerRemoteDc set to a very high number. Cassandra cluster has two datacenters. I would expect that when the cluster is operating normally (all local nodes reachable), ONE would behave the same as LOCAL_ONE. The Does anyone know why this is not the case?
Re: Consistency Level vs. Retry Policy when no local nodes are available
Thanks for the perspective Ben, it's food for thought. At minimum, it seems like the documentation should be updated to mention that the retry policy will not be consulted when using a local consistency level but with no local nodes available. That way, people won't be surprised by it. It looks like the docs are included in the Github repo, so I guess I'll try to contribute an update there. From: Ben Slater <ben.sla...@instaclustr.com<mailto:ben.sla...@instaclustr.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Monday, March 20, 2017 at 6:25 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: Consistency Level vs. Retry Policy when no local nodes are available I think the general assumption is that DC failover happens at the client app level rather than the Cassandra level due to the potentially very significant difference in request latency if you move from a app-local DC to a remote DC. The preferred pattern for most people is that the app fails in a failed DC and some load balancer above the app redirects traffic to a different DC. The other factor is that the fail-back scenario from a failed DC and LOCAL_* consistencies is potentially complex. Do you want to immediately start using the new DC when it becomes available (with missing data) or wait until it catches up on writes (and how do you know when that has happened)? Note also QUORUM is a clear majority of replicas across both DCs. Some people run 3 DCs with RF 3 in each and QUORUM to maintain strong consistency across DCs even with DC failure. Cheers Ben On Tue, 21 Mar 2017 at 10:00 Shannon Carey <sca...@expedia.com<mailto:sca...@expedia.com>> wrote: Specifically, this puts us in an awkward position because LOCAL_QUORUM is desirable so that we don't have unnecessary cross-DC traffic from the client by default, but we can't use it because it will cause complete failure if the local DC goes down. And we can't use QUORUM because it would fail if there's not a quorum in either DC (as would happen if one DC goes down). So it seems like we are forced to use a lesser consistency such as ONE or TWO. -Shannon From: Shannon Carey <sca...@expedia.com<mailto:sca...@expedia.com>> Date: Monday, March 20, 2017 at 5:25 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Consistency Level vs. Retry Policy when no local nodes are available I am running DSE 5.0, and I have a Java client using the Datastax 3.0.0 client library. The client is configured to use a DCAwareRoundRobinPolicy wrapped in a TokenAwarePolicy. Nothing special. When I run my query, I set a custom retry policy. I am testing cross-DC failover. I have disabled connectivity to the "local" DC (relative to my client) in order to perform the test. When I run a query with the first consistency level set to LOCAL_ONE (or local anything), my retry policy is never called and I always get this exception: "com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)" getErrors() on the exception is empty. This is contrary to my expectation that the first attempt would fail and would allow my RetryPolicy to attempt a different (non-LOCAL) consistency level. I have no choice but to avoid using any kind of LOCAL consistency level throughout my applications. Is this expected? Or is there anything I can do about it? Thanks! It certainly seems like a bug to me or at least something that should be improved. -Shannon -- Ben Slater Chief Product Officer [https://cdn2.hubspot.net/hubfs/2549680/Instaclustr-Navy-logo-new.png]<https://www.instaclustr.com/> [http://cdn2.hubspot.net/hubfs/184235/dev_images/signature_app/facebook_sig.png]<https://www.facebook.com/instaclustr> [http://cdn2.hubspot.net/hubfs/184235/dev_images/signature_app/twitter_sig.png] <https://twitter.com/instaclustr> [http://cdn2.hubspot.net/hubfs/184235/dev_images/signature_app/linkedin_sig.png] <https://www.linkedin.com/company/instaclustr> Read our latest technical blog posts here<https://www.instaclustr.com/blog/>. This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message.
Re: Consistency Level vs. Retry Policy when no local nodes are available
Specifically, this puts us in an awkward position because LOCAL_QUORUM is desirable so that we don't have unnecessary cross-DC traffic from the client by default, but we can't use it because it will cause complete failure if the local DC goes down. And we can't use QUORUM because it would fail if there's not a quorum in either DC (as would happen if one DC goes down). So it seems like we are forced to use a lesser consistency such as ONE or TWO. -Shannon From: Shannon Carey <sca...@expedia.com<mailto:sca...@expedia.com>> Date: Monday, March 20, 2017 at 5:25 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Consistency Level vs. Retry Policy when no local nodes are available I am running DSE 5.0, and I have a Java client using the Datastax 3.0.0 client library. The client is configured to use a DCAwareRoundRobinPolicy wrapped in a TokenAwarePolicy. Nothing special. When I run my query, I set a custom retry policy. I am testing cross-DC failover. I have disabled connectivity to the "local" DC (relative to my client) in order to perform the test. When I run a query with the first consistency level set to LOCAL_ONE (or local anything), my retry policy is never called and I always get this exception: "com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)" getErrors() on the exception is empty. This is contrary to my expectation that the first attempt would fail and would allow my RetryPolicy to attempt a different (non-LOCAL) consistency level. I have no choice but to avoid using any kind of LOCAL consistency level throughout my applications. Is this expected? Or is there anything I can do about it? Thanks! It certainly seems like a bug to me or at least something that should be improved. -Shannon
Is Repair complete?
Can anyone give a better answer to the question "How do I know if a "nodetool
repair" is finished?" on
http://stackoverflow.com/questions/25064717/how-do-i-know-if-nodetool-repair-is-finished
I am trying to prepare to upgrade to DSE 5. I am migrating to incremental
repairs first. So I disabled autocompaction and I kicked off a full sequential
repair ("nodetool repair") on one node in DSE 4.8.5, and more than 24 hours
later, I still often (but not always) see "Repair of …" showing up in the
Activities in OpsCenter, and often (but not always) "nodetool compactionstats"
shows a "Validation" compaction under "pending tasks". Is the repair done? How
can I tell for certain?
Also, the node is an AWS EC2 i2.xlarge and only contains ~311GB… is >24 hr a
reasonable/expected duration for a single-node full sequential repair?
Thanks,
Shannon
