[
https://issues.apache.org/jira/browse/CASSANDRA-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13898928#comment-13898928
]
Kirill Bogdanov commented on CASSANDRA-6683:
--------------------------------------------
Thank you for your answer.
I have came across this part of code in DES because I have observed suboptimal
choice of nodes in my configuration and started to investigate it.
This is my config:
* PropertyFileSnitch
* dynamic_snitch_badness_threshold 0.1,
* 4 DCs,
* keyspace with replication quota 1 for each DC.
* Read repair and speculative_retry are disabled for my tables.
* Performing read operations with consistency TWO
I am observing that local DC that serves read request has about the same
probability of asking any of the 3 remote replicas to confirm consistency TWO
regardless of their score (is that correct?).
Since all nodes are in different DCs {{subsnitch.sortByProximity}} places local
node at the start of the list (first) but does not sort other remote DCs.
After {{subsnitch.sortByProximity}} addresses list with scores may look
something like that:
- DC1: 0.1 (first)
- DC2: 0.7
- DC3: 0.2
- DC4: 0.2
Since we are not calling {{sortByProximityWithScore}} we returning this list to
{{AbstractReadExecutor getReadExecutor}} where
{{consistencyLevel.filterForQuery}} (based on consistency TWO) picks up first 2
addresses from the list. As a result we are sending read request to suboptimal
DC2.
By implementing my change ({{Math.abs()}}) I am seeing ~15% read throughput
improvement in my setup with cassandra stress tool.
Due to my limited knowledge of Cassandra internals I am probably wrong to blame
DES and BADNESS_THRESHOLD, but I would greatly appreciate if you could point
out what is the correct behaviour in the situation above and which module is
responsible for sorting nodes by the scores.
Thank you.
> BADNESS_THRESHOLD does not working correctly with DynamicEndpointSnitch
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-6683
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6683
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Linux 3.8.0-33-generic
> Reporter: Kirill Bogdanov
> Labels: snitch
> Fix For: 2.0.6
>
>
> There is a problem in *DynamicEndpointSnitch.java* in
> sortByProximityWithBadness()
> Before calling sortByProximityWithScore we comparing each nodes score ratios
> to the badness threshold.
> {code}
> if ((first - next) / first > BADNESS_THRESHOLD)
> {
> sortByProximityWithScore(address, addresses);
> return;
> }
> {code}
> This is not always the correct comparison because *first* score can be less
> than *next* score and in that case we will compare a negative number with
> positive.
> The solution is to compute absolute value of the ratio:
> {code}
> if (Math.abs((first - next) / first) > BADNESS_THRESHOLD)
> {code}
> This issue causing an incorrect sorting of DCs based on their performance and
> affects performance of the snitch.
> Thanks.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
