[
https://issues.apache.org/jira/browse/CASSANDRA-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298289#comment-17298289
]
Yixin Zou edited comment on CASSANDRA-16507 at 3/9/21, 7:49 PM:
----------------------------------------------------------------
We did an experiment to test how cassandra behaves when some messages are
dropped. Our experiment was:
Inside of cassandra/src/java/org/apache/cassandra/net/MessagingService.java,
when some condition is met, return the sendOneWay function before message is
sent.
We have three nodes. To mimic mutation message loss between node 1 and node
2,3. We did:
On node 1, add code : when message.verb == Verb.MUTATION && (to ==
node2_address || to == node3_address) return the function
On node 2, 3, add code : when message.verb == Verb.MUTATION && to ==
node1_address) return the function
In such a way, we tried to mimi mutation message drop between nodes. At the
same time, inserted same key, different values on node 1 and node 2.
What we observed:
# nodes disconnected from each other, as shown in above ./nodetool
describecluster result
# read that key on node1,2, got different values, which is inconsistency
was (Author: yixin_zou):
We did an experiment to test how cassandra behaves when some messages are
dropped. Our experiment was:
Inside of cassandra/src/java/org/apache/cassandra/net/MessagingService.java,
when some condition is met, return the sendOneWay function before message is
sent.
We have three nodes. To mimic mutation message drop between node 1 and node
2,3. We did:
On node 1, add code : when message.verb == Verb.MUTATION && (to ==
node2_address || to == node3_address) return the function
On node 2, 3, add code : when message.verb == Verb.MUTATION && to ==
node1_address) return the function
In such a way, we tried to mimi mutation message drop between nodes. At the
same time, inserted same key, different values on node 1 and node 2.
What we observed:
# nodes disconnected from each other, as shown in above ./nodetool
describecluster result
# read that key on node1,2, got different values, which is inconsistency
> Node unreachable to each other with MUTATION message dropped
> ------------------------------------------------------------
>
> Key: CASSANDRA-16507
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16507
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
> Reporter: Yixin Zou
> Priority: Normal
>
> Cluster setup: Ubuntu 18.04-LTS, cassandra-3.11.10. One DC, 3 nodes, 2
> replication factor;
> Using an injector to filter and drop MUTATION message between one node and
> the other two (drop MUTATION messages sent from node 1 to node 2, 3 and drop
> MUTATION messages sent from node 2, 3 to node 1). At the same time, inserted
> same key, different values on node 1 and node 2. When insertion happened,
> MUTATION message drops were caught on both node 1 and node2. Then we observed
> the different values for that key on two nodes and observed all three nodes
> disconnected from each other.
>
> {code:java}
> ./nodetool describecluster
> /Cluster Information:Cluster Information:
> Name: Test Cluster Snitch:
> org.apache.cassandra.locator.GossipingPropertyFileSnitch
> DynamicEndPointSnitch: enabled Partitioner:
> org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions: UNREACHABLE: [10.0.1.8, 10.0.1.7, 10.0.1.9]
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
