Gossip propagation is generally best modelled by epidemic algorithms.
Luckily for us Cassandra's gossip protocol is fairly simply.
Cassandra will perform one Gossip Task every second. Within each gossip
task it will randomly gossip with another available node in the cluster, it
will also possibly attempt to gossip with a down node (based on a random
chance that increases as the number of down nodes increases) and if it
hasn't gossiped with seed that round it may also attempt to gossip with a
defined seed. So Cassandra can do up to 3 rounds per second, however these
extra rounds are supposed to be optimizations for improving average case
convergence and recovering from split brain scenarios quicker than would
Assuming just one gossip round per second, for a new piece of information
to spread to all members of the cluster via gossip, you would see a worst
case performance of O(n) gossip rounds where n is the number of nodes in
the cluster. This is because each Cassandra node can gossip to any other
node irrespective of topology (a fully connected mesh).
There is some ongoing discussion about expanding gossip to utilise partial
views of the cluster and exchanging those, or using spanning/broadcast
trees to speed up convergence and reduce workload in large clusters (1000+)
nodes, see https://issues.apache.org/jira/browse/CASSANDRA-12345 for
On Fri, 16 Sep 2016 at 01:01 Jens Rantil <jens.ran...@tink.se> wrote:
> > Is a minute a reasonable upper bound for most clusters?
> I have no numbers and I'm sure this differs depending on how large your
> cluster is. We have a small cluster of around 12 nodes and I statuses
> generally propagate in under 5 seconds for sure. So, it will definitely be
> less than 1 minute.
> On Wed, Sep 14, 2016 at 8:49 PM jerome <jeromefroel...@hotmail.com> wrote:
>> I was curious if anyone had any kind of statistics or ballpark figures on
>> how long it takes information to propagate through a cluster with Gossip?
>> I'm particularly interested in how fast information about the liveness of a
>> node spreads. For example, in an n-node cluster the median amount of time
>> it takes for all nodes to learn that a node went down is f(n) seconds. Is a
>> minute a reasonable upper bound for most clusters? Too high, too low?
> Jens Rantil
> Backend Developer @ Tink
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> For urgent matters you can reach me at +46-708-84 18 32.
CTO | Instaclustr <https://www.instaclustr.com/>
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer