I've inherited an integration of Cassandra's Codahale Metrics reporting with Ganglia that looks sensible enough on the Metrics side. The metrics-reporter-config.yaml points to a gmond.conf on the node. Excerpt: ganglia: - period: 60 timeunit: 'SECONDS' gmondConf: '/etc/ganglia/gmond.conf'
Now if I look at the gmond.conf on one of the nodes in the Cassandra cluster, I see that there's a UDP entry for EVERY node in the cluster, plus OpsCenter: udp_send_channel { host = sandbox-cas00 port = 8649 ttl = 1 } udp_send_channel { host = sandbox-cas01 port = 8649 ttl = 1 } udp_send_channel { host = sandbox-cas02 port = 8649 ttl = 1 } udp_send_channel { host = sandbox-cas03 port = 8649 ttl = 1 } udp_send_channel { host = sandbox-cas04 port = 8649 ttl = 1 } udp_send_channel { host = sandbox-opscenter port = 8649 ttl = 1 } What strikes me funny about this configuration is that every node in the cluster is referenced. I was expecting that each node's gmond.conf would only define a udp_send_channel for itself. This is fine now with a five-node cluster, and we have enough deployment automation in place to mitigate the fact that every time we add a Cassandra node every other Cassandra node's gmond.conf would have to be updated. But is it really the intention that every Ganglia host needs to be aware of every other host? I was expecting that it would only be the Ganglia server these hosts are reporting to that would need to be aware of the full membership of the cluster. OK, I realize at this point I'm asking more of a Ganglia question than a Cassandra or Metrics question. Apparently, we're using Ganglia in unicast mode rather than multicast.