Hi all, Does Ganglia use Intel ANS Probes and/or UDP packets with the destination port 8649? I encountered a strange situation a couple of days ago. Normally I have drives mapped to a Cluster Master (Linux machine with Ganglia on it) and two days ago, I could not connect to the Cluster Master at all. No SSH, no X-windows, not even a monitor display.
At the same time, I discovered that there was a broadcast storm on the network which the Cluster Master is on. The Cluster Master and the Windows 2000 Server machien both were sending Intel ANS Probe broadcasts to the network, and the Cluster Master was also sending UDP packets to the destination IP of 239.2.11.71, port 8649. I've never seen anything like it, nor do I know what caused this broadcast storm. I had to hard-reboot the Cluster Master, and disable the Probes on both the Cluster Master and the Win2k Server to stop the broadcast storm. And when I checked the /var/log/messages on the Cluster Master, everything had stopped logging at 7:00 pm the previous night. And checking the other logs, I could not find any clues to what caused the Cluster Master to stop logging. I went to the Ganglia interface, and I found that there was a gap in all the graphs between 7:00 pm the previous night and the time I hard-rebooted the Cluster Master. The only clue I have is that the stopped logging and the broadcast storm occurred coincidentally, so there must have been a root cause for both of these. And I suspect it is Ganglia, or some glitch in any of the processes related to (spawned by) Ganglia. However, after the probe disable and the hard reboot, everything was working just fine and running normally, as if nothing had happened. I asked Microway about this, and they could not provide a clear answer as to why this happened. Perhaps this has happened to some of you before, and I was hoping you could provide me some insight on this. Thanks.
