I recently removed metrics from ganglia and recompiled/redistributed accross our 300 node cluster.
We have since noticed errors like these filling the logs. /var/log messages.x is regularly 320k or so, but we've been seeing up to 25 Mb files filled with these errors. We also tried to add metrics via gmetric. At the time we noticed a huge increase in CPU usage, but this was likely due to people using the cluster during maintenance. Any ideas as to what these erros mean? Is this directly related to the recompilation? Thanks in advance. Aug 28 12:07:04 medusa-slave001 /usr/sbin/gmond[20524]: pre_process_node() failed to get node location (0) Aug 28 12:07:09 medusa-slave001 /usr/sbin/gmond[20524]: pre_process_node() failed to get node location (0) Aug 28 12:07:09 medusa-slave001 /usr/sbin/gmond[20525]: pre_process_node() failed to get node location (0 Aug 25 08:45:58 medusa-slave001 /usr/sbin/gmond[20524]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:46:29 medusa-slave001 /usr/sbin/gmond[20525]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:46:34 medusa-slave001 /usr/sbin/gmond[20525]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:46:45 medusa-slave001 /usr/sbin/gmond[20524]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:47:01 medusa-slave001 /usr/sbin/gmond[20525]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:47:35 medusa-slave001 /usr/sbin/gmond[20524]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:47:47 medusa-slave001 /usr/sbin/gmond[20525]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:48:04 medusa-slave001 /usr/sbin/gmond[20524]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:48:20 medusa-slave001 /usr/sbin/gmond[20525]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:48:36 medusa-slave001 /usr/sbin/gmond[20524]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:49:07 medusa-slave001 /usr/sbin/gmond[20524]: mcast_listen_thread() xdr_string() error: Interrupted system call Aug 25 08:50:37 medusa-slave001 last message repeated 4 times
