More info about this dashboard "no metrics" issue: I restarted Ganglia server and monitor from Ambari Web UI (from each host) successfully. On Ganglia server host, I run command "service hdp-gmond status", and get response like this:
======================================= Checking status of hdp-gmond... ======================================= /usr/sbin/gmond for cluster HDPSlaves running with PID 30136 /usr/sbin/gmond for cluster HDPNimbus running with PID 30164 /usr/sbin/gmond for cluster HDPSupervisor running with PID 30192 So I believe all 3 gmond services are up and running properly. Also on Ganglia server (gmetad) host, I can see all metrics rrd files (like cpu_num.rrd, disk_free.rrd, etc) are under /var/lib/ganglia/rrds. But still, dashboard does not show any graph, instead only display "No Data. There was no data available. Possible reasons include inaccessible Ganglia service." Any advice? From: Yu, Tao [mailto:[email protected]] Sent: Thursday, January 29, 2015 11:24 AM To: '[email protected]' Subject: Ambari dashboard display no data Hi all, I have a newly installed HDP-2.2 on a small cluster (2 hosts). The installation ( via install wizard), including services like Ganglia / Zookeeper / Storm etc, went smoothly (CentOS 6.5) and ambari-server and all desired services are up and running. But when I login to the Ambari web UI, the dashboard shows nothing for any of the standard Metrics (Memory / Network / CPU /Cluster load), instead the UI shows below message: No data There was no data available. Possible reasons include inaccessible Ganglia service I then checked both hosts where Ganglia server daemon (gmetad) and client daemon (gmond) run, all daemons are running: #ps -ef |grep gmetad nobody 31970 1 0 10:38 ? 00:00:02 /usr/sbin/gmetad --conf=/etc/ganglia/hdp/gmetad.conf --pid-file=/var/run/ganglia/hdp/gmetad.pid #ps -ef | grep gmond nobody 13681 1 0 10:38 ? 00:00:19 /usr/sbin/gmond --conf=/etc/ganglia/hdp/HDPSlaves/gmond.core.conf --pid-file=/var/run/ganglia/hdp/HDPSlaves/gmond.pid But when I check the ambari-server.log, got some ERROR message with Java exception: ERROR [qtp1835310612-22] GangliaReportPropertyProvider:153 - Caught exception getting Ganglia metrics : java.net.ConnectException: Connection refused : spec=http://vm-galaxy04.aware.com/ganglia/graph.php?g=cpu_report&json=1 The ERROR message seems like network connection issue, but the hosts look OK in the network. I have tried restarting Ganglia services with no luck, restarted all services but still the same. Does anyone have any ideas how I solve the Java Exception and get the dashboard to work properly? Thank you!
