Sid and Yusaku:
Thank you both for your inputs as they allowed me to identify the cause and
resolve the issue which appears to have been primarily a result of host
configuration that was not returning the FQDN when checked with hostname -f as
recommended in the install docs. The metrics started showing once the host
configuration on namenode and datanodes was corrected by updating the
/etc/hosts file.
Additional detail:- This cluster is setup with the namenode, which serves as
the Ambari and Ganglia host as well, being publicly accessible and datanodes
only being accessible from the namenode- The hosts on the Hadoop cluster-local
network were configured with private IPs and ".localdomain" as the domainname -
/etc/sysconfig/network contained the FQDN, and the pre-issue-resolution
/etc/hosts config files contained "local-IP <hostname> <hostname>.localdomain"
entries which resulted in 'hostname' and 'hostname -f' only returning the base
hostname- updating /etc/hosts to contain entries of the form "local-IP
<hostname>.localdomain <hostname>" (FQDN first) resulted in hostname -f
returning FQDN- all metrics, including Heatmaps, now populate and display as
designed
Thanks again!
> Date: Tue, 23 Sep 2014 13:18:11 -0700
> Subject: Re: Not all metrics are populating
> From: [email protected]
> To: [email protected]
>
> Sid,
>
> > 16:13:28,435 ERROR [pool-1-thread-6163] JMXPropertyProvider:540 - Caught
> > exception getting JMX metrics : Server returned HTTP response code: 500 for
> > URL: http://namenode.localdomain:8745/api/cluster/summary
>
> This is Ambari Server trying to get metrics from Storm REST API
> endpoint (I presume it's just co-hosted with NameNode). This error
> should be independent of HFDS/YARN metrics not coming through in
> Ambari.
>
> Yusaku
>
> On Tue, Sep 23, 2014 at 8:38 AM, Siddharth Wagle <[email protected]>
> wrote:
> > The JMX URL posted in the log looks wrong.
> > What is the value of "dfs.namenode.http-address" in your hdf-site.xml ?
> > And, can you access : http://namenode.localdomain:8745/jmx, from your
> > browser?
> >
> > Make sure the Ganglia host resolves from your ambari host by executing the
> > URL's that you see in the ambari server log from the ambari host using "curl
> > -u admin:admin <url-smaple>"
> >
> > -Sid
> >
> >
> >
> >
> >
> > On Tue, Sep 23, 2014 at 7:51 AM, . . <[email protected]> wrote:
> >>
> >> [Ambari 1.6.1, HDP 2.1, CentOS 6.5, multi-node cluster]
> >>
> >> On a fresh install, not all metrics are getting populated in Ambari.
> >>
> >> Some of the missing metrics include:
> >>
> >> Dashboard: Namenode CPU WIO (displays "n/a" instead)
> >> HDFS: All metrics except HDFS I/O (displays "No Data. There was no data
> >> available Possible reasons include inaccessible Ganglia service")
> >> YARN: All metrics (displays "No Data. There was no data available Possible
> >> reasons include inaccessible Ganglia service")
> >>
> >>
> >> The ideas suggested in the following two threads have not helped to
> >> resolve the issue:
> >>
> >> Hortonworks forum post
> >> Ambari mailing list post
> >>
> >>
> >> ambari-server.log shows the following warnings and errors
> >> (/var/log/messages has no Ganglia specific errors):
> >>
> >> 16:12:52,905 WARN [Thread-1] HeartbeatMonitor:126 - Heartbeat lost from
> >> host localhost.localdomain
> >> 16:13:28,435 ERROR [pool-1-thread-6163] JMXPropertyProvider:540 - Caught
> >> exception getting JMX metrics : Server returned HTTP response code: 500 for
> >> URL: http://namenode.localdomain:8745/api/cluster/summary
> >>
> >>
> >> Any suggestions on how to identify the cause and resolve this issue ?
> >>
> >> Thanks.
> >>
> >> Note: Cross-posted to Hortonworks forums.
> >
> >
> >
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader of
> > this message is not the intended recipient, you are hereby notified that any
> > printing, copying, dissemination, distribution, disclosure or forwarding of
> > this communication is strictly prohibited. If you have received this
> > communication in error, please contact the sender immediately and delete it
> > from your system. Thank You.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
