Hi Sid,
I changes debug level of gmetad.conf to 1 and restarted the process [pts/2][10:19:52:root@INBBRDSSVM50 ] /var/lib/ganglia/rrds/HDPHBaseMaster/INBBRDSSVM50>service hdp-gmetad restart ================================== Shutting down hdp-gmetad... ================================== Stopped /usr/sbin/gmetad (with PID 11110) Stopped /usr/bin/rrdcached (with PID 11080) ============================= Starting hdp-gmetad... ============================= Started /usr/bin/rrdcached with PID 10344 Sources are ... Source: [HDPHBaseMaster, step 15] has 1 sources 10.18.15.75 Source: [HDPResourceManager, step 15] has 1 sources 10.18.15.75 Source: [HDPSlaves, step 15] has 1 sources 10.18.15.75 Source: [HDPHistoryServer, step 15] has 1 sources 10.18.15.75 Source: [HDPNameNode, step 15] has 1 sources 10.18.15.75 Data thread 139698190071552 is monitoring [HDPHBaseMaster] data source 10.18.15.75 Data thread 139698179581696 is monitoring [HDPResourceManager] data source 10.18.15.75 Data thread 139698169091840 is monitoring [HDPSlaves] data source 10.18.15.75 Data thread 139698024384256 is monitoring [HDPHistoryServer] data source 10.18.15.75 Data thread 139698158601984 is monitoring [HDPNameNode] data source 10.18.15.75 There is enough space available, Even Namenode rrd files and other related files are there under /var/lib/ganglia/rrds. [pts/0][10:26:37:root@INBBRDSSVM50 ] /var/lib/ganglia/rrds>ll total 72 drwxr-xr-x. 4 nobody nobody 4096 Jun 9 16:29 HDPHBaseMaster drwxr-xr-x. 4 nobody nobody 4096 Jun 9 16:29 HDPNameNode drwxr-xr-x. 4 nobody nobody 4096 Jun 9 16:29 HDPResourceManager drwxr-xr-x. 4 nobody nobody 4096 Jun 9 16:29 HDPSlaves drwxr-xr-x. 2 nobody nobody 57344 Jun 12 17:25 __SummaryInfo__ In /var/log/messeges there is no Error Jun 13 10:01:23 INBBRDSSVM50 ntpd[1935]: time reset +0.155663 s Jun 13 10:03:52 INBBRDSSVM50 ntpd[1935]: synchronized to 172.18.225.143, stratum 5 Jun 13 10:20:06 INBBRDSSVM50 rrdcached[11080]: caught SIGTERM Jun 13 10:20:06 INBBRDSSVM50 rrdcached[11080]: starting shutdown Jun 13 10:20:07 INBBRDSSVM50 rrdcached[11080]: clean shutdown; all RRDs flushed Jun 13 10:20:07 INBBRDSSVM50 rrdcached[11080]: goodbye Jun 13 10:20:12 INBBRDSSVM50 rrdcached[10344]: starting up Jun 13 10:20:12 INBBRDSSVM50 rrdcached[10344]: listening for connections Jun 13 10:24:27 INBBRDSSVM50 nagios: Auto-save of retention data completed successfully. Graphs are displayed on Ganglia WEB UI but the same are not Displayed on Ambari Web-UI Thanks & Regards, Meghavi Sugandhi Tata Consultancy Services Mailto: [email protected] Website: http://www.tcs.com -----Siddharth Wagle <[email protected]> wrote: ----- To: "[email protected]" <[email protected]> From: Siddharth Wagle <[email protected]> Date: 06/12/2014 08:59PM Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI Hi Meghavi, You should be able to debug this by editing, /etc/ganglia/hdp/gmetad.conf Search for debug_level and set to above 1. Setting debug level above 1 should start process in foreground and you can also look at the log file (/var/log/messages) for errors. Also, make there are rrd files present for HDFS by looking a say Namenode rrd files under /var/lib/ganglia/rrds. Some other this to look for, make sure there is enough space on the mount point where ganglia is writing files. -Sid On Thu, Jun 12, 2014 at 6:41 AM, Meghavi Sugandhi <[email protected]> wrote: Hi, I have disabled the SELinux. [pts/2][19:06:59:root@INBBRDSSVM50 ] /u01/HomeDir/NGA>getenforce Disabled But stil I am not able to view all graphs on Ambari Web_UI. Only Hbase Graphs are visible but HDFS and YARN realated graphs are Not visible. Same case with Heatmaps. /var/log/messeges Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: sys_logger;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for HistoryServer;OK;service_sys_logger;TCP OK - 0.000 second response time on port 8666 Jun 12 17:29:47 INBBRDSSVM50 Hadoop[30409]: OK: Hadoop: ganglia_monitor_process_ok# Event Host=INBBRDSSVM50.india.tcs.com Service Description=GANGLIA::Ganglia Monitor process for HistoryServer(OK), TCP OK - 0.000 second response time on port 8666 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: nagiosadmin;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for HistoryServer;OK;notify-service-by-email;TCP OK - 0.000 second response time on port 8666 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE ALERT: INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for HBase Master;OK;HARD;4;TCP OK - 0.001 second response time on port 8663 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: sys_logger;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for HBase Master;OK;service_sys_logger;TCP OK - 0.001 second response time on port 8663 Jun 12 17:29:47 INBBRDSSVM50 Hadoop[30417]: OK: Hadoop: ganglia_monitor_process_ok# Event Host=INBBRDSSVM50.india.tcs.com Service Description=GANGLIA::Ganglia Monitor process for HBase Master(OK), TCP OK - 0.001 second response time on port 8663 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: nagiosadmin;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for HBase Master;OK;notify-service-by-email;TCP OK - 0.001 second response time on port 8663 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE ALERT: INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for NameNode;OK;HARD;4;TCP OK - 0.000 second response time on port 8661 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: sys_logger;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for NameNode;OK;service_sys_logger;TCP OK - 0.000 second response time on port 8661 Jun 12 17:29:47 INBBRDSSVM50 Hadoop[30425]: OK: Hadoop: ganglia_monitor_process_ok# Event Host=INBBRDSSVM50.india.tcs.com Service Description=GANGLIA::Ganglia Monitor process for NameNode(OK), TCP OK - 0.000 second response time on port 8661 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: nagiosadmin;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for NameNode;OK;notify-service-by-email;TCP OK - 0.000 second response time on port 8661 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE ALERT: INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for ResourceManager;OK;HARD;4;TCP OK - 0.000 second response time on port 8664 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: sys_logger;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for ResourceManager;OK;service_sys_logger;TCP OK - 0.000 second response time on port 8664 Jun 12 17:29:47 INBBRDSSVM50 Hadoop[30433]: OK: Hadoop: ganglia_monitor_process_ok# Event Host=INBBRDSSVM50.india.tcs.com Service Description=GANGLIA::Ganglia Monitor process for ResourceManager(OK), TCP OK - 0.000 second response time on port 8664 Jun 12 17:29:47 INBBRDSSVM50 nagios: SERVICE NOTIFICATION: nagiosadmin;INBBRDSSVM50.india.tcs.com;GANGLIA::Ganglia Monitor process for ResourceManager;OK;notify-service-by-email;TCP OK - 0.000 second response time on port 8664 Jun 12 18:24:27 INBBRDSSVM50 nagios: Auto-save of retention data completed successfully. I tried by restting whole setup of ambari. Even tried ambari-server and aganet, hdp-gmond and hdp-gmetad restart, but Graphs are not displayed. I am able to view all graphs on ganglia web UI but those are not visible on Ambari Web UI :( Thanks & Regards, Meghavi Sugandhi Tata Consultancy Services Mailto: [email protected] Website: http://www.tcs.com -----Meghavi Sugandhi <[email protected]> wrote: ----- To: [email protected] From: Meghavi Sugandhi <[email protected]> Date: 06/10/2014 06:19PM Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI Hi, Is SELinux disabling is last option to resolve the problem. Thanks & Regards, Meghavi Sugandhi Tata Consultancy Services Mailto: [email protected] Website: http://www.tcs.com -----Olivier Renault <[email protected]> wrote: ----- To: [email protected] From: Olivier Renault <[email protected]> Date: 06/10/2014 04:30PM Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI If it's permissive, it's not in your way. It will show what it would have stop but it should not stop anything.ÿ Olivier On 10 June 2014 09:36, Meghavi Sugandhi <[email protected]> wrote: Hi Getenforce status is Permissive only. [pts/0][14:05:43:root@INBBRDSSVM50 ] /var/nagios>getenforce Permissive but its not working. Do I have to disable it? Is there any other option? Thanks & Regards, Meghavi Sugandhi Tata Consultancy Services Mailto: [email protected] Website: http://www.tcs.com -----Olivier Renault <[email protected]> wrote: ----- To: [email protected] From: Olivier Renault <[email protected]> Date: 06/10/2014 12:58PM Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI You can run getenforce, it should give you the status (enforcing, disabled or permissive) If you're in enforcing, you can switch it to permissive by running setenforce 0 Fyi, I don't believe that Ambari can runÿ with selinux enabled. It might be better to switch it off once for all by editing /etc/selinux/config Olivier On 10 Jun 2014 08:21, "Meghavi Sugandhi" <[email protected]> wrote: Hi After running the below command, echo 0 >/selinux/enforce I am still getting same error in /var/log/messages. Junÿ 9 15:40:59 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/bin/python from getattr access on the file /var/lib/ganglia/rrds/HDPNameNode/INBBRDSSVM50/rpcdetailed.rpcdetailed.RegisterDatanodeNumOps.rrd. For complete SELinux messages. run sealert -l 62ff979d-6c24-4a58-8a1b-c463f6b34533 Junÿ 9 15:41:00 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/bin/python from read access on the file /var/lib/ganglia/rrds/HDPSlaves/__SummaryInfo__/dfs.datanode.BytesWritten.rrd. For complete SELinux messages. run sealert -l 3dbb1d53-e603-490b-9395-6166581428e3 Junÿ 9 15:41:01 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/sbin/httpd from getattr access on the file /var/nagios/status.dat. For complete SELinux messages. run sealert -l f693d59d-37b4-489e-837b-eb6b79375298 Junÿ 9 15:41:02 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/sbin/httpd from read access on the file /var/nagios/status.dat. For complete SELinux messages. run sealert -l a7ea6343-d141-4d9f-bf35-8930130ac881 Junÿ 9 15:41:48 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/bin/python from getattr access on the file /var/lib/ganglia/rrds/HDPResourceManager/INBBRDSSVM50/load_one.rrd. For complete SELinux messages. run sealert -l 62ff979d-6c24-4a58-8a1b-c463f6b34533 Thanks & Regards, Meghavi Sugandhi Tata Consultancy Services Mailto: [email protected] Website: http://www.tcs.com -----Yusaku Sako <[email protected]> wrote: ----- To: [email protected] From: Yusaku Sako <[email protected]> Date: 06/10/2014 11:21AM Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI You can run the following to temporarily disable SELinux until the next server reboot. echo 0 >/selinux/enforce Yusaku On Mon, Jun 9, 2014 at 10:46 PM, Meghavi Sugandhi <[email protected]> wrote: Hi Sid, Thanks for reply. All rrd files related to Namenode, HBase, Mapreduceare are being written to /var/lib/ganglia/rrds/ I cant disable SELinux. Because disabling it requires rebooting of system. Is there any other way which can handle this issue. Few graphs that of hbase are visible but that of HDFS and YARN Mapreduce are Not Displyed Thanks & Regards, Meghavi Sugandhi Tata Consultancy Services Mailto: [email protected] Website: http://www.tcs.com -----Siddharth Wagle <[email protected]> wrote: ----- To: "[email protected]" <[email protected]> From: Siddharth Wagle <[email protected]> Date: 06/09/2014 11:45PM Subject: Re: Graphs and HeatMaps are Not Visible on Ambari Web-UI Hi Meghavi, Are you able to obtain metrics data by turning off SELinux temporarily on Ganglia host ? Also, are rrd files being written to /var/lib/ganglia/rrds/ ? -Sid On Mon, Jun 9, 2014 at 4:19 AM, Meghavi Sugandhi <[email protected]> wrote: Hello, I am using Ambari 1.4.4. I have installed it successfully and all services are up and running But I am not able to view few graphs and heatmaps I am getting following error in /var/log/messeges--> Junÿ 9 15:40:59 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/bin/python from getattr access on the file /var/lib/ganglia/rrds/HDPNameNode/INBBRDSSVM50/rpcdetailed.rpcdetailed.RegisterDatanodeNumOps.rrd. For complete SELinux messages. run sealert -l 62ff979d-6c24-4a58-8a1b-c463f6b34533 Junÿ 9 15:41:00 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/bin/python from read access on the file /var/lib/ganglia/rrds/HDPSlaves/__SummaryInfo__/dfs.datanode.BytesWritten.rrd. For complete SELinux messages. run sealert -l 3dbb1d53-e603-490b-9395-6166581428e3 Junÿ 9 15:41:01 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/sbin/httpd from getattr access on the file /var/nagios/status.dat. For complete SELinux messages. run sealert -l f693d59d-37b4-489e-837b-eb6b79375298 Junÿ 9 15:41:02 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/sbin/httpd from read access on the file /var/nagios/status.dat. For complete SELinux messages. run sealert -l a7ea6343-d141-4d9f-bf35-8930130ac881 Junÿ 9 15:41:48 INBBRDSSVM50 setroubleshoot: SELinux is preventing /usr/bin/python from getattr access on the file /var/lib/ganglia/rrds/HDPResourceManager/INBBRDSSVM50/load_one.rrd. For complete SELinux messages. run sealert -l 62ff979d-6c24-4a58-8a1b-c463f6b34533 SElinux is in Permissive mode and owner of /var/nagios drwxr-xr-x.ÿ 4 nagios nagios 4096 Junÿ 9 15:52 nagios and owner of /var/lib/ganglia/rrds is nobody:nobody drwxr-xr-x. 7 nobody nobody 4096 Junÿ 6 11:50 rrds Thanks & Regards, Meghavi Sugandhi Tata Consultancy Services Mailto: [email protected] Website: http://www.tcs.com =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Olivier Renault Solution Engineer , Hortonworks Mobile: +44 7500 933 036 Email: [email protected] Website: http://www.hortonworks.com/ Latest From Our Blog: How to Analyze Server Logs with Cascading and Apache Hadoop on Hortonworks Data Platform CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
