Hi Folks,
I think I can get what I need done if I can figure out how to run
multiple instances of gmond on my gmetad server.
I created several custom gmond files in /etc/ganglia/conf.d which have
different cluster names and run on different ports.
However, when I set up my clients, they all seem to gather under only
the name of one of the files under conf.d
For example:
I have conf.d/cluster1.conf and conf.d/cluster2.conf.
I then have two client machines -
client 1:
--------
cluster {
name = "cluster1"
owner = "unspecified"
latlong = "unspecified"
url = "unspecified"
}
host {
location = "unspecified"
}
udp_send_channel {
host = "gmeta server"
port = 8601
ttl = 1
}
tcp_accept_channel {
port = 8601
}
client 2:
-------
cluster {
name = "cluster2"
owner = "unspecified"
latlong = "unspecified"
url = "unspecified"
}
host {
location = "unspecified"
}
udp_send_channel {
host = "gmeta server"
port = 8602
ttl = 1
}
tcp_accept_channel {
port = 8602
}
My gmetad.conf file then looks like:
data_source "Servers" 60 IP_of_ganglia_server:8601
data_source "Desktops" 60 IP_of_ganglia_ser:8602
When I start gmond and gmeta on my ganglia server both machines ignore
the ports and pool both systems under "Desktops".
When I do a "nc localhost 8601" or 8602, I see both the servers on the
ports. I would have thought I'd see one server on each.
Can someone point me to a source where someone has configured multiple
gmonds on their gmeta/ganglia server with multiple
clusters reporting?
I can't seem to find a solution. I'd use multicast, but the network
gods aren't allowing that protocol across vlans.
thanks,
On 6/25/18 11:07 AM, jlh wrote:
Hi Folks,
I'm a newbie to Ganglia and just installed in on my CentOS 7.5
server. The IP of this server is 10.10.100.13.
I have two two building locations each consisting of servers and
workstations. I'm trying to create a Ganglia tree
that looks like the following:
Building 1 / 2 (my gridname)
|
|
/ \
/ \
/ \
Building 1 Building 2
-------- ---------------
/ \ / \
/ \ / \
Servers Desktops Servers Desktops
For Building 1, an example Server/Desktop gmond.conf file looks like
the following: (I'm only showing changes
from the default installed using "yum install ganglia-gmond")
NOTE: I'm using unicast and not multicast
Let's only focus on Building 1's configuration since I can't even get
that one to work.
---------------- generic server in Building 1---------------
/*
* The cluster attributes specified will be used as part of the <CLUSTER>
* tag that will wrap all hosts collected by this instance.
*/
cluster {
name = "Building1_Servers"
owner = "unspecified"
latlong = "unspecified"
url = "unspecified"
}
/* The host section describes attributes of the host, like the
location */
host {
location = "unspecified"
}
/* Feel free to specify as many udp_send_channels as you like. Gmond
used to only support having a single channel */
udp_send_channel {
#bind_hostname = yes # Highly recommended, soon to be default.
# This option tells gmond to use a source address
# that resolves to the machine's hostname. Without
# this, the metrics may appear to come from any
# interface and the DNS names associated with
# those IPs will be used to create the RRDs.
#mcast_join = 239.2.11.71
host = 10.10.100.13
port = 8649
ttl = 1
}
------------------------------
--------------generic workstation in Building 1 ------------
/*
* The cluster attributes specified will be used as part of the <CLUSTER>
* tag that will wrap all hosts collected by this instance.
*/
cluster {
name = "Building1_Desktops"
owner = "unspecified"
latlong = "unspecified"
url = "unspecified"
}
/* The host section describes attributes of the host, like the
location */
host {
location = "unspecified"
}
/* Feel free to specify as many udp_send_channels as you like. Gmond
used to only support having a single channel */
udp_send_channel {
#bind_hostname = yes # Highly recommended, soon to be default.
# This option tells gmond to use a source address
# that resolves to the machine's hostname. Without
# this, the metrics may appear to come from any
# interface and the DNS names associated with
# those IPs will be used to create the RRDs.
#mcast_join = 239.2.11.71
host = 10.10.100.13
port = 8650
ttl = 1
}
-----------------------------
My ganglia server's (10.10.100.13) gmeta.conf file looks like:
----------------------gmeta.conf ---------
#
gridname "Building1/Building2"
data_source "Building1_Servers" 10 10.10.100.5:8649 10.10.100.10:8649
data_source "Building1_Desktops" 10 10.10.100.100:8650 10.10.100.101:8650
A start of the gmond and gmetad daemons will create an entry in the
rrds directory (/var/lib/ganglia/rrds) with
the name Building1_Servers but never creates Building1_Destops.
If I look at the status of the gmetad executable I get the following:
[ganglia]# systemctl status gmetad
â gmetad.service - Ganglia Meta Daemon
Loaded: loaded (/usr/lib/systemd/system/gmetad.service; enabled;
vendor preset: disabled)
Active: active (running) since Mon 2018-06-25 10:08:45 MDT; 30min ago
Main PID: 29525 (gmetad)
Tasks: 10
CGroup: /system.slice/gmetad.service
ââ29525 /usr/sbin/gmetad -d 1
Jun 25 10:08:46 cal gmetad[29525]: Source: [Building1_Desktops, step
10] has 2 sources
Jun 25 10:08:46 cal gmetad[29525]: 10.10.100.100
Jun 25 10:08:46 cal gmetad[29525]: 10.10.100.101
Jun 25 10:08:46 cal gmetad[29525]: Data thread 140135988676352 is
monitoring [Building1_Servers] data source
Jun 25 10:08:46 cal gmetad[29525]: 10.10.100.5
Jun 25 10:08:46 cal gmetad[29525]: 10.10.100.10
Jun 25 10:08:46 cal gmetad[29525]: Data thread 140135980283648 is
monitoring [Building1_Desktops] data source
Jun 25 10:08:46 cal gmetad[29525]: 10.10.100.100
Jun 25 10:08:46 cal gmetad[29525]: 10.10.100.101
I can also see data being sent to the ports if I snoop the network
using the following:
tcpdump -vv -x -X -s 1500 -i eth0 'port 8649'
tcpdump -vv -x -X -s 1500 -i eth0 'port 8650'
My firewall is disabled (systemctl stop firewalld; systemctl disable
firewalld) for debugging.
The web gui shows me the layout I want, but no data gets populated
into the cluster using the non-default port of 8650.
Looking at the connections on a desktop and server, I see the following:
Server:
------
[~]# netstat -plane | egrep 'gmon|gme'
tcp 0 0 0.0.0.0:8649 0.0.0.0:*
LISTEN 987 7705816 129077/gmond
udp 2816 0 10.10.100.5:40719 10.10.100.13:8649
ESTABLISHED 987 7705819 129077/gmond
udp 0 0 0.0.0.0:8649
0.0.0.0:* 987
7705815 129077/gmond
Desktop:
--------
[root@nso2 ganglia]# netstat -plane | egrep 'gmon|gme'
tcp 0 0 0.0.0.0:8650 0.0.0.0:* LISTEN
987 7675105 164827/gmond
udp 0 0 10.10.100.100:37084 10.10.100.13:8650
ESTABLISHED 987 7675108 164827/gmond
udp 0 0 0.0.0.0:8650
0.0.0.0:* 987 7675104
164827/gmond
unix 2 [ ] DGRAM 7675110 164827/gmond
Which seems to indicate gmond is flowing data to the ganglia server on
ports 8649 and 8650. Unfortunately, while I see
the following created in the rrds directory:
[rrds]# ls
Building1_Servers __SummaryInfo__
I never see anything for Building1_Desktops....
The nmap command on my ganglia server shows:
[root@cal ganglia]# nmap -sT -O localhost
Starting Nmap 6.40 ( http://nmap.org ) at 2018-06-25 10:51 MDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00015s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 991 closed ports
PORT STATE SERVICE
22/tcp open ssh
25/tcp open smtp
80/tcp open http
199/tcp open smux
443/tcp open https
3306/tcp open mysql
8649/tcp open unknown
8651/tcp open unknown
8652/tcp open unknown
Device type: general purpose
Running: Linux 3.X
OS CPE: cpe:/o:linux:linux_kernel:3
OS details: Linux 3.7 - 3.9
Network Distance: 0 hops
I'm thinking that the port 8650 isn't open since I can't do a "telnet
localhost 8650" on the ganglia server but my firewall is down
and selinux is disabled.
So it seems I'm forgetting to configure something in the software to
let it know I'm using the port 8650 for my cluster of machines.
Any suggestions would be appreciated.
Cheers,
(note that IP's have been changed for security reasons before posting
this message)
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general
