We're getting tons of messages in /var/log/messages like this:
"illegal attempt to update using time 1462910094 when last update time is
1462910094 (minimum one second step)"
This has been happening for a while and I'm not too concerned, because
everything seems to be working fine, but I'm getting annoyed by this and feel
like I should clean up whatever is causing this... However, I can't figure out
conclusively why this is happening nor how to stop it. I've found lots of
useful posts that point to a variety of possibilities like redundant cluster
names, improper reverse DNS lookups, duplicate hostnames, etc. However, I have
double, triple, and quadruple checked all of these things.
We have two ganglia installs, one running 3.1.7 and one running 3.7.2, this
behavior is seen on both installs.
As mentioned in the subject, we have multiple gmetads running, on the same
host. I think that is the cause of this, but I don't completely understand why.
I even trimmed down our config to the smallest grid and the smallest cluster in
that grid, so all told, only 4 machines including the server running the
gmetads, and I still get this error. So, I think it has something to do with
the multiple metad daemons running.
So there's 3 separate gmetad services running on the box, each with their own
config file, each with their own data sources and ports and web roots
(actually, there's 5, but for brevity, I'm just going to talk about 3)
Here is our daemon layout and the data sources for each. Hopefully this
representation makes sense
-- Master gmetad -p8651 -- local
|-- Data source - HPCgrid- local gmetad1-p8653
| |-- GPUCluster--external gmond
| |-- Gen5Cluster--external gmond
| |-- Gen6Cluster--external gmond
| |-- HeadNodesCluster--external gmond
| `-- InteractiveCluster--external gmond
|-- Data Source AppServersgrid- local gmetad2-p8655
| |-- App1Cluster--external gmond
| `-- App2Cluster--external gmond
|-- AdminCluster--local gmond
|-- MiscCluster--external gmond
`-- Workstations--external gmond
So I guess the real question I have is.... Is this a setup valid? Should this
work? If so, what have I done wrong? If not, why?
The more I look into this as I'm writing this message the more I've convinced
myself that it is indeed because of the multiple gmetads running. The messages
log shows two entries for each metric in the __SummaryInfo__ dir, but each
entry in messages is coming from a different gmetad. It is coming from both
the Master gmetad, and also from the sub-grid gmetad daemon, not necessarily in
that order.
May 10 16:29:05 localhost /usr/sbin/gmetad2[25468]: RRD_update
(/var/lib/ganglia/rrds/Main/App Servers/__SummaryInfo__/bytes_in.rrd):
/var/lib/ganglia/rrds/Main/App Servers/__SummaryInfo__/bytes_in.rrd: illegal
attempt to update using time 1462912145 when last update time is 1462912145
(minimum one second step)
May 10 16:30:51 localhost /usr/sbin/gmetad[1645]: RRD_update
(/var/lib/ganglia/rrds/Main/App Servers/__SummaryInfo__/bytes_in.rrd):
/var/lib/ganglia/rrds/Main/App Servers/__SummaryInfo__/bytes_in.rrd: illegal
attempt to update using time 1462912251 when last update time is 1462912251
(minimum one second step)
So why are both of these daemons trying to update the summary info for this
cluster? I expected the master gmetad daemon to only be updating the
__SummaryInfo__ for the top level data under
/var/lib/ganglia/rrds/Main/__SummaryInfo__ and not trying to also update the
info under the sub-grids.
Can anyone shed some light on this for me?
Thanks!
Patty
------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general
