Hey all,
I recently wrote/applied a few of the patches lingering in JIRA in
order to get working Ganglia metrics reporting. I'd like to share a
few notes, and want to know if anyone else has experience running these:
0) A few patches have to be applied to get Ganglia reporting working
in 0.18.1, in order to squash a few NPEs
- The most significant one is that metrics which are reported as a
Java Long had no corresponding type; I mapped this to int32, which had
results you can predict.
1) IP address was set to the multicast group, not the localhost IP
address. I'm not too familiar with all the possible configurations of
Ganglia, but I did at least make a note in the wiki that one will have
to set the hadoop-metrics.properties to match gmond.conf.
2) Does anyone look closely at the average metrics? I noted that
heartBeats_avg_time is currently 46k; combined with the fact that the
recorded number of heartbeats is 54.6k, I think someone forgot to
divide something somewhere; 46k / 54.6k = .84 (what's the unit,
seconds?) average heartbeat time sounds a lot more realistic
- Same thing with the writeBlockOp_avg_time statistic.
- Does anyone else notice this discrepency? If so, I'll open a bug.
3) As foretold by my note on (0), mapping Long to int32 has caused an
integer overflow for bytes_written (it doesn't take too much work to
get bytes_written to go over 2GB). Ganglia has no long type natively;
is it more realistic to map Longs to float [or double] (preventing
overflows, but sacrificing accuracy past the first 16 [or 32] or so
digits) - or should the DFS data node be reporting numbers in
megabytes/gigabytes? Switching to megabytes would mean that overflow
won't happen until the node has written petabytes of data without
restarting. Sounds large, but not really that large.
If you *aren't* using the Ganglia plots and already have Ganglia
running on your cluster, I would highly recommend trying them out...
Brian
