Re: Problem configuring Ganglia(jmxtrans) with Accumulo 1.7.0

Thu, 28 Jan 2016 02:23:58 -0800 

Thanks ... It worked :)

Is there any configurations for Nagios to generate alerts?


On 01/27/2016 09:50 PM, Josh Elser wrote:

Mohit,
I would strongly recommend you take a look at the Hadoop Metrics2 support that was added in 1.7.0 to push metrics to Ganglia instead of using JMXTrans.
This was one of the big reasons we chose to adopt the new metrics system. Using this integration will prevent you from having to run/monitor jmxtrans (which is always nice in a production environment). There should be a commented out example of how to configure Ganglia in the provided hadoop-metrics2-accumulo.properties file in the example configurations.
http://accumulo.apache.org/1.7/accumulo_user_manual.html#_metrics2_configuration 

- Josh

mohit.kaushik wrote:

I am trying to setup Ganglia for Monitoring Accumulo 1.7.0 Cluster
having 5 nodes. I also configured ganglia for Accumulo 1.6.1 standalone
cluster which is working fine and shows all the required metrices
related to Accumulo.

After scanning jmxtrans.log, I found that the its not exexuting quring
properly. It starts the job and finish in next line
[27 Jan 2016 17:38:15] [ServerScheduler_Worker-1] 3960536 DEBUG
(com.googlecode.jmxtrans.jobs.ServerJob:31) - +++++ Started server job:
Server [host=localhost, port=9006,
url=service:jmx:rmi:///jndi/rmi://localhost:9006/jmxrmi,
cronExpression=null, numQueryThreads=null]
[27 Jan 2016 17:38:15] [ServerScheduler_Worker-1] 3960542 DEBUG
(com.googlecode.jmxtrans.jobs.ServerJob:50) - +++++ Finished server job:
Server [host=localhost, port=9006,
url=service:jmx:rmi:///jndi/rmi://localhost:9006/jmxrmi,
cronExpression=null, numQueryThreads=null]

But this not the case with Accumulo 1.6.1 cluster it executes the query
and send the ganglia metrices which I can see on ganglia web ui...

[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360736 DEBUG
(com.googlecode.jmxtrans.jobs.ServerJob:31) - +++++ Started server job:
Server [host=mohit, port=9006,
url=service:jmx:rmi:///jndi/rmi://mohit:9006/jmxrmi,
cronExpression=null, numQueryThreads=null]
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360742 DEBUG
(com.googlecode.jmxtrans.util.JmxUtils:195) - Executing queryName:
accumulo.server.metrics:instance=tserver,name=TabletServerMBean,service=TServerInfo 
from query: Query
[obj=accumulo.server.metrics:service=TServerInfo,name=TabletServerMBean,instance=tserver, 
resultAlias=Accumulo, attr=[Entries, EntriesInMemory, Ingest,
MajorCompactions, MajorCompactionsQueued, MinorCompactions,
MinorCompactionsQueued, OnlineCount, OpeningCount, Queries,
UnopenedCount, TotalMinorCompactions, HoldTime, AverageFilesPerTablet,
Name]]
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360746 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.Entries=14
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360746 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.EntriesInMemory=0
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360747 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.Ingest=0
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360748 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.MajorCompactions=0
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360748 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.MajorCompactionsQueued=0
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360749 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.MinorCompactions=0
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360749 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.MinorCompactionsQueued=0
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360750 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.OnlineCount=3
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360750 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.OpeningCount=0
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360751 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.Queries=995
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360751 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.UnopenedCount=0
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360752 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.TotalMinorCompactions=33
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360752 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.HoldTime=0.0
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360752 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.AverageFilesPerTablet=0.3333333333333333
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360753 DEBUG
(com.googlecode.jmxtrans.model.output.GangliaWriter:141) - Sending
Ganglia metric Accumulo.Name=mohit:9997
[27 Jan 2016 17:39:25] [ServerScheduler_Worker-9] 360753 DEBUG
(com.googlecode.jmxtrans.util.JmxUtils:209) - Finished running
outputWriters for query: Query
[obj=accumulo.server.metrics:service=TServerInfo,name=TabletServerMBean,instance=tserver, 
resultAlias=Accumulo, attr=[Entries, EntriesInMemory, Ingest,
MajorCompactions, MajorCompactionsQueued, MinorCompactions,
MinorCompactionsQueued, OnlineCount, OpeningCount, Queries,
UnopenedCount, TotalMinorCompactions, HoldTime, AverageFilesPerTablet,
Name]]

At the ganglia Web UI, I can not see any Accumulo metrices for this
cluster.I have checked every configurations hundreds of times.
If anyone want, I can upload configurations, procedure or any required
info. Please comment on what could be the problem in query execution?
and how can I fix it?
Is it a compatibility issue with Accumulo 1.7.0?

Thanks
Mohit Kaushik

Reply via email to