Re: mesos-collectd-plugin
Hi Dan I can see a couple of things that could be wrong (NB: not a collectd expert, but these are differences I see from my working config). 1. Is /opt/collectd/etc/collectd.conf your main collectd config file? otherwise, it's not being read at all by collectd. 2. I configure the plugin in that file i.e. the Module mesos-master block should be in /opt/collectd/etc/collectd.conf , not tucked down in the python module path directory. 3. Are you sure your master listens on localhost? Mine doesn't, I needed to set that Host line to match the IP I set that master to listen on ( e.g. in /etc/mesos-master/ip ). Pretty sure one of those will do the trick (NB: you'll only get metrics from the elected master; the 'standby' masters still get polled but collectd will ignore any data from them unless they're the primary) On 11 March 2015 at 19:52, Dan Dong dongda...@gmail.com wrote: Hi, Dick, I put the plugin under: $ ls -l /opt/collectd/lib/collectd/plugins/python/ total 504 -rw-r--r-- 1 root root345 Mar 10 19:40 mesos-master.conf -rw-r--r-- 1 root root 1 Mar 10 15:06 mesos-master.py -rw-r--r-- 1 root root322 Mar 10 19:44 mesos-slave.conf -rw-r--r-- 1 root root 6808 Mar 10 15:06 mesos-slave.py -rw-r--r-- 1 root root 288892 Mar 10 19:35 python.a -rwxr-xr-x 1 root root969 Mar 10 19:35 python.la -rwxr-xr-x 1 root root 188262 Mar 10 19:35 python.so And in /opt/collectd/etc/collectd.conf, I set: LoadPlugin python Globals true /LoadPlugin . Plugin python ModulePath /opt/collectd/lib/collectd/plugins/python/ LogTraces true /Plugin $ cat /opt/collectd/lib/collectd/plugins/python/mesos-master.conf LoadPlugin python Globals true /LoadPlugin Plugin python ModulePath /opt/collectd/lib/collectd/plugins/python/ Import mesos-master Module mesos-master Host localhost Port 5050 Verbose false Version 0.21.0 /Module /Plugin Anything wrong with the above settings? Cheers, Dan 2015-03-10 17:21 GMT-05:00 Dick Davies d...@hellooperator.net: Hi Dan The .py files (the plugin) live in the collectd python path, it sounds like maybe you're not loading the plugin .conf file into your collectd config? The output will depend on what your collectd is set to write to, I use it with write_graphite. On 10 March 2015 at 20:41, Dan Dong dongda...@gmail.com wrote: Hi, All, Does anybody use this mesos-collectd-plugin: https://github.com/rayrod2030/collectd-mesos I have installed collectd and this plugin, then configured it as instructions and restarted the collectd daemon, why seems nothing happens on the mesos:5050 web UI( python plugin has been turned on in collectd.conf). My question is: 1. Should I install collectd and this mesos-collectd-plugin on each master and slave nodes and restart collectd daemon? (This is what I have done.) 2. Should the config file mesos-master.conf only configured on master node and mesos-slave.conf only configured on slave node?(This is what I have done.) Or both of them should only appear on master node? 3. Is there an example( or a figure) of what output one is expected to see by this plugin? Cheers, Dan
Re: mesos-collectd-plugin
Hi, Dick, 1. Yes, /opt/collectd/etc/collectd.conf is my main and only collectd config file. 2. Have put the Module mesos-master block into collectd.conf now. 3. I have set only 1 master, and set the Host line to ip address of master node and restarted collectd. I think I have to install graphite on my cluster too to let the following line works in collectd.conf: LoadPlugin write_graphite Cheers, Dan 2015-03-11 15:09 GMT-05:00 Dick Davies d...@hellooperator.net: Hi Dan I can see a couple of things that could be wrong (NB: not a collectd expert, but these are differences I see from my working config). 1. Is /opt/collectd/etc/collectd.conf your main collectd config file? otherwise, it's not being read at all by collectd. 2. I configure the plugin in that file i.e. the Module mesos-master block should be in /opt/collectd/etc/collectd.conf , not tucked down in the python module path directory. 3. Are you sure your master listens on localhost? Mine doesn't, I needed to set that Host line to match the IP I set that master to listen on ( e.g. in /etc/mesos-master/ip ). Pretty sure one of those will do the trick (NB: you'll only get metrics from the elected master; the 'standby' masters still get polled but collectd will ignore any data from them unless they're the primary) On 11 March 2015 at 19:52, Dan Dong dongda...@gmail.com wrote: Hi, Dick, I put the plugin under: $ ls -l /opt/collectd/lib/collectd/plugins/python/ total 504 -rw-r--r-- 1 root root345 Mar 10 19:40 mesos-master.conf -rw-r--r-- 1 root root 1 Mar 10 15:06 mesos-master.py -rw-r--r-- 1 root root322 Mar 10 19:44 mesos-slave.conf -rw-r--r-- 1 root root 6808 Mar 10 15:06 mesos-slave.py -rw-r--r-- 1 root root 288892 Mar 10 19:35 python.a -rwxr-xr-x 1 root root969 Mar 10 19:35 python.la -rwxr-xr-x 1 root root 188262 Mar 10 19:35 python.so And in /opt/collectd/etc/collectd.conf, I set: LoadPlugin python Globals true /LoadPlugin . Plugin python ModulePath /opt/collectd/lib/collectd/plugins/python/ LogTraces true /Plugin $ cat /opt/collectd/lib/collectd/plugins/python/mesos-master.conf LoadPlugin python Globals true /LoadPlugin Plugin python ModulePath /opt/collectd/lib/collectd/plugins/python/ Import mesos-master Module mesos-master Host localhost Port 5050 Verbose false Version 0.21.0 /Module /Plugin Anything wrong with the above settings? Cheers, Dan 2015-03-10 17:21 GMT-05:00 Dick Davies d...@hellooperator.net: Hi Dan The .py files (the plugin) live in the collectd python path, it sounds like maybe you're not loading the plugin .conf file into your collectd config? The output will depend on what your collectd is set to write to, I use it with write_graphite. On 10 March 2015 at 20:41, Dan Dong dongda...@gmail.com wrote: Hi, All, Does anybody use this mesos-collectd-plugin: https://github.com/rayrod2030/collectd-mesos I have installed collectd and this plugin, then configured it as instructions and restarted the collectd daemon, why seems nothing happens on the mesos:5050 web UI( python plugin has been turned on in collectd.conf). My question is: 1. Should I install collectd and this mesos-collectd-plugin on each master and slave nodes and restart collectd daemon? (This is what I have done.) 2. Should the config file mesos-master.conf only configured on master node and mesos-slave.conf only configured on slave node?(This is what I have done.) Or both of them should only appear on master node? 3. Is there an example( or a figure) of what output one is expected to see by this plugin? Cheers, Dan
Re: mesos-collectd-plugin
Alright. You've convinced me I should put this online, but I'll have to clean it up first. I would have originally but had one weird question. In my company, we almost certainly will have multiple potentially large Mesos clusters as different failure domains. The issue I've got is relatively simple, the solution is not. The collectd daemon runs on every single Mesos node (master or slave) in the firm. If it is running on a master, it will get data from localhost:$master_port/metrics/snapshot. It then checked if json_data[master/elected] == 1 and ONLY reports the data if it is the elected master. By default it uses a namespace like: mesos.master.* If you have multiple clusters this obviously won't work, so I added an optional cluster name attribute to the collectd config. That way it will report to: mesos.archive.* where your mesos cluster name is archive. How would you propose a decent way to namespace master stats that are cluster wide. I had to lie (in the collectd plugin code) and change the host as a metric such as: collectd.$HOSTNAME.master.* seems to make absolutely no sense when there is a single elected master per cluster. On Tuesday, March 10, 2015, Steven Schlansker sschlans...@opentable.com wrote: We would use (and probably contribute back to!) such an improved plugin as well, if you do polish it up be sure to announce to this list :) On Mar 10, 2015, at 2:05 PM, Dan Dong dongda...@gmail.com javascript:; wrote: Hi, Jeff, Thanks, is your plugin working together with collectd? It would be great to publish it! A general question for Mesos: Is there a method to monitor CPU/Memory/Disk usages of jobs from different frameworks(e.g: Hadoop/Mapreduce, Spark etc)? (Not necessarily to generate figures, text format numbers are quite enough.) So e.g for a hadoop job, when it's finished, we can collect the general metrics of it? Ideally although there are many jobs from different frameworks running at the same time on mesos, we still could get their metrics respectively. Cheers, Dan 2015-03-10 15:46 GMT-05:00 Jeff Schroeder jeffschroe...@computer.org javascript:;: I installed it and played with it for a bit but was somewhat underwhelmed with it. It doesn't support slaves and all of the hardcoding with duplication isn't my favorite. I ended up writing a single plugin to support both masters and slaves and putting it on every node in my Mesos cluster. Would it be worth polishing up a bit and throwing on github? On Tuesday, March 10, 2015, Dan Dong dongda...@gmail.com javascript:; wrote: Hi, All, Does anybody use this mesos-collectd-plugin: https://github.com/rayrod2030/collectd-mesos I have installed collectd and this plugin, then configured it as instructions and restarted the collectd daemon, why seems nothing happens on the mesos:5050 web UI( python plugin has been turned on in collectd.conf). My question is: 1. Should I install collectd and this mesos-collectd-plugin on each master and slave nodes and restart collectd daemon? (This is what I have done.) 2. Should the config file mesos-master.conf only configured on master node and mesos-slave.conf only configured on slave node?(This is what I have done.) Or both of them should only appear on master node? 3. Is there an example( or a figure) of what output one is expected to see by this plugin? Cheers, Dan -- Text by Jeff, typos by iPhone -- Text by Jeff, typos by iPhone
