Re: [openstack-dev] [Fuel] fuel master monitoring

2015-01-27 Thread Fabrizio Soppelsa
The proposed monitoring options make sense, since the Fuel master has nothing yet, but I would like to ressurect this thread to see if we can discuss some strategies in order to avoid the /var/log fillup with consequent docker containers corruption. Now, a customer facing this corruption can

Re: [openstack-dev] [Fuel] fuel master monitoring

2015-01-07 Thread Przemyslaw Kaminski
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hello, The updated version of monitoring code is available here: https://review.openstack.org/#/c/137785/ This is based on monit as was agreed in this thread. The drawback of monit is that basically it's a very simple system that doesn't track

Re: [openstack-dev] [Fuel] fuel master monitoring

2015-01-07 Thread Andrew Woodward
On Wed, Jan 7, 2015 at 12:59 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hello, The updated version of monitoring code is available here: https://review.openstack.org/#/c/137785/ This is based on monit as was agreed in this

Re: [openstack-dev] [Fuel] fuel master monitoring

2015-01-05 Thread Andrew Woodward
There are two threads here that need to be unraveled from each other. 1. We need to prevent fuel from doing anything if the OS is out of disk space. It leads to a very broken database from which it requires a developer to reset to a usable state. From this point we need to * develop a method for

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-27 Thread Dmitriy Shulyak
Is it possible to send http requests from monit, e.g for creating notifications? I scanned through the docs and found only alerts for sending mail, also where token (username/pass) for monit will be stored? Or maybe there is another plan? without any api interaction On Thu, Nov 27, 2014 at 9:39

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-27 Thread Przemyslaw Kaminski
I mean with monit you can execute arbitrary scripts so use curl? Or save them directly to DB? http://omgitsmgp.com/2013/09/07/a-monit-primer/ I guess some data has to be stored in a configuration file (either DB credentials or Nailgun API URL at least, if we were to create notifications via

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-27 Thread Aleksandr Didenko
Hi, Dmitriy, first of all, monit can provide HTTP interface for communication - so it's possible to poll that this interface to get info or even control monit (stop/start/restart service, stop/start monitoring of a service, etc). Secondly, you can configure different triggers in monit and set

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-27 Thread Simon Pasquier
I've added another option to the Etherpad: collectd can do basic threshold monitoring and run any kind of scripts on alert notifications. The other advantage of collectd would be the RRD graphs for (almost) free. Of course since monit is already supported in Fuel, this is the fastest path to get

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Stanislaw Bogatkin
:* Monday, November 24, 2014 6:42:39 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Przemyslaw Kaminski
(not for usage questions) *Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com mailto:sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Fox, Kevin M
From: Przemyslaw Kaminski Sent: Wednesday, November 26, 2014 2:50:03 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Fuel] fuel master monitoring I agree, this was supposed to be small. P. On 11/26/2014 11:03 AM

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Igor Kalnitsky
Folks, Maybe I understand some things wrong, but Zabbix is a different story. We deploy Zabbix per cluster, so it doesn't monitor for *all* slaves or master node. It monitors only one cluster. Therefore I see no reasons to choose Zabbix over monit. I mean, it shouldn't be We MUST use Zabbix

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Przemyslaw Kaminski
We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Jay Pipes
On 11/26/2014 10:22 AM, Przemyslaw Kaminski wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc.

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Sergii Golovatiuk
Hi, I would do both to compare. monit and Sensu have own advantages. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Stanislaw Bogatkin
As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Sergii Golovatiuk
Jay, Fuel uses watchdog service for container to restart it in case of issues. We have the same problem with containers when disk is full -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:39 PM, Jay Pipes jaypi...@gmail.com wrote: On 11/26/2014 10:22

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Sergii Golovatiuk
Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com wrote: As for me - zabbix is overkill for one node. Zabbix

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Jay Pipes
On 11/26/2014 11:54 AM, Sergii Golovatiuk wrote: Jay, Fuel uses watchdog service for container to restart it in case of issues. We have the same problem with containers when disk is full I see. I guess I don't quite understand why Zabbix isn't just used for everything -- after all, the

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-26 Thread Przemyslaw Kaminski
This I didn't know. It's true in fact, I checked the manifests. Though monit is not deployed yet because of lack of packages in Fuel ISO. Anyways, I think the argument about using yet another monitoring service is now rendered invalid. So +1 for monit? :) P. On 11/26/2014 05:55 PM, Sergii

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-24 Thread Przemyslaw Kaminski
, November 21, 2014 12:57:45 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring I have nothing against using some 3rd party service. But I thought this was to be small -- disk monitoring only notifying the user

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-24 Thread Sergii Golovatiuk
Kevin -- *From:* Dmitriy Shulyak *Sent:* Friday, November 21, 2014 12:57:45 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring I have nothing against using some 3rd party service. But I

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-24 Thread Tomasz Napierala
On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol. I am proposing Sensu. It's architecture is tiny and elegant. Also it uses rabbitmq as

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-24 Thread Rob Basham
/24/2014 06:46 AM Subject: Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-24 Thread Przemyslaw Kaminski
@lists.openstack.org Date: 11/24/2014 06:46 AM Subject: Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-24 Thread Fox, Kevin M
:42:39 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-21 Thread Przemyslaw Kaminski
I have nothing against using some 3rd party service. But I thought this was to be small -- disk monitoring only notifying the user, not stats collecting. That's why I added the code to Fuel codebase. If you want external service you need to remember about such details as, say, duplicate

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-21 Thread Przemyslaw Kaminski
BTW, there's also Monit http://mmonit.com/monit/ (though it's in C) that looks quite nice. Some config examples: http://omgitsmgp.com/2013/09/07/a-monit-primer/ P. On 11/20/2014 09:13 PM, Dmitriy Shulyak wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-21 Thread Dmitriy Shulyak
I have nothing against using some 3rd party service. But I thought this was to be small -- disk monitoring only notifying the user, not stats collecting. That's why I added the code to Fuel codebase. If you want external service you need to remember about such details as, say, duplicate

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-21 Thread Matthew Mosesohn
I'm okay with Sensu or Monit, just as long as the results of monitoring can be represented in a web UI and has a configurable option for email alerting. Tight integration with Fuel Web is a nice-to-have (via AMQP perhaps), but anything that can solve our out-of-disk scenario is ideal. I did my

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-21 Thread Igor Kalnitsky
I heard about Monit a lot of good reviews, but unfortunately it looks like Monit doesn't support plugins and doesn't provide API. It may be a stumbling block if one day we decide to go deeper in monitoring task. On Fri, Nov 21, 2014 at 11:01 AM, Matthew Mosesohn mmoses...@mirantis.com wrote: I'm

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-21 Thread Przemyslaw Kaminski
There's also OpenStack's monasca-agent: https://github.com/stackforge/monasca-agent We could try to run it standalone (without Monasca API), add a plugin for it that checks disk and sends notification straight to Fuel DB and omits generating Forwarder requests. Or set up a fake API though

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-21 Thread Fox, Kevin M
How about this? https://wiki.openstack.org/wiki/Monasca Kevin From: Dmitriy Shulyak Sent: Friday, November 21, 2014 12:57:45 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Fuel] fuel master monitoring I have

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-20 Thread Dmitriy Shulyak
Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-20 Thread Igor Kalnitsky
+1 for Dima's suggestion. We need to stop reinventing the wheel. There are a lot tools around us, so let's pick one of them and will use it. On Thu, Nov 20, 2014 at 10:13 PM, Dmitriy Shulyak dshul...@mirantis.com wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-20 Thread Lukasz Oles
+1 also. There are so many monitoring tools, but maybe not something written in ruby ;) On Thu, Nov 20, 2014 at 10:40 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: +1 for Dima's suggestion. We need to stop reinventing the wheel. There are a lot tools around us, so let's pick one of them and

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-20 Thread Sergii Golovatiuk
I would rather compare features than ruby vs python. Best Regards, Sergii Golovatiuk On 20 Nov 2014, at 23:20, Lukasz Oles lo...@mirantis.com wrote: +1 also. There are so many monitoring tools, but maybe not something written in ruby ;) On Thu, Nov 20, 2014 at 10:40 PM, Igor Kalnitsky

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-20 Thread Lukasz Oles
On Thu, Nov 20, 2014 at 11:28 PM, Sergii Golovatiuk sgolovat...@mirantis.com wrote: I would rather compare features than ruby vs python. It make sense only as long as you don't need to debug it Best Regards, Sergii Golovatiuk On 20 Nov 2014, at 23:20, Lukasz Oles lo...@mirantis.com

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-12 Thread Tomasz Napierala
On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-06 Thread Anton Zemlyanov
We can add a notification to FuelWeb, no additional software or user actions are required. I would not overestimate this method though, it is in no way the robust monitoring system. Forcing user to do something on a regular basis is unlikely to work. Anton On Thu, Nov 6, 2014 at 11:55 AM,

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-06 Thread Przemyslaw Kaminski
I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. P. On 11/06/2014 09:59 AM, Anton Zemlyanov wrote: We can add a notification to FuelWeb, no additional software or user actions are required. I would not overestimate this method though,

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-05 Thread Anton Zemlyanov
Monitoring of the Fuel master's disk space is the special case. I really wonder why Fuel master have no HA option, disk overflow can be predicted but many other failures cannot. HA is a solution of the 'single point of failure' problem. The current monitoring recommendations (

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-05 Thread Oleg Gelbukh
Hello, As far as I can tell, disk space monitoring is pretty useless, unless Fuel provides user with some means to automatically cleanup of stored data (i.e. remove obsolete diagnostic snapshots, etc). Otherwise, it will be only useful for experienced Fuel developers who know how to properly

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-05 Thread Dmitry Borodaenko
Even one additional hardware node required to host the Fuel master is seen by many users as excessive. Unless you can come up with an architecture that adds HA capability to Fuel without increasing its hardware footprint by 2 more nodes, it's just not worth it. The only operational aspect of the

Re: [openstack-dev] [Fuel] fuel master monitoring

2014-11-05 Thread Przemyslaw Kaminski
I think we're missing the point here. What I meant adding a simple monitoring system that informed the user via UI/CLI/email/whatever of low resources on fuel master node. That's it. HA here is not an option -- if, despite of warnings, the user still continues to use fuel and disk becomes