Re: [openstack-dev] [Fuel] fuel master monitoring
The proposed monitoring options make sense, since the Fuel master has nothing yet, but I would like to ressurect this thread to see if we can discuss some strategies in order to avoid the /var/log fillup with consequent docker containers corruption. Now, a customer facing this corruption can recover the master, as described in our documentation [1]. Of course, if we could avoid the problem at all, it would be even better. So, do we have a strategy in progress on this? As a first attempt, I filed a blueprint, proposing to separate /var/log [2]. Thanks, Fabrizio. [1] http://docs.mirantis.com/openstack/fuel/fuel-5.1/operations.html#fuel-master-and-docker-disk-space-troubleshooting http://docs.mirantis.com/openstack/fuel/fuel-5.1/operations.html#fuel-master-and-docker-disk-space-troubleshooting [2] https://blueprints.launchpad.net/fuel/+spec/isolate-var-log-on-master https://blueprints.launchpad.net/fuel/+spec/isolate-var-log-on-master On 21 Nov 2014, at 12:01, Matthew Mosesohn mmoses...@mirantis.com wrote: I'm okay with Sensu or Monit, just as long as the results of monitoring can be represented in a web UI and has a configurable option for email alerting. Tight integration with Fuel Web is a nice-to-have (via AMQP perhaps), but anything that can solve our out-of-disk scenario is ideal. I did my best to tune our logging and logs rotation, but monitoring is the most sensible approach. -Matthew On Fri, Nov 21, 2014 at 12:21 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: BTW, there's also Monit http://mmonit.com/monit/ (though it's in C) that looks quite nice. Some config examples: http://omgitsmgp.com/2013/09/07/a-monit-primer/ P. On 11/20/2014 09:13 PM, Dmitriy Shulyak wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hello, The updated version of monitoring code is available here: https://review.openstack.org/#/c/137785/ This is based on monit as was agreed in this thread. The drawback of monit is that basically it's a very simple system that doesn't track state of checkers so still some Python code is needed so that user isn't spammed with low disk space notifications every minute. On 01/05/2015 10:40 PM, Andrew Woodward wrote: There are two threads here that need to be unraveled from each other. 1. We need to prevent fuel from doing anything if the OS is out of disk space. It leads to a very broken database from which it requires a developer to reset to a usable state. From this point we need to * develop a method for locking down the DB writes so that fuel becomes RO until space is freed It's true that full disk space + DB writes can result in fatal database failure. I just don't know if we can lock the DB just like that? What if deployment is in progress? I think the first way to reduce disk space usage would be to set logging level to WARNING instead of DEBUG. It's good to have DEBUG during development but I don't think it's that good for production. Besides it slows down deployment much, from what I observed. * develop a method (or re-use existing) to notify the user that a serious error state exists on the host. ( that could not be dismissed) Well this is done already in the review I've linked above. It basically posts a notification to the UI system. Everything still works as before though until the disk is full. The CLI doesn't communicate in any way with notifications AFAIK so the warning is not shown there. * we need some API that can lock / unlock the DB * we need some monitor process that will trigger the lock/unlock This one can be easily changed with the code in the above review request. 2. We need monitoring for the master node and fuel components in general as discussed at length above. unless we intend to use this to also monitor the services on deployed nodes (likely bad), then what we use to do this is irrelevant to getting this started. If we are intending to use this to also monitor deployed nodes, (again bad for the fuel node to do) then we need to standardize with what we monitor the cloud with (Zabbix currently) and offer a single pane of glass. Federation in the monitoring becomes a critical requirement here as having more than one pane of glass is an operations nightmare. AFAIK installation of Zabbix is optional. We want obligatory monitoring of the master which would somehow force its installation on the cloud nodes. P. Completing #1 is very important in the near term as I have had to un-brick several deployments over it already. Also, in my mind these are also separate tasks. On Thu, Nov 27, 2014 at 1:19 AM, Simon Pasquier spasqu...@mirantis.com wrote: I've added another option to the Etherpad: collectd can do basic threshold monitoring and run any kind of scripts on alert notifications. The other advantage of collectd would be the RRD graphs for (almost) free. Of course since monit is already supported in Fuel, this is the fastest path to get something done. Simon On Thu, Nov 27, 2014 at 9:53 AM, Dmitriy Shulyak dshul...@mirantis.com wrote: Is it possible to send http requests from monit, e.g for creating notifications? I scanned through the docs and found only alerts for sending mail, also where token (username/pass) for monit will be stored? Or maybe there is another plan? without any api interaction On Thu, Nov 27, 2014 at 9:39 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: This I didn't know. It's true in fact, I checked the manifests. Though monit is not deployed yet because of lack of packages in Fuel ISO. Anyways, I think the argument about using yet another monitoring service is now rendered invalid. So +1 for monit? :) P. On 11/26/2014 05:55 PM, Sergii Golovatiuk wrote: Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com wrote: As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov 26, 2014 at 6:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On
Re: [openstack-dev] [Fuel] fuel master monitoring
On Wed, Jan 7, 2015 at 12:59 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hello, The updated version of monitoring code is available here: https://review.openstack.org/#/c/137785/ This is based on monit as was agreed in this thread. The drawback of monit is that basically it's a very simple system that doesn't track state of checkers so still some Python code is needed so that user isn't spammed with low disk space notifications every minute. can we make the alert an asserted state that needs to be cleared to remove the warning? that way once asserted it won't re-raise the error. On 01/05/2015 10:40 PM, Andrew Woodward wrote: There are two threads here that need to be unraveled from each other. 1. We need to prevent fuel from doing anything if the OS is out of disk space. It leads to a very broken database from which it requires a developer to reset to a usable state. From this point we need to * develop a method for locking down the DB writes so that fuel becomes RO until space is freed It's true that full disk space + DB writes can result in fatal database failure. I just don't know if we can lock the DB just like that? What if deployment is in progress? We could do some form of complicated maths around guessing how much space we need to finish a task but lets say at 20% free space we warn 5% free space we block tasks from starting (both should be configurable, and probably ignore-able ) then we also need to have a separate volume for the DB from the logs. This will remove the need do any complicated logic around blocking in the DB I think the first way to reduce disk space usage would be to set logging level to WARNING instead of DEBUG. It's good to have DEBUG during development but I don't think it's that good for production. Besides it slows down deployment much, from what I observed. The default logging level is supposed to be WARNING not debug. * develop a method (or re-use existing) to notify the user that a serious error state exists on the host. ( that could not be dismissed) Well this is done already in the review I've linked above. It basically posts a notification to the UI system. Everything still works as before though until the disk is full. The CLI doesn't communicate in any way with notifications AFAIK so the warning is not shown there. * we need some API that can lock / unlock the DB * we need some monitor process that will trigger the lock/unlock This one can be easily changed with the code in the above review request. I think this should become blocking tasks, not the db its self as above 2. We need monitoring for the master node and fuel components in general as discussed at length above. unless we intend to use this to also monitor the services on deployed nodes (likely bad), then what we use to do this is irrelevant to getting this started. If we are intending to use this to also monitor deployed nodes, (again bad for the fuel node to do) then we need to standardize with what we monitor the cloud with (Zabbix currently) and offer a single pane of glass. Federation in the monitoring becomes a critical requirement here as having more than one pane of glass is an operations nightmare. AFAIK installation of Zabbix is optional. We want obligatory monitoring of the master which would somehow force its installation on the cloud nodes. P. Completing #1 is very important in the near term as I have had to un-brick several deployments over it already. Also, in my mind these are also separate tasks. On Thu, Nov 27, 2014 at 1:19 AM, Simon Pasquier spasqu...@mirantis.com wrote: I've added another option to the Etherpad: collectd can do basic threshold monitoring and run any kind of scripts on alert notifications. The other advantage of collectd would be the RRD graphs for (almost) free. Of course since monit is already supported in Fuel, this is the fastest path to get something done. Simon On Thu, Nov 27, 2014 at 9:53 AM, Dmitriy Shulyak dshul...@mirantis.com wrote: Is it possible to send http requests from monit, e.g for creating notifications? I scanned through the docs and found only alerts for sending mail, also where token (username/pass) for monit will be stored? Or maybe there is another plan? without any api interaction On Thu, Nov 27, 2014 at 9:39 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: This I didn't know. It's true in fact, I checked the manifests. Though monit is not deployed yet because of lack of packages in Fuel ISO. Anyways, I think the argument about using yet another monitoring service is now rendered invalid. So +1 for monit? :) P. On 11/26/2014 05:55 PM, Sergii Golovatiuk wrote: Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw
Re: [openstack-dev] [Fuel] fuel master monitoring
There are two threads here that need to be unraveled from each other. 1. We need to prevent fuel from doing anything if the OS is out of disk space. It leads to a very broken database from which it requires a developer to reset to a usable state. From this point we need to * develop a method for locking down the DB writes so that fuel becomes RO until space is freed * develop a method (or re-use existing) to notify the user that a serious error state exists on the host. ( that could not be dismissed) * we need some API that can lock / unlock the DB * we need some monitor process that will trigger the lock/unlock 2. We need monitoring for the master node and fuel components in general as discussed at length above. unless we intend to use this to also monitor the services on deployed nodes (likely bad), then what we use to do this is irrelevant to getting this started. If we are intending to use this to also monitor deployed nodes, (again bad for the fuel node to do) then we need to standardize with what we monitor the cloud with (Zabbix currently) and offer a single pane of glass. Federation in the monitoring becomes a critical requirement here as having more than one pane of glass is an operations nightmare. Completing #1 is very important in the near term as I have had to un-brick several deployments over it already. Also, in my mind these are also separate tasks. On Thu, Nov 27, 2014 at 1:19 AM, Simon Pasquier spasqu...@mirantis.com wrote: I've added another option to the Etherpad: collectd can do basic threshold monitoring and run any kind of scripts on alert notifications. The other advantage of collectd would be the RRD graphs for (almost) free. Of course since monit is already supported in Fuel, this is the fastest path to get something done. Simon On Thu, Nov 27, 2014 at 9:53 AM, Dmitriy Shulyak dshul...@mirantis.com wrote: Is it possible to send http requests from monit, e.g for creating notifications? I scanned through the docs and found only alerts for sending mail, also where token (username/pass) for monit will be stored? Or maybe there is another plan? without any api interaction On Thu, Nov 27, 2014 at 9:39 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: This I didn't know. It's true in fact, I checked the manifests. Though monit is not deployed yet because of lack of packages in Fuel ISO. Anyways, I think the argument about using yet another monitoring service is now rendered invalid. So +1 for monit? :) P. On 11/26/2014 05:55 PM, Sergii Golovatiuk wrote: Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com wrote: As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov 26, 2014 at 6:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [Fuel] fuel master monitoring
Is it possible to send http requests from monit, e.g for creating notifications? I scanned through the docs and found only alerts for sending mail, also where token (username/pass) for monit will be stored? Or maybe there is another plan? without any api interaction On Thu, Nov 27, 2014 at 9:39 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: This I didn't know. It's true in fact, I checked the manifests. Though monit is not deployed yet because of lack of packages in Fuel ISO. Anyways, I think the argument about using yet another monitoring service is now rendered invalid. So +1 for monit? :) P. On 11/26/2014 05:55 PM, Sergii Golovatiuk wrote: Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com wrote: As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov 26, 2014 at 6:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I mean with monit you can execute arbitrary scripts so use curl? Or save them directly to DB? http://omgitsmgp.com/2013/09/07/a-monit-primer/ I guess some data has to be stored in a configuration file (either DB credentials or Nailgun API URL at least, if we were to create notifications via the API). I proposed a hand-crafted solution https://review.openstack.org/#/c/135314/ that lives in fuel-web code and uses settings.yaml so no config file is necessary. It has the drawback though that the nailgun code lives inside a Docker container so the monitoring data isn't reliable. P. On 11/27/2014 09:53 AM, Dmitriy Shulyak wrote: Is it possible to send http requests from monit, e.g for creating notifications? I scanned through the docs and found only alerts for sending mail, also where token (username/pass) for monit will be stored? Or maybe there is another plan? without any api interaction On Thu, Nov 27, 2014 at 9:39 AM, Przemyslaw Kaminski pkamin...@mirantis.com mailto:pkamin...@mirantis.com wrote: This I didn't know. It's true in fact, I checked the manifests. Though monit is not deployed yet because of lack of packages in Fuel ISO. Anyways, I think the argument about using yet another monitoring service is now rendered invalid. So +1 for monit? :) P. On 11/26/2014 05:55 PM, Sergii Golovatiuk wrote: Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com mailto:sbogat...@mirantis.com wrote: As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov 26, 2014 at 6:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com mailto:pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Hi, Dmitriy, first of all, monit can provide HTTP interface for communication - so it's possible to poll that this interface to get info or even control monit (stop/start/restart service, stop/start monitoring of a service, etc). Secondly, you can configure different triggers in monit and set appropriate actions, for example running some script - and that script can do what ever you like. As for built-in possibility to send http requests - I've never heard about it. Regards, Alex On Thu, Nov 27, 2014 at 10:53 AM, Dmitriy Shulyak dshul...@mirantis.com wrote: Is it possible to send http requests from monit, e.g for creating notifications? I scanned through the docs and found only alerts for sending mail, also where token (username/pass) for monit will be stored? Or maybe there is another plan? without any api interaction On Thu, Nov 27, 2014 at 9:39 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: This I didn't know. It's true in fact, I checked the manifests. Though monit is not deployed yet because of lack of packages in Fuel ISO. Anyways, I think the argument about using yet another monitoring service is now rendered invalid. So +1 for monit? :) P. On 11/26/2014 05:55 PM, Sergii Golovatiuk wrote: Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com wrote: As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov 26, 2014 at 6:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I've added another option to the Etherpad: collectd can do basic threshold monitoring and run any kind of scripts on alert notifications. The other advantage of collectd would be the RRD graphs for (almost) free. Of course since monit is already supported in Fuel, this is the fastest path to get something done. Simon On Thu, Nov 27, 2014 at 9:53 AM, Dmitriy Shulyak dshul...@mirantis.com wrote: Is it possible to send http requests from monit, e.g for creating notifications? I scanned through the docs and found only alerts for sending mail, also where token (username/pass) for monit will be stored? Or maybe there is another plan? without any api interaction On Thu, Nov 27, 2014 at 9:39 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: This I didn't know. It's true in fact, I checked the manifests. Though monit is not deployed yet because of lack of packages in Fuel ISO. Anyways, I think the argument about using yet another monitoring service is now rendered invalid. So +1 for monit? :) P. On 11/26/2014 05:55 PM, Sergii Golovatiuk wrote: Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com wrote: As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov 26, 2014 at 6:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Hi all, As I understand, we just need to monitoring one node - Fuel master. For slave nodes we already have a solution - zabbix. So, in that case why we need some complicated stuff like monasca? Let's use something small, like monit or sensu. On Mon, Nov 24, 2014 at 10:36 PM, Fox, Kevin M kevin@pnnl.gov wrote: One of the selling points of tripleo is to reuse as much as possible from the cloud, to make it easier to deploy. While monasca may be more complicated, if it ends up being a component everyone learns, then its not as bad as needing to learn two different monitoring technologies. You could say the same thing cobbler vs ironic. the whole Ironic stack is much more complicated. But for an openstack admin, its easier since a lot of existing knowlege applies. Just something to consider. Thanks, Kevin -- *From:* Tomasz Napierala *Sent:* Monday, November 24, 2014 6:42:39 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol. I am proposing Sensu. It's architecture is tiny and elegant. Also it uses rabbitmq as transport so we won't need to introduce new protocol. Do we really need such complicated stuff? Sensu is huge project, and it's footprint is quite large. Monit can alert using scripts, can we use it instead of API? Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I agree, this was supposed to be small. P. On 11/26/2014 11:03 AM, Stanislaw Bogatkin wrote: Hi all, As I understand, we just need to monitoring one node - Fuel master. For slave nodes we already have a solution - zabbix. So, in that case why we need some complicated stuff like monasca? Let's use something small, like monit or sensu. On Mon, Nov 24, 2014 at 10:36 PM, Fox, Kevin M kevin@pnnl.gov mailto:kevin@pnnl.gov wrote: One of the selling points of tripleo is to reuse as much as possible from the cloud, to make it easier to deploy. While monasca may be more complicated, if it ends up being a component everyone learns, then its not as bad as needing to learn two different monitoring technologies. You could say the same thing cobbler vs ironic. the whole Ironic stack is much more complicated. But for an openstack admin, its easier since a lot of existing knowlege applies. Just something to consider. Thanks, Kevin * * *From:* Tomasz Napierala *Sent:* Monday, November 24, 2014 6:42:39 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com mailto:sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol. I am proposing Sensu. It's architecture is tiny and elegant. Also it uses rabbitmq as transport so we won't need to introduce new protocol. Do we really need such complicated stuff? Sensu is huge project, and it's footprint is quite large. Monit can alert using scripts, can we use it instead of API? Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com mailto:tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Thanks, Kevin From: Przemyslaw Kaminski Sent: Wednesday, November 26, 2014 2:50:03 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Fuel] fuel master monitoring I agree, this was supposed to be small. P. On 11/26/2014 11:03 AM, Stanislaw Bogatkin wrote: Hi all, As I understand, we just need to monitoring one node - Fuel master. For slave nodes we already have a solution - zabbix. So, in that case why we need some complicated stuff like monasca? Let's use something small, like monit or sensu. On Mon, Nov 24, 2014 at 10:36 PM, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: One of the selling points of tripleo is to reuse as much as possible from the cloud, to make it easier to deploy. While monasca may be more complicated, if it ends up being a component everyone learns, then its not as bad as needing to learn two different monitoring technologies. You could say the same thing cobbler vs ironic. the whole Ironic stack is much more complicated. But for an openstack admin, its easier since a lot of existing knowlege applies. Just something to consider. Thanks, Kevin From: Tomasz Napierala Sent: Monday, November 24, 2014 6:42:39 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.commailto:sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol. I am proposing Sensu. It's architecture is tiny and elegant. Also it uses rabbitmq as transport so we won't need to introduce new protocol. Do we really need such complicated stuff? Sensu is huge project, and it's footprint is quite large. Monit can alert using scripts, can we use it instead of API? Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.commailto:tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Folks, Maybe I understand some things wrong, but Zabbix is a different story. We deploy Zabbix per cluster, so it doesn't monitor for *all* slaves or master node. It monitors only one cluster. Therefore I see no reasons to choose Zabbix over monit. I mean, it shouldn't be We MUST use Zabbix because we use it for our clusters. - Igor P.S: Personally, I'd like to use either Monit or Sensu. On Wed, Nov 26, 2014 at 3:58 PM, Jay Pipes jaypi...@gmail.com wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
On 11/26/2014 10:22 AM, Przemyslaw Kaminski wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. Why not use the same thing for monitoring the Fuel master host as we do for the docker containers/cluster? P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Hi, I would do both to compare. monit and Sensu have own advantages. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov 26, 2014 at 6:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Jay, Fuel uses watchdog service for container to restart it in case of issues. We have the same problem with containers when disk is full -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:39 PM, Jay Pipes jaypi...@gmail.com wrote: On 11/26/2014 10:22 AM, Przemyslaw Kaminski wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. Why not use the same thing for monitoring the Fuel master host as we do for the docker containers/cluster? P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com wrote: As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov 26, 2014 at 6:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
On 11/26/2014 11:54 AM, Sergii Golovatiuk wrote: Jay, Fuel uses watchdog service for container to restart it in case of issues. We have the same problem with containers when disk is full I see. I guess I don't quite understand why Zabbix isn't just used for everything -- after all, the puppet manifests already exist for it and are used for monitoring other things apparently. -jay On Wed, Nov 26, 2014 at 4:39 PM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 11/26/2014 10:22 AM, Przemyslaw Kaminski wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. Why not use the same thing for monitoring the Fuel master host as we do for the docker containers/cluster? P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
This I didn't know. It's true in fact, I checked the manifests. Though monit is not deployed yet because of lack of packages in Fuel ISO. Anyways, I think the argument about using yet another monitoring service is now rendered invalid. So +1 for monit? :) P. On 11/26/2014 05:55 PM, Sergii Golovatiuk wrote: Monit is easy and is used to control states of Compute nodes. We can adopt it for master node. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com mailto:sbogat...@mirantis.com wrote: As for me - zabbix is overkill for one node. Zabbix Server + Agent + Frontend + DB + HTTP server, and all of it for one node? Why not use something that was developed for monitoring one node, doesn't have many deps and work out of the box? Not necessarily Monit, but something similar. On Wed, Nov 26, 2014 at 6:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com mailto:pkamin...@mirantis.com wrote: We want to monitor Fuel master node while Zabbix is only on slave nodes and not on master. The monitoring service is supposed to be installed on Fuel master host (not inside a Docker container) and provide basic info about free disk space, etc. P. On 11/26/2014 02:58 PM, Jay Pipes wrote: On 11/26/2014 08:18 AM, Fox, Kevin M wrote: So then in the end, there will be 3 monitoring systems to learn, configure, and debug? Monasca for cloud users, zabbix for most of the physical systems, and sensu or monit to be small? Seems very complicated. If not just monasca, why not the zabbix thats already being deployed? Yes, I had the same thoughts... why not just use zabbix since it's used already? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I proposed monasca-agent in a previous mail in this thread. P. On 11/21/2014 04:48 PM, Fox, Kevin M wrote: How about this? https://wiki.openstack.org/wiki/Monasca Kevin * * *From:* Dmitriy Shulyak *Sent:* Friday, November 21, 2014 12:57:45 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring I have nothing against using some 3rd party service. But I thought this was to be small -- disk monitoring only notifying the user, not stats collecting. That's why I added the code to Fuel codebase. If you want external service you need to remember about such details as, say, duplicate settings (database credentials at least) and I thought this was an overkill for such simple functionality. Yes, it will be much more complex than simple daemon that creates notifications but our application is operating in isolated containers, and most of the resources cant be discovered from any particular container. So if we will want to extend it, with another task, like monitoring pool of dhcp addresses - we will end up with some kindof server-agent architecture, and this is a lot of work to do Also, for a 3rd party service, notification injecting code still needs to be written as a plugin -- that's why I also don't think Ruby is a good idea :) Afaik there is a way to write python plugins for sensu, but if there is monitoring app in python, that have friendly support for extensions, I am +1 for python So in the end I don't know if we'll have that much less code with a 3rd party service. But if you want a statistics collector then maybe it's OK. I think that monitoring application is fits there, and we kindof already introducing our whell for collecting statistic from openstack. I would like to know what guys who was working on stats in 6.0 thinking about it. So it is TBD ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol. I am proposing Sensu. It's architecture is tiny and elegant. Also it uses rabbitmq as transport so we won't need to introduce new protocol. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Mon, Nov 24, 2014 at 10:55 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I proposed monasca-agent in a previous mail in this thread. P. On 11/21/2014 04:48 PM, Fox, Kevin M wrote: How about this? https://wiki.openstack.org/wiki/Monasca Kevin -- *From:* Dmitriy Shulyak *Sent:* Friday, November 21, 2014 12:57:45 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring I have nothing against using some 3rd party service. But I thought this was to be small -- disk monitoring only notifying the user, not stats collecting. That's why I added the code to Fuel codebase. If you want external service you need to remember about such details as, say, duplicate settings (database credentials at least) and I thought this was an overkill for such simple functionality. Yes, it will be much more complex than simple daemon that creates notifications but our application is operating in isolated containers, and most of the resources cant be discovered from any particular container. So if we will want to extend it, with another task, like monitoring pool of dhcp addresses - we will end up with some kindof server-agent architecture, and this is a lot of work to do Also, for a 3rd party service, notification injecting code still needs to be written as a plugin -- that's why I also don't think Ruby is a good idea :) Afaik there is a way to write python plugins for sensu, but if there is monitoring app in python, that have friendly support for extensions, I am +1 for python So in the end I don't know if we'll have that much less code with a 3rd party service. But if you want a statistics collector then maybe it's OK. I think that monitoring application is fits there, and we kindof already introducing our whell for collecting statistic from openstack. I would like to know what guys who was working on stats in 6.0 thinking about it. So it is TBD ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol. I am proposing Sensu. It's architecture is tiny and elegant. Also it uses rabbitmq as transport so we won't need to introduce new protocol. Do we really need such complicated stuff? Sensu is huge project, and it's footprint is quite large. Monit can alert using scripts, can we use it instead of API? Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Rob Basham Cloud Systems Software Architecture 971-344-1999 Tomasz Napierala tnapier...@mirantis.com wrote on 11/24/2014 06:42:39 AM: From: Tomasz Napierala tnapier...@mirantis.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 11/24/2014 06:46 AM Subject: Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol. What scale are you proposing to support? I am proposing Sensu. It's architecture is tiny and elegant. Also it uses rabbitmq as transport so we won't need to introduce new protocol. We use Sensu on our smaller clouds and really like it there, but it doesn't scale sufficiently for our bigger clouds. Do we really need such complicated stuff? Sensu is huge project, and it's footprint is quite large. Monit can alert using scripts, can we use it instead of API? I assume you weren't talking about Sensu here and rather about Monasca. I like Monasca for monitoring at large scale. Kafka and Apache Storm are proven technologies at scale. Do you really think you can just pick one monitoring protocol that fits the needs of everybody? Frankly, I'm skeptical of that. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
And it all started out with simple free disk space monitoring :) I created a document https://etherpad.openstack.org/p/fuel-master-monitoring Let's write what exactly we want to monitor and what actions to take. Then it would be easier to decide which system we want. P. On 11/24/2014 04:32 PM, Rob Basham wrote: Rob Basham Cloud Systems Software Architecture 971-344-1999 Tomasz Napierala tnapier...@mirantis.com wrote on 11/24/2014 06:42:39 AM: From: Tomasz Napierala tnapier...@mirantis.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: 11/24/2014 06:46 AM Subject: Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol. What scale are you proposing to support? I am proposing Sensu. It's architecture is tiny and elegant. Also it uses rabbitmq as transport so we won't need to introduce new protocol. We use Sensu on our smaller clouds and really like it there, but it doesn't scale sufficiently for our bigger clouds. Do we really need such complicated stuff? Sensu is huge project, and it's footprint is quite large. Monit can alert using scripts, can we use it instead of API? I assume you weren't talking about Sensu here and rather about Monasca. I like Monasca for monitoring at large scale. Kafka and Apache Storm are proven technologies at scale. Do you really think you can just pick one monitoring protocol that fits the needs of everybody? Frankly, I'm skeptical of that. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
One of the selling points of tripleo is to reuse as much as possible from the cloud, to make it easier to deploy. While monasca may be more complicated, if it ends up being a component everyone learns, then its not as bad as needing to learn two different monitoring technologies. You could say the same thing cobbler vs ironic. the whole Ironic stack is much more complicated. But for an openstack admin, its easier since a lot of existing knowlege applies. Just something to consider. Thanks, Kevin From: Tomasz Napierala Sent: Monday, November 24, 2014 6:42:39 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Fuel] fuel master monitoring On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote: Hi, monasca looks overcomplicated for the purposes we need. Also it requires Kafka which is Java based transport protocol. I am proposing Sensu. It's architecture is tiny and elegant. Also it uses rabbitmq as transport so we won't need to introduce new protocol. Do we really need such complicated stuff? Sensu is huge project, and it's footprint is quite large. Monit can alert using scripts, can we use it instead of API? Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I have nothing against using some 3rd party service. But I thought this was to be small -- disk monitoring only notifying the user, not stats collecting. That's why I added the code to Fuel codebase. If you want external service you need to remember about such details as, say, duplicate settings (database credentials at least) and I thought this was an overkill for such simple functionality. Also, for a 3rd party service, notification injecting code still needs to be written as a plugin -- that's why I also don't think Ruby is a good idea :) So in the end I don't know if we'll have that much less code with a 3rd party service. But if you want a statistics collector then maybe it's OK. I found some Python services that might suit us: https://github.com/google/grr https://github.com/BrightcoveOS/Diamond P. On 11/20/2014 09:13 PM, Dmitriy Shulyak wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com mailto:tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com mailto:pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com mailto:tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
BTW, there's also Monit http://mmonit.com/monit/ (though it's in C) that looks quite nice. Some config examples: http://omgitsmgp.com/2013/09/07/a-monit-primer/ P. On 11/20/2014 09:13 PM, Dmitriy Shulyak wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com mailto:tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com mailto:pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com mailto:tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I have nothing against using some 3rd party service. But I thought this was to be small -- disk monitoring only notifying the user, not stats collecting. That's why I added the code to Fuel codebase. If you want external service you need to remember about such details as, say, duplicate settings (database credentials at least) and I thought this was an overkill for such simple functionality. Yes, it will be much more complex than simple daemon that creates notifications but our application is operating in isolated containers, and most of the resources cant be discovered from any particular container. So if we will want to extend it, with another task, like monitoring pool of dhcp addresses - we will end up with some kindof server-agent architecture, and this is a lot of work to do Also, for a 3rd party service, notification injecting code still needs to be written as a plugin -- that's why I also don't think Ruby is a good idea :) Afaik there is a way to write python plugins for sensu, but if there is monitoring app in python, that have friendly support for extensions, I am +1 for python So in the end I don't know if we'll have that much less code with a 3rd party service. But if you want a statistics collector then maybe it's OK. I think that monitoring application is fits there, and we kindof already introducing our whell for collecting statistic from openstack. I would like to know what guys who was working on stats in 6.0 thinking about it. So it is TBD ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I'm okay with Sensu or Monit, just as long as the results of monitoring can be represented in a web UI and has a configurable option for email alerting. Tight integration with Fuel Web is a nice-to-have (via AMQP perhaps), but anything that can solve our out-of-disk scenario is ideal. I did my best to tune our logging and logs rotation, but monitoring is the most sensible approach. -Matthew On Fri, Nov 21, 2014 at 12:21 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: BTW, there's also Monit http://mmonit.com/monit/ (though it's in C) that looks quite nice. Some config examples: http://omgitsmgp.com/2013/09/07/a-monit-primer/ P. On 11/20/2014 09:13 PM, Dmitriy Shulyak wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I heard about Monit a lot of good reviews, but unfortunately it looks like Monit doesn't support plugins and doesn't provide API. It may be a stumbling block if one day we decide to go deeper in monitoring task. On Fri, Nov 21, 2014 at 11:01 AM, Matthew Mosesohn mmoses...@mirantis.com wrote: I'm okay with Sensu or Monit, just as long as the results of monitoring can be represented in a web UI and has a configurable option for email alerting. Tight integration with Fuel Web is a nice-to-have (via AMQP perhaps), but anything that can solve our out-of-disk scenario is ideal. I did my best to tune our logging and logs rotation, but monitoring is the most sensible approach. -Matthew On Fri, Nov 21, 2014 at 12:21 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: BTW, there's also Monit http://mmonit.com/monit/ (though it's in C) that looks quite nice. Some config examples: http://omgitsmgp.com/2013/09/07/a-monit-primer/ P. On 11/20/2014 09:13 PM, Dmitriy Shulyak wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
There's also OpenStack's monasca-agent: https://github.com/stackforge/monasca-agent We could try to run it standalone (without Monasca API), add a plugin for it that checks disk and sends notification straight to Fuel DB and omits generating Forwarder requests. Or set up a fake API though both ways seem a bit hackish. I wasn't aware that we want email notifications too? P. On 11/21/2014 10:35 AM, Igor Kalnitsky wrote: I heard about Monit a lot of good reviews, but unfortunately it looks like Monit doesn't support plugins and doesn't provide API. It may be a stumbling block if one day we decide to go deeper in monitoring task. On Fri, Nov 21, 2014 at 11:01 AM, Matthew Mosesohn mmoses...@mirantis.com wrote: I'm okay with Sensu or Monit, just as long as the results of monitoring can be represented in a web UI and has a configurable option for email alerting. Tight integration with Fuel Web is a nice-to-have (via AMQP perhaps), but anything that can solve our out-of-disk scenario is ideal. I did my best to tune our logging and logs rotation, but monitoring is the most sensible approach. -Matthew On Fri, Nov 21, 2014 at 12:21 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: BTW, there's also Monit http://mmonit.com/monit/ (though it's in C) that looks quite nice. Some config examples: http://omgitsmgp.com/2013/09/07/a-monit-primer/ P. On 11/20/2014 09:13 PM, Dmitriy Shulyak wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
How about this? https://wiki.openstack.org/wiki/Monasca Kevin From: Dmitriy Shulyak Sent: Friday, November 21, 2014 12:57:45 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Fuel] fuel master monitoring I have nothing against using some 3rd party service. But I thought this was to be small -- disk monitoring only notifying the user, not stats collecting. That's why I added the code to Fuel codebase. If you want external service you need to remember about such details as, say, duplicate settings (database credentials at least) and I thought this was an overkill for such simple functionality. Yes, it will be much more complex than simple daemon that creates notifications but our application is operating in isolated containers, and most of the resources cant be discovered from any particular container. So if we will want to extend it, with another task, like monitoring pool of dhcp addresses - we will end up with some kindof server-agent architecture, and this is a lot of work to do Also, for a 3rd party service, notification injecting code still needs to be written as a plugin -- that's why I also don't think Ruby is a good idea :) Afaik there is a way to write python plugins for sensu, but if there is monitoring app in python, that have friendly support for extensions, I am +1 for python So in the end I don't know if we'll have that much less code with a 3rd party service. But if you want a statistics collector then maybe it's OK. I think that monitoring application is fits there, and we kindof already introducing our whell for collecting statistic from openstack. I would like to know what guys who was working on stats in 6.0 thinking about it. So it is TBD ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
+1 for Dima's suggestion. We need to stop reinventing the wheel. There are a lot tools around us, so let's pick one of them and will use it. On Thu, Nov 20, 2014 at 10:13 PM, Dmitriy Shulyak dshul...@mirantis.com wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
+1 also. There are so many monitoring tools, but maybe not something written in ruby ;) On Thu, Nov 20, 2014 at 10:40 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: +1 for Dima's suggestion. We need to stop reinventing the wheel. There are a lot tools around us, so let's pick one of them and will use it. On Thu, Nov 20, 2014 at 10:13 PM, Dmitriy Shulyak dshul...@mirantis.com wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Łukasz Oleś ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I would rather compare features than ruby vs python. Best Regards, Sergii Golovatiuk On 20 Nov 2014, at 23:20, Lukasz Oles lo...@mirantis.com wrote: +1 also. There are so many monitoring tools, but maybe not something written in ruby ;) On Thu, Nov 20, 2014 at 10:40 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: +1 for Dima's suggestion. We need to stop reinventing the wheel. There are a lot tools around us, so let's pick one of them and will use it. On Thu, Nov 20, 2014 at 10:13 PM, Dmitriy Shulyak dshul...@mirantis.com wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Łukasz Oleś ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
On Thu, Nov 20, 2014 at 11:28 PM, Sergii Golovatiuk sgolovat...@mirantis.com wrote: I would rather compare features than ruby vs python. It make sense only as long as you don't need to debug it Best Regards, Sergii Golovatiuk On 20 Nov 2014, at 23:20, Lukasz Oles lo...@mirantis.com wrote: +1 also. There are so many monitoring tools, but maybe not something written in ruby ;) On Thu, Nov 20, 2014 at 10:40 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: +1 for Dima's suggestion. We need to stop reinventing the wheel. There are a lot tools around us, so let's pick one of them and will use it. On Thu, Nov 20, 2014 at 10:13 PM, Dmitriy Shulyak dshul...@mirantis.com wrote: Guys, maybe we can use existing software, for example Sensu [1]? Maybe i am wrong, but i dont like the idea to start writing our small monitoring applications.. Also something well designed and extendable can be reused for statistic collector 1. https://github.com/sensu On Wed, Nov 12, 2014 at 12:47 PM, Tomasz Napierala tnapier...@mirantis.com wrote: On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Łukasz Oleś ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Łukasz Oleś ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. I’m all for that, but if we add it, we need to document ways to clean up space. We could also add some kind of simple job to remove rotated logs, obsolete spanshots, etc., but this is out of scope for 6.0 I guess. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
We can add a notification to FuelWeb, no additional software or user actions are required. I would not overestimate this method though, it is in no way the robust monitoring system. Forcing user to do something on a regular basis is unlikely to work. Anton On Thu, Nov 6, 2014 at 11:55 AM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: I think we're missing the point here. What I meant adding a simple monitoring system that informed the user via UI/CLI/email/whatever of low resources on fuel master node. That's it. HA here is not an option -- if, despite of warnings, the user still continues to use fuel and disk becomes full, it's the user's fault. By adding these warnings we have a way of saying We told you so! Without warnings we get bugs like [1] I mentioned in the first post. Of course user can check disk space by hand but since we do have a full-blown UI telling the user to periodically log in to the console and check disks by hand seems a bit of a burden. We can even implement such monitoring functionality as a Nailgun plugin -- installing it would be optional and at the same time we would grow our plugin ecosystem. P. On 11/05/2014 08:42 PM, Dmitry Borodaenko wrote: Even one additional hardware node required to host the Fuel master is seen by many users as excessive. Unless you can come up with an architecture that adds HA capability to Fuel without increasing its hardware footprint by 2 more nodes, it's just not worth it. The only operational aspect of the Fuel master node that you don't want to lose even for a short while is logging. You'd be better off redirecting OpenStack environments' logs to a dedicated highly available logging server (which, of course, you already have in your environment), and deal with Fuel master node failures by restoring it from backups. On Wed, Nov 5, 2014 at 8:26 AM, Anton Zemlyanov azemlya...@mirantis.com wrote: Monitoring of the Fuel master's disk space is the special case. I really wonder why Fuel master have no HA option, disk overflow can be predicted but many other failures cannot. HA is a solution of the 'single point of failure' problem. The current monitoring recommendations ( http://docs.openstack.org/openstack-ops/content/logging_monitoring.html) are based on analyzing logs and manual checks, that are rather reactive way of fixing problems. Zabbix is quite good for preventing failures that are predictable but for the abrupt problems Zabbix just reports them 'post mortem'. The only way to remove the single failure point is to implement redundancy/HA Anton On Tue, Nov 4, 2014 at 6:26 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: Hello, In extension to my comment in this bug [1] I'd like to discuss the possibility of adding Fuel master node monitoring. As I wrote in the comment, when disk is full it might be already too late to perform any action since for example Nailgun could be down because DB shut itself down. So we should somehow warn the user that disk is running low (in the UI and fuel CLI on stderr for example) before it actually happens. For now the only meaningful value to monitor would be disk usage -- do you have other suggestions? If not then probably a simple API endpoint with statvfs calls would suffice. If you see other usages of this then maybe it would be better to have some daemon collecting the stats we want. If we opted for a daemon, then I'm aware that the user can optionally install Zabbix server although looking at blueprints in [2] I don't see anything about monitoring Fuel master itself -- is it possible to do? Though the installation of Zabbix though is not mandatory so it still doesn't completely solve the problem. [1] https://bugs.launchpad.net/fuel/+bug/1371757 [2] https://blueprints.launchpad.net/fuel/+spec/monitoring-system Przemek ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Dmitry Borodaenko ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I didn't mean a robust monitoring system, just something simpler. Notifications is a good idea for FuelWeb. P. On 11/06/2014 09:59 AM, Anton Zemlyanov wrote: We can add a notification to FuelWeb, no additional software or user actions are required. I would not overestimate this method though, it is in no way the robust monitoring system. Forcing user to do something on a regular basis is unlikely to work. Anton On Thu, Nov 6, 2014 at 11:55 AM, Przemyslaw Kaminski pkamin...@mirantis.com mailto:pkamin...@mirantis.com wrote: I think we're missing the point here. What I meant adding a simple monitoring system that informed the user via UI/CLI/email/whatever of low resources on fuel master node. That's it. HA here is not an option -- if, despite of warnings, the user still continues to use fuel and disk becomes full, it's the user's fault. By adding these warnings we have a way of saying We told you so! Without warnings we get bugs like [1] I mentioned in the first post. Of course user can check disk space by hand but since we do have a full-blown UI telling the user to periodically log in to the console and check disks by hand seems a bit of a burden. We can even implement such monitoring functionality as a Nailgun plugin -- installing it would be optional and at the same time we would grow our plugin ecosystem. P. On 11/05/2014 08:42 PM, Dmitry Borodaenko wrote: Even one additional hardware node required to host the Fuel master is seen by many users as excessive. Unless you can come up with an architecture that adds HA capability to Fuel without increasing its hardware footprint by 2 more nodes, it's just not worth it. The only operational aspect of the Fuel master node that you don't want to lose even for a short while is logging. You'd be better off redirecting OpenStack environments' logs to a dedicated highly available logging server (which, of course, you already have in your environment), and deal with Fuel master node failures by restoring it from backups. On Wed, Nov 5, 2014 at 8:26 AM, Anton Zemlyanov azemlya...@mirantis.com mailto:azemlya...@mirantis.com wrote: Monitoring of the Fuel master's disk space is the special case. I really wonder why Fuel master have no HA option, disk overflow can be predicted but many other failures cannot. HA is a solution of the 'single point of failure' problem. The current monitoring recommendations (http://docs.openstack.org/openstack-ops/content/logging_monitoring.html) are based on analyzing logs and manual checks, that are rather reactive way of fixing problems. Zabbix is quite good for preventing failures that are predictable but for the abrupt problems Zabbix just reports them 'post mortem'. The only way to remove the single failure point is to implement redundancy/HA Anton On Tue, Nov 4, 2014 at 6:26 PM, Przemyslaw Kaminski pkamin...@mirantis.com mailto:pkamin...@mirantis.com wrote: Hello, In extension to my comment in this bug [1] I'd like to discuss the possibility of adding Fuel master node monitoring. As I wrote in the comment, when disk is full it might be already too late to perform any action since for example Nailgun could be down because DB shut itself down. So we should somehow warn the user that disk is running low (in the UI and fuel CLI on stderr for example) before it actually happens. For now the only meaningful value to monitor would be disk usage -- do you have other suggestions? If not then probably a simple API endpoint with statvfs calls would suffice. If you see other usages of this then maybe it would be better to have some daemon collecting the stats we want. If we opted for a daemon, then I'm aware that the user can optionally install Zabbix server although looking at blueprints in [2] I don't see anything about monitoring Fuel master itself -- is it possible to do? Though the installation of Zabbix though is not mandatory so it still doesn't completely solve the problem. [1] https://bugs.launchpad.net/fuel/+bug/1371757 [2] https://blueprints.launchpad.net/fuel/+spec/monitoring-system Przemek ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list
Re: [openstack-dev] [Fuel] fuel master monitoring
Monitoring of the Fuel master's disk space is the special case. I really wonder why Fuel master have no HA option, disk overflow can be predicted but many other failures cannot. HA is a solution of the 'single point of failure' problem. The current monitoring recommendations ( http://docs.openstack.org/openstack-ops/content/logging_monitoring.html) are based on analyzing logs and manual checks, that are rather reactive way of fixing problems. Zabbix is quite good for preventing failures that are predictable but for the abrupt problems Zabbix just reports them 'post mortem'. The only way to remove the single failure point is to implement redundancy/HA Anton On Tue, Nov 4, 2014 at 6:26 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: Hello, In extension to my comment in this bug [1] I'd like to discuss the possibility of adding Fuel master node monitoring. As I wrote in the comment, when disk is full it might be already too late to perform any action since for example Nailgun could be down because DB shut itself down. So we should somehow warn the user that disk is running low (in the UI and fuel CLI on stderr for example) before it actually happens. For now the only meaningful value to monitor would be disk usage -- do you have other suggestions? If not then probably a simple API endpoint with statvfs calls would suffice. If you see other usages of this then maybe it would be better to have some daemon collecting the stats we want. If we opted for a daemon, then I'm aware that the user can optionally install Zabbix server although looking at blueprints in [2] I don't see anything about monitoring Fuel master itself -- is it possible to do? Though the installation of Zabbix though is not mandatory so it still doesn't completely solve the problem. [1] https://bugs.launchpad.net/fuel/+bug/1371757 [2] https://blueprints.launchpad.net/fuel/+spec/monitoring-system Przemek ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Hello, As far as I can tell, disk space monitoring is pretty useless, unless Fuel provides user with some means to automatically cleanup of stored data (i.e. remove obsolete diagnostic snapshots, etc). Otherwise, it will be only useful for experienced Fuel developers who know how to properly cleanup the Master node. -- Best regards, Oleg Gelbukh On Tue, Nov 4, 2014 at 3:26 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: Hello, In extension to my comment in this bug [1] I'd like to discuss the possibility of adding Fuel master node monitoring. As I wrote in the comment, when disk is full it might be already too late to perform any action since for example Nailgun could be down because DB shut itself down. So we should somehow warn the user that disk is running low (in the UI and fuel CLI on stderr for example) before it actually happens. For now the only meaningful value to monitor would be disk usage -- do you have other suggestions? If not then probably a simple API endpoint with statvfs calls would suffice. If you see other usages of this then maybe it would be better to have some daemon collecting the stats we want. If we opted for a daemon, then I'm aware that the user can optionally install Zabbix server although looking at blueprints in [2] I don't see anything about monitoring Fuel master itself -- is it possible to do? Though the installation of Zabbix though is not mandatory so it still doesn't completely solve the problem. [1] https://bugs.launchpad.net/fuel/+bug/1371757 [2] https://blueprints.launchpad.net/fuel/+spec/monitoring-system Przemek ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
Even one additional hardware node required to host the Fuel master is seen by many users as excessive. Unless you can come up with an architecture that adds HA capability to Fuel without increasing its hardware footprint by 2 more nodes, it's just not worth it. The only operational aspect of the Fuel master node that you don't want to lose even for a short while is logging. You'd be better off redirecting OpenStack environments' logs to a dedicated highly available logging server (which, of course, you already have in your environment), and deal with Fuel master node failures by restoring it from backups. On Wed, Nov 5, 2014 at 8:26 AM, Anton Zemlyanov azemlya...@mirantis.com wrote: Monitoring of the Fuel master's disk space is the special case. I really wonder why Fuel master have no HA option, disk overflow can be predicted but many other failures cannot. HA is a solution of the 'single point of failure' problem. The current monitoring recommendations ( http://docs.openstack.org/openstack-ops/content/logging_monitoring.html) are based on analyzing logs and manual checks, that are rather reactive way of fixing problems. Zabbix is quite good for preventing failures that are predictable but for the abrupt problems Zabbix just reports them 'post mortem'. The only way to remove the single failure point is to implement redundancy/HA Anton On Tue, Nov 4, 2014 at 6:26 PM, Przemyslaw Kaminski pkamin...@mirantis.com wrote: Hello, In extension to my comment in this bug [1] I'd like to discuss the possibility of adding Fuel master node monitoring. As I wrote in the comment, when disk is full it might be already too late to perform any action since for example Nailgun could be down because DB shut itself down. So we should somehow warn the user that disk is running low (in the UI and fuel CLI on stderr for example) before it actually happens. For now the only meaningful value to monitor would be disk usage -- do you have other suggestions? If not then probably a simple API endpoint with statvfs calls would suffice. If you see other usages of this then maybe it would be better to have some daemon collecting the stats we want. If we opted for a daemon, then I'm aware that the user can optionally install Zabbix server although looking at blueprints in [2] I don't see anything about monitoring Fuel master itself -- is it possible to do? Though the installation of Zabbix though is not mandatory so it still doesn't completely solve the problem. [1] https://bugs.launchpad.net/fuel/+bug/1371757 [2] https://blueprints.launchpad.net/fuel/+spec/monitoring-system Przemek ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Dmitry Borodaenko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] fuel master monitoring
I think we're missing the point here. What I meant adding a simple monitoring system that informed the user via UI/CLI/email/whatever of low resources on fuel master node. That's it. HA here is not an option -- if, despite of warnings, the user still continues to use fuel and disk becomes full, it's the user's fault. By adding these warnings we have a way of saying We told you so! Without warnings we get bugs like [1] I mentioned in the first post. Of course user can check disk space by hand but since we do have a full-blown UI telling the user to periodically log in to the console and check disks by hand seems a bit of a burden. We can even implement such monitoring functionality as a Nailgun plugin -- installing it would be optional and at the same time we would grow our plugin ecosystem. P. On 11/05/2014 08:42 PM, Dmitry Borodaenko wrote: Even one additional hardware node required to host the Fuel master is seen by many users as excessive. Unless you can come up with an architecture that adds HA capability to Fuel without increasing its hardware footprint by 2 more nodes, it's just not worth it. The only operational aspect of the Fuel master node that you don't want to lose even for a short while is logging. You'd be better off redirecting OpenStack environments' logs to a dedicated highly available logging server (which, of course, you already have in your environment), and deal with Fuel master node failures by restoring it from backups. On Wed, Nov 5, 2014 at 8:26 AM, Anton Zemlyanov azemlya...@mirantis.com mailto:azemlya...@mirantis.com wrote: Monitoring of the Fuel master's disk space is the special case. I really wonder why Fuel master have no HA option, disk overflow can be predicted but many other failures cannot. HA is a solution of the 'single point of failure' problem. The current monitoring recommendations (http://docs.openstack.org/openstack-ops/content/logging_monitoring.html) are based on analyzing logs and manual checks, that are rather reactive way of fixing problems. Zabbix is quite good for preventing failures that are predictable but for the abrupt problems Zabbix just reports them 'post mortem'. The only way to remove the single failure point is to implement redundancy/HA Anton On Tue, Nov 4, 2014 at 6:26 PM, Przemyslaw Kaminski pkamin...@mirantis.com mailto:pkamin...@mirantis.com wrote: Hello, In extension to my comment in this bug [1] I'd like to discuss the possibility of adding Fuel master node monitoring. As I wrote in the comment, when disk is full it might be already too late to perform any action since for example Nailgun could be down because DB shut itself down. So we should somehow warn the user that disk is running low (in the UI and fuel CLI on stderr for example) before it actually happens. For now the only meaningful value to monitor would be disk usage -- do you have other suggestions? If not then probably a simple API endpoint with statvfs calls would suffice. If you see other usages of this then maybe it would be better to have some daemon collecting the stats we want. If we opted for a daemon, then I'm aware that the user can optionally install Zabbix server although looking at blueprints in [2] I don't see anything about monitoring Fuel master itself -- is it possible to do? Though the installation of Zabbix though is not mandatory so it still doesn't completely solve the problem. [1] https://bugs.launchpad.net/fuel/+bug/1371757 [2] https://blueprints.launchpad.net/fuel/+spec/monitoring-system Przemek ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Dmitry Borodaenko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
