The proposed monitoring options make sense, since the Fuel master has nothing
yet, but I would like to ressurect this thread to see if we can discuss some
strategies in order to avoid the /var/log fillup with consequent docker
containers corruption.
Now, a customer facing this corruption can
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hello,
The updated version of monitoring code is available here:
https://review.openstack.org/#/c/137785/
This is based on monit as was agreed in this thread. The drawback of
monit is that basically it's a very simple system that doesn't track
On Wed, Jan 7, 2015 at 12:59 AM, Przemyslaw Kaminski
pkamin...@mirantis.com wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Hello,
The updated version of monitoring code is available here:
https://review.openstack.org/#/c/137785/
This is based on monit as was agreed in this
There are two threads here that need to be unraveled from each other.
1. We need to prevent fuel from doing anything if the OS is out of
disk space. It leads to a very broken database from which it requires
a developer to reset to a usable state.
From this point we need to
* develop a method for
Is it possible to send http requests from monit, e.g for creating
notifications?
I scanned through the docs and found only alerts for sending mail,
also where token (username/pass) for monit will be stored?
Or maybe there is another plan? without any api interaction
On Thu, Nov 27, 2014 at 9:39
I mean with monit you can execute arbitrary scripts so use curl? Or save
them directly to DB?
http://omgitsmgp.com/2013/09/07/a-monit-primer/
I guess some data has to be stored in a configuration file (either DB
credentials or Nailgun API URL at least, if we were to create
notifications via
Hi,
Dmitriy, first of all, monit can provide HTTP interface for communication -
so it's possible to poll that this interface to get info or even control
monit (stop/start/restart service, stop/start monitoring of a service,
etc). Secondly, you can configure different triggers in monit and set
I've added another option to the Etherpad: collectd can do basic threshold
monitoring and run any kind of scripts on alert notifications. The other
advantage of collectd would be the RRD graphs for (almost) free.
Of course since monit is already supported in Fuel, this is the fastest
path to get
:* Monday, November 24, 2014 6:42:39 AM
*To:* OpenStack Development Mailing List (not for usage questions)
*Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring
On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com
wrote:
Hi,
monasca looks overcomplicated
(not for usage questions)
*Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring
On 24 Nov 2014, at 11:09, Sergii Golovatiuk
sgolovat...@mirantis.com mailto:sgolovat...@mirantis.com wrote:
Hi,
monasca looks overcomplicated for the purposes we need. Also
From: Przemyslaw Kaminski
Sent: Wednesday, November 26, 2014 2:50:03 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Fuel] fuel master monitoring
I agree, this was supposed to be small.
P.
On 11/26/2014 11:03 AM
Folks,
Maybe I understand some things wrong, but Zabbix is a different story.
We deploy Zabbix per cluster, so it doesn't monitor for *all* slaves
or master node. It monitors only one cluster.
Therefore I see no reasons to choose Zabbix over monit. I mean, it
shouldn't be We MUST use Zabbix
We want to monitor Fuel master node while Zabbix is only on slave nodes
and not on master. The monitoring service is supposed to be installed on
Fuel master host (not inside a Docker container) and provide basic info
about free disk space, etc.
P.
On 11/26/2014 02:58 PM, Jay Pipes wrote:
On
On 11/26/2014 10:22 AM, Przemyslaw Kaminski wrote:
We want to monitor Fuel master node while Zabbix is only on slave nodes
and not on master. The monitoring service is supposed to be installed on
Fuel master host (not inside a Docker container) and provide basic info
about free disk space, etc.
Hi,
I would do both to compare. monit and Sensu have own advantages.
--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser
On Wed, Nov 26, 2014 at 4:22 PM, Przemyslaw Kaminski pkamin...@mirantis.com
wrote:
We want to monitor Fuel master node while Zabbix is only on slave nodes
and
As for me - zabbix is overkill for one node. Zabbix Server + Agent +
Frontend + DB + HTTP server, and all of it for one node? Why not use
something that was developed for monitoring one node, doesn't have many
deps and work out of the box? Not necessarily Monit, but something similar.
On Wed, Nov
Jay,
Fuel uses watchdog service for container to restart it in case of issues.
We have the same problem with containers when disk is full
--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser
On Wed, Nov 26, 2014 at 4:39 PM, Jay Pipes jaypi...@gmail.com wrote:
On 11/26/2014 10:22
Monit is easy and is used to control states of Compute nodes. We can adopt
it for master node.
--
Best regards,
Sergii Golovatiuk,
Skype #golserge
IRC #holser
On Wed, Nov 26, 2014 at 4:46 PM, Stanislaw Bogatkin sbogat...@mirantis.com
wrote:
As for me - zabbix is overkill for one node. Zabbix
On 11/26/2014 11:54 AM, Sergii Golovatiuk wrote:
Jay,
Fuel uses watchdog service for container to restart it in case of
issues. We have the same problem with containers when disk is full
I see. I guess I don't quite understand why Zabbix isn't just used for
everything -- after all, the
This I didn't know. It's true in fact, I checked the manifests. Though
monit is not deployed yet because of lack of packages in Fuel ISO.
Anyways, I think the argument about using yet another monitoring service
is now rendered invalid.
So +1 for monit? :)
P.
On 11/26/2014 05:55 PM, Sergii
, November 21, 2014 12:57:45 AM
*To:* OpenStack Development Mailing List (not for usage questions)
*Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring
I have nothing against using some 3rd party service. But I thought
this was to be small -- disk monitoring only notifying the user
Kevin
--
*From:* Dmitriy Shulyak
*Sent:* Friday, November 21, 2014 12:57:45 AM
*To:* OpenStack Development Mailing List (not for usage questions)
*Subject:* Re: [openstack-dev] [Fuel] fuel master monitoring
I have nothing against using some 3rd party service. But I
On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote:
Hi,
monasca looks overcomplicated for the purposes we need. Also it requires
Kafka which is Java based transport protocol.
I am proposing Sensu. It's architecture is tiny and elegant. Also it uses
rabbitmq as
/24/2014 06:46 AM
Subject: Re: [openstack-dev] [Fuel] fuel master monitoring
On 24 Nov 2014, at 11:09, Sergii Golovatiuk
sgolovat...@mirantis.com wrote:
Hi,
monasca looks overcomplicated for the purposes we need. Also it
requires Kafka which is Java based transport protocol
@lists.openstack.org
Date: 11/24/2014 06:46 AM
Subject: Re: [openstack-dev] [Fuel] fuel master monitoring
On 24 Nov 2014, at 11:09, Sergii Golovatiuk
sgolovat...@mirantis.com wrote:
Hi,
monasca looks overcomplicated for the purposes we need. Also it
requires Kafka which is Java based
:42:39 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Fuel] fuel master monitoring
On 24 Nov 2014, at 11:09, Sergii Golovatiuk sgolovat...@mirantis.com wrote:
Hi,
monasca looks overcomplicated for the purposes we need. Also it requires
I have nothing against using some 3rd party service. But I thought this
was to be small -- disk monitoring only notifying the user, not stats
collecting. That's why I added the code to Fuel codebase. If you want
external service you need to remember about such details as, say,
duplicate
BTW, there's also Monit
http://mmonit.com/monit/
(though it's in C) that looks quite nice. Some config examples:
http://omgitsmgp.com/2013/09/07/a-monit-primer/
P.
On 11/20/2014 09:13 PM, Dmitriy Shulyak wrote:
Guys, maybe we can use existing software, for example Sensu [1]?
Maybe i am
I have nothing against using some 3rd party service. But I thought this
was to be small -- disk monitoring only notifying the user, not stats
collecting. That's why I added the code to Fuel codebase. If you want
external service you need to remember about such details as, say, duplicate
I'm okay with Sensu or Monit, just as long as the results of
monitoring can be represented in a web UI and has a configurable
option for email alerting. Tight integration with Fuel Web is a
nice-to-have (via AMQP perhaps), but anything that can solve our
out-of-disk scenario is ideal. I did my
I heard about Monit a lot of good reviews, but unfortunately it looks
like Monit doesn't support plugins and doesn't provide API. It may be
a stumbling block if one day we decide to go deeper in monitoring
task.
On Fri, Nov 21, 2014 at 11:01 AM, Matthew Mosesohn
mmoses...@mirantis.com wrote:
I'm
There's also OpenStack's monasca-agent:
https://github.com/stackforge/monasca-agent
We could try to run it standalone (without Monasca API), add a plugin
for it that checks disk and sends notification straight to Fuel DB and
omits generating Forwarder requests. Or set up a fake API though
How about this?
https://wiki.openstack.org/wiki/Monasca
Kevin
From: Dmitriy Shulyak
Sent: Friday, November 21, 2014 12:57:45 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Fuel] fuel master monitoring
I have
Guys, maybe we can use existing software, for example Sensu [1]?
Maybe i am wrong, but i dont like the idea to start writing our small
monitoring applications..
Also something well designed and extendable can be reused for statistic
collector
1. https://github.com/sensu
On Wed, Nov 12, 2014 at
+1 for Dima's suggestion. We need to stop reinventing the wheel. There
are a lot tools around us, so let's pick one of them and will use it.
On Thu, Nov 20, 2014 at 10:13 PM, Dmitriy Shulyak dshul...@mirantis.com wrote:
Guys, maybe we can use existing software, for example Sensu [1]?
Maybe i am
+1 also. There are so many monitoring tools, but maybe not something
written in ruby ;)
On Thu, Nov 20, 2014 at 10:40 PM, Igor Kalnitsky
ikalnit...@mirantis.com wrote:
+1 for Dima's suggestion. We need to stop reinventing the wheel. There
are a lot tools around us, so let's pick one of them and
I would rather compare features than ruby vs python.
Best Regards,
Sergii Golovatiuk
On 20 Nov 2014, at 23:20, Lukasz Oles lo...@mirantis.com wrote:
+1 also. There are so many monitoring tools, but maybe not something
written in ruby ;)
On Thu, Nov 20, 2014 at 10:40 PM, Igor Kalnitsky
On Thu, Nov 20, 2014 at 11:28 PM, Sergii Golovatiuk
sgolovat...@mirantis.com wrote:
I would rather compare features than ruby vs python.
It make sense only as long as you don't need to debug it
Best Regards,
Sergii Golovatiuk
On 20 Nov 2014, at 23:20, Lukasz Oles lo...@mirantis.com
On 06 Nov 2014, at 12:20, Przemyslaw Kaminski pkamin...@mirantis.com wrote:
I didn't mean a robust monitoring system, just something simpler.
Notifications is a good idea for FuelWeb.
I’m all for that, but if we add it, we need to document ways to clean up space.
We could also add some kind
We can add a notification to FuelWeb, no additional software or user
actions are required. I would not overestimate this method though, it is in
no way the robust monitoring system. Forcing user to do something on a
regular basis is unlikely to work.
Anton
On Thu, Nov 6, 2014 at 11:55 AM,
I didn't mean a robust monitoring system, just something simpler.
Notifications is a good idea for FuelWeb.
P.
On 11/06/2014 09:59 AM, Anton Zemlyanov wrote:
We can add a notification to FuelWeb, no additional software or user
actions are required. I would not overestimate this method though,
Monitoring of the Fuel master's disk space is the special case. I really
wonder why Fuel master have no HA option, disk overflow can be predicted
but many other failures cannot. HA is a solution of the 'single point of
failure' problem.
The current monitoring recommendations (
Hello,
As far as I can tell, disk space monitoring is pretty useless, unless Fuel
provides user with some means to automatically cleanup of stored data (i.e.
remove obsolete diagnostic snapshots, etc). Otherwise, it will be only
useful for experienced Fuel developers who know how to properly
Even one additional hardware node required to host the Fuel master is seen
by many users as excessive. Unless you can come up with an architecture
that adds HA capability to Fuel without increasing its hardware footprint
by 2 more nodes, it's just not worth it.
The only operational aspect of the
I think we're missing the point here. What I meant adding a simple
monitoring system that informed the user via UI/CLI/email/whatever of
low resources on fuel master node. That's it. HA here is not an option
-- if, despite of warnings, the user still continues to use fuel and
disk becomes
Hello,
In extension to my comment in this bug [1] I'd like to discuss the
possibility of adding Fuel master node monitoring. As I wrote in the
comment, when disk is full it might be already too late to perform any
action since for example Nailgun could be down because DB shut itself
down. So
46 matches
Mail list logo
