On 06/24/2018 11:04 PM, Michael Andescavage wrote:
What kind of data will ultimately be gathered? Are we just considering
performance counters, or custom monitors that can aggregate data of varying
types so that our users can create their own alerting? (i'm thinking
splunk-esque custom log digestion) What platform(s) are being targeted for
monitoring - sorry if this is documented already elsewhere. What rights does
the agent need to access the data? (i'm not a *unix guru, but the same
principals apply - least privilege is best)
My thinking right now is that this belongs in one or more specific
emails surrounding the nodes/agents and the server, not so much the
general idea in this thread, so I won't go into complete detail here.
Suffice it to say, I personally envision both generic (standardized)
data and custom data in various ways.
For the data that is being collected, how is the data protected in transit as
well as at rest?
The initial thought (I'll touch on that very soon in a new thread) is
that we rely on both encryption (HTTPS) and signing of results with an
async keypair. The same keypair would be used in reverse, when sending
test parameters to a node/agent (this time encrypted instead of just
signed). We could add encryption of the data once stored, I think sqlite
et al have encryption options, but it's not a thing I personally want to
invest too much time in before we need to.
I don't know how customize-able we want to be, but is user management being consider?
Can the admin define who can see what data, or define "Team X can see their services
and define their own counters/metrics, but not other teams." or is the thinking more
along the lines of there's one central team who manages all of the monitoring and they
are the POC for defining alerts?
It's definitely something I've been pondering, and something PMB did
very well (IMHO). For starters, I think we should just get a basic
system going with normal users, super users, and robits (the
nodes/agents have pseudo-accounts via their API key). Once we have that,
we can expand it to teams and roles. I would love to generate a final
design that would allow people to offer commercial installations of
Warble in a multi-tenant sense. I'll put a pin in this discussion for later.
With regards,
Daniel.
Much like Chris, many thoughts in my head and rewrote this a few times.
________________________________
From: Chris Thistlethwaite <chr...@apache.org>
Sent: Sunday, June 24, 2018 5:11 PM
To: dev@warble.apache.org
Subject: Re: The general idea, design and goal of Apache Warble
I completely agree, easy setup is key. This is starting to sound like
Etherpad, out of the box you get DirtyDB and then can pipe it to
something bigger/better if you want too.
This email has been re-written a few times as I type out ideas, flush
them out a bit, then realize they don't make any sense.
-Chris T.
On Sun, 2018-06-24 at 09:53 -0500, Daniel Gruno wrote:
On 06/23/2018 11:59 AM, Chris Lambertus wrote:
This is a good summary. As we move forward with the design specs,
I’d also like to see some comparisons with existing off-the-shelf
monitoring systems (zabbix, nagios, etc.,) to show that Warble is
first and foremost easy to set up. This is typically the biggest
barrier to entry for folks trying to set up a monitoring solution.
Zabbix and Nagios specifically require a fairly steep learning
curve to get any useful data.
the potential turnkey aspect of Warble is very much on my mind. I
want
it to be as easy as downloading, running a setup script, and things
should be operational. While we can't "compete" against other
software
products (this is very anathema to ASF projects), we can highlight
what
we think are Warble's strengths and mebbe do a matrix comparison of
features, pros and cons etc.
As for the database, I'm leaning towards using ElasticSearch for
the
permanent storage, and possibly Redis for the ephemeral lookup
cache for
the alerter.
If it can be made turn-key, I’m OK with ES, but it’s a heavy
solution for small scale monitoring. It may be worth considering a
tiered approach with something as simple as sqllite as the
“default” backend with ES for a more enterprise-sized deployment.
I think this is a reasonable approach. For the initial development,
we
can definitely start with something simple like sqlite, and as
development continues, we can add in the more advanced stuff that
would
require a proper timeseries/aggregated database system once we get
closer to scoping out the visualization aspects of gathered data
(that
is, beyond the basic alerting and up/down stats).
Any thoughts on being backwards-compatible with existing nagios
check scripts? There’s a fairly broad ecosystem there that might be
leveraged.
This is a tricky question, and my immediate reaction is that this is
not
something _I_ would focus on, as building an additional API endpoint
for
Nagios could take quite a while. Instead, I'll be working on building
a
simple and 'modular' base class for tests as possible, so anyone can
quickly build a test for whatever they want to test for, or quickly
port
from other systems to Warble. This is not to say that the Nagios idea
is
bad, but rather that I would prefer starting out with what I know.
To give an idea of what a test class currently looks like, I have an
example class at
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpaste.apache.org%2FJi05&data=02%7C01%7C%7C931b1694761f45d8705808d5da304301%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636654823166178998&sdata=LfuxkZHEiUOmeD1XP3IPMiUUrB7Via1h42kDd34CTvw%3D&reserved=0
- all tests are
shaped
the same way, and have an init and a run function, which relies on a
bunch of common libraries to quickly do stuff. You pass test
parameters
to the test class, and it spits out a generic report object. We
prooobably can make wrappers for nagios and other systems in the
future.
-Chris
-----------------------------------------------------------------
----
To unsubscribe, e-mail: dev-unsubscr...@warble.apache.org
For additional commands, e-mail: dev-h...@warble.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@warble.apache.org
For additional commands, e-mail: dev-h...@warble.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@warble.apache.org
For additional commands, e-mail: dev-h...@warble.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@warble.apache.org
For additional commands, e-mail: dev-h...@warble.apache.org
