Hi Charlie,
if you take a look on the github, you will see the elasticsearch and
kibana feature, directly available in Karaf (no need of additional stuff).
Regards
JB
On 10/15/2014 09:41 AM, Charlie Mordant wrote:
Hi J.B.,
I saw an HawtIO plugin for Kibana (but didn't tested it), so it could be
possible that you'll have nothing to do for you HawtIO integration :p.
As an addition, a Decanter Karaf feature could be made to ease the user
interaction with ElasticSearch (installing either Jelastic or
Spring-data-elasticsearch).
May be a Docker container can be implemented to ease
Elastic+Kibana+logstash installation (and may be karaf command for
installing all: the container in an Unix env, boot2docker + the
container in a MacOSX one, Vagrant + CoreOS + container in a windows
one....).
Nice name however: Decanter sounds well with a Karaf and an old Pomard
:), it also explains well the goal of the product.
It's also nice idea, I thought to do the same for my distro and it's far
better if it's an Apache/Karaf product!
Good luck for the job, I'll try an Elastic Karaf feature on my side
(then give it to you).
Best regards,
2014-10-15 5:17 GMT+02:00 Andreas Pieber <[email protected]
<mailto:[email protected]>>:
Hey,
The collection definitely sounds like a perfect idea for a Karaf sub
project to me. Beside the great potential for the components I like
the especially fitting name 😊 +1
Kind regards,
Andreas
On Oct 14, 2014 5:13 PM, "Jean-Baptiste Onofré" <[email protected]
<mailto:[email protected]>> wrote:
Hi all,
First of all, sorry for this long e-mail ;)
Some weeks ago, I blogged about the usage of ELK
(Logstash/Elasticsearch/__Kibana) with Karaf, Camel, ActiveMQ,
etc to provide a monitoring dashboard (know what's happen in
Karaf and be able to store it for a long period):
http://blog.nanthrax.net/2014/__03/apache-karaf-cellar-camel-__activemq-monitoring-with-elk-__elasticsearch-logstash-and-__kibana/
<http://blog.nanthrax.net/2014/03/apache-karaf-cellar-camel-activemq-monitoring-with-elk-elasticsearch-logstash-and-kibana/>
If this solution works fine, there are some drawbacks:
- it requires additional middlewares on the machines.
Additionally to Karaf itself, we have to install logstash,
elasticsearch nodes, and kibana console
- it's not usable "out of the box": you need at least to
configure logstash (with the different input/output plugins),
kibana (to create the dashboard that you need)
- it doesn't cover all the monitoring needs, especially in term
of SLA: we want to be able to raise some alerts depending of
some events (for instance, when a regex is match in the log
messages, when a feature is uninstalled, when a JMX metric is
greater than a given value, etc)
Actually, Karaf (and related projects) already provides most
(all) data required for the monitoring. However, it would be
very helpful to have a "glue", ready to use and more user
friendly, including a storage of the metrics/monitoring data.
Regarding this, I started a prototype of a monitoring solution
for Karaf and the applications running in Karaf.
The purpose is to be very extendible, flexible, easy to install
and use.
In term of architecture, we can find the following component:
1/ Collectors & SLA Policies
The collectors are services responsible of harvesting monitoring
data.
We have two kinds of collectors:
- the polling collectors are invoked by a scheduler periodically.
- the event driven collectors react to some events.
Two collectors are already available:
- the JMX collector is a polling collector which harvest all
MBeans attributes
- the Log collector is a event driven collector, implementing a
PaxAppender which react when a log message occurs
We can planned the following collectors:
- a Camel Tracer collector would be an event driven collector,
acting as a Camel Interceptor. It would allow to trace any
Exchange in Camel.
It's very dynamic (thanks to OSGi services), so it's possible to
add a new custom collector (user/custom implementation).
The Collectors are also responsible of checking the SLA. As the
SLA policies are tight to the collected data, it makes sense
that the collector validates the SLA and call/delegate the alert
to SLA services.
2/ Scheduler
The scheduler service is responsible to call the Polling
Collectors, gather the harvested data, and delegate to the
dispatcher.
We already have a simple scheduler (just a thread), but we can
plan a quartz scheduler (for advanced cron/trigger
configuration), and another one leveraging the Karaf scheduler.
3/ Dispatcher
The dispatcher is called by the scheduler or the event driven
collectors to dispatch the collected data to the appenders.
4/ Appenders
The appender services are responsible to send/store the
collected data to target systems.
For now, we have two appenders:
- a log appender which just log the collected data
- a elasticsearch appender which send the collected data to a
elasticsearch instance. For now, it uses "external"
elasticsearch, but I'm working on an elasticsearch feature
allowing to embed elasticsearch in Karaf (it's mostly done).
We can plan the following other appenders:
- redis to send the collected data in Redis messaging system
- jdbc to store the collected data in a database
- jms to send the collected data to a JMS broker (like ActiveMQ)
- camel to send the collected data to a Camel direct-vm/vm
endpoint of a route (it would create an internal route)
5/ Console/Kibana
The console is composed by two parts:
- a angularjs or bootstrap layer allowing to configure the SLA
and global settings
- embedded kibana instance with pre-configured dashboard (when
the elasticsearch appender is used). We will have a set of
already created lucene queries and a kind of
"Karaf/Camel/ActiveMQ/CXF" dashboard template. The kibana
instance will be embedded in Karaf (not external).
Of course, we have ready to use features, allowing to very
easily install modules that we want.
I named the prototype Karaf Decanter. I don't have preference
about the name, and the location of the code (it could be as
Karaf subproject like Cellar or Cave, or directly in the Karaf
codebase).
Thoughts ?
Regards
JB
--
Jean-Baptiste Onofré
[email protected] <mailto:[email protected]>
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Charlie Mordant
Full OSGI/EE stack made with Karaf:
https://github.com/OsgiliathEnterprise/net.osgiliath.parent
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
