Hi Charles, I think it's what JB proposed in
4/ Appenders > The appender services are responsible to send/store the collected data to > target systems. > For now, we have two appenders: > - a log appender which just log the collected data > - a elasticsearch appender which send the collected data to a > elasticsearch instance. For now, it uses "external" elasticsearch, but I'm > working on an elasticsearch feature allowing to embed elasticsearch in > Karaf (it's mostly done). > We can plan the following other appenders: > - redis to send the collected data in Redis messaging system > - jdbc to store the collected data in a database > - jms to send the collected data to a JMS broker (like ActiveMQ) > - camel to send the collected data to a Camel direct-vm/vm endpoint of a > route (it would create an internal route) so yes, this sounds to me like a good idea. As usual make everything OSGi like, preferable services and therefore extendable/exchangeable :) regards, Achim 2014-10-15 11:27 GMT+02:00 Charles Moulliard <[email protected]>: > I don't know if you have planned something like this but that could be > interesting to have a layer to plugin the backend (elasticsearch, ...) if > we would like to plug later on another no-sql backend (mongodb, influxdb, > ...) > > On Wed, Oct 15, 2014 at 11:20 AM, Jean-Baptiste Onofré <[email protected]> > wrote: > > > Hi Charles, > > > > Good idea yes, I will prepare a README.md for the vote. > > > > Regards > > JB > > > > > > On 10/15/2014 11:18 AM, Charles Moulliard wrote: > > > >> Hi Jean Baptiste, > >> > >> I like the project name "decanter = décanter in French I suspect". Can > you > >> (when you will have the time of course) add a README.md or README.adoc > >> file > >> to your project (github) to explain what it does, ... ? > >> > >> Regards, > >> > >> On Wed, Oct 15, 2014 at 11:12 AM, Jean-Baptiste Onofré <[email protected] > > > >> wrote: > >> > >> It's the collected data. > >>> > >>> Basically, it's: > >>> > >>> - timestamp of the data/metric > >>> - map of key/value (for instance, JMX Attribute Name => JMX Attribute > >>> Value) > >>> > >>> Regards > >>> JB > >>> > >>> > >>> On 10/15/2014 11:11 AM, Guillaume Nodet wrote: > >>> > >>> Great thx ! > >>>> > >>>> First technical question, can you explain what does the Map<Long, > >>>> Map<String > >>>> , Object>> in the api interfaces (Collector, Appender, etc...) > >>>> represents > >>>> ? > >>>> > >>>> Guillaume > >>>> > >>>> 2014-10-15 11:08 GMT+02:00 Jean-Baptiste Onofré <[email protected]>: > >>>> > >>>> Oh by the way, I forgot the github link: > >>>> > >>>>> > >>>>> https://github.com/jbonofre/karaf-decanter > >>>>> > >>>>> Sorry about that guys ! > >>>>> > >>>>> Regards > >>>>> JB > >>>>> > >>>>> > >>>>> On 10/14/2014 05:12 PM, Jean-Baptiste Onofré wrote: > >>>>> > >>>>> Hi all, > >>>>> > >>>>>> > >>>>>> First of all, sorry for this long e-mail ;) > >>>>>> > >>>>>> Some weeks ago, I blogged about the usage of ELK > >>>>>> (Logstash/Elasticsearch/Kibana) with Karaf, Camel, ActiveMQ, etc to > >>>>>> provide a monitoring dashboard (know what's happen in Karaf and be > >>>>>> able > >>>>>> to store it for a long period): > >>>>>> > >>>>>> http://blog.nanthrax.net/2014/03/apache-karaf-cellar-camel- > >>>>>> activemq-monitoring-with-elk-elasticsearch-logstash-and-kibana/ > >>>>>> > >>>>>> > >>>>>> If this solution works fine, there are some drawbacks: > >>>>>> - it requires additional middlewares on the machines. Additionally > to > >>>>>> Karaf itself, we have to install logstash, elasticsearch nodes, and > >>>>>> kibana console > >>>>>> - it's not usable "out of the box": you need at least to configure > >>>>>> logstash (with the different input/output plugins), kibana (to > create > >>>>>> the dashboard that you need) > >>>>>> - it doesn't cover all the monitoring needs, especially in term of > >>>>>> SLA: > >>>>>> we want to be able to raise some alerts depending of some events > (for > >>>>>> instance, when a regex is match in the log messages, when a feature > is > >>>>>> uninstalled, when a JMX metric is greater than a given value, etc) > >>>>>> > >>>>>> Actually, Karaf (and related projects) already provides most (all) > >>>>>> data > >>>>>> required for the monitoring. However, it would be very helpful to > >>>>>> have a > >>>>>> "glue", ready to use and more user friendly, including a storage of > >>>>>> the > >>>>>> metrics/monitoring data. > >>>>>> > >>>>>> Regarding this, I started a prototype of a monitoring solution for > >>>>>> Karaf > >>>>>> and the applications running in Karaf. > >>>>>> The purpose is to be very extendible, flexible, easy to install and > >>>>>> use. > >>>>>> > >>>>>> In term of architecture, we can find the following component: > >>>>>> > >>>>>> 1/ Collectors & SLA Policies > >>>>>> The collectors are services responsible of harvesting monitoring > data. > >>>>>> We have two kinds of collectors: > >>>>>> - the polling collectors are invoked by a scheduler periodically. > >>>>>> - the event driven collectors react to some events. > >>>>>> Two collectors are already available: > >>>>>> - the JMX collector is a polling collector which harvest all MBeans > >>>>>> attributes > >>>>>> - the Log collector is a event driven collector, implementing a > >>>>>> PaxAppender which react when a log message occurs > >>>>>> We can planned the following collectors: > >>>>>> - a Camel Tracer collector would be an event driven collector, > acting > >>>>>> as > >>>>>> a Camel Interceptor. It would allow to trace any Exchange in Camel. > >>>>>> > >>>>>> It's very dynamic (thanks to OSGi services), so it's possible to > add a > >>>>>> new custom collector (user/custom implementation). > >>>>>> > >>>>>> The Collectors are also responsible of checking the SLA. As the SLA > >>>>>> policies are tight to the collected data, it makes sense that the > >>>>>> collector validates the SLA and call/delegate the alert to SLA > >>>>>> services. > >>>>>> > >>>>>> 2/ Scheduler > >>>>>> The scheduler service is responsible to call the Polling Collectors, > >>>>>> gather the harvested data, and delegate to the dispatcher. > >>>>>> We already have a simple scheduler (just a thread), but we can plan > a > >>>>>> quartz scheduler (for advanced cron/trigger configuration), and > >>>>>> another > >>>>>> one leveraging the Karaf scheduler. > >>>>>> > >>>>>> 3/ Dispatcher > >>>>>> The dispatcher is called by the scheduler or the event driven > >>>>>> collectors > >>>>>> to dispatch the collected data to the appenders. > >>>>>> > >>>>>> 4/ Appenders > >>>>>> The appender services are responsible to send/store the collected > data > >>>>>> to target systems. > >>>>>> For now, we have two appenders: > >>>>>> - a log appender which just log the collected data > >>>>>> - a elasticsearch appender which send the collected data to a > >>>>>> elasticsearch instance. For now, it uses "external" elasticsearch, > but > >>>>>> I'm working on an elasticsearch feature allowing to embed > >>>>>> elasticsearch > >>>>>> in Karaf (it's mostly done). > >>>>>> We can plan the following other appenders: > >>>>>> - redis to send the collected data in Redis messaging system > >>>>>> - jdbc to store the collected data in a database > >>>>>> - jms to send the collected data to a JMS broker (like ActiveMQ) > >>>>>> - camel to send the collected data to a Camel direct-vm/vm endpoint > >>>>>> of a > >>>>>> route (it would create an internal route) > >>>>>> > >>>>>> 5/ Console/Kibana > >>>>>> The console is composed by two parts: > >>>>>> - a angularjs or bootstrap layer allowing to configure the SLA and > >>>>>> global settings > >>>>>> - embedded kibana instance with pre-configured dashboard (when the > >>>>>> elasticsearch appender is used). We will have a set of already > created > >>>>>> lucene queries and a kind of "Karaf/Camel/ActiveMQ/CXF" dashboard > >>>>>> template. The kibana instance will be embedded in Karaf (not > >>>>>> external). > >>>>>> > >>>>>> Of course, we have ready to use features, allowing to very easily > >>>>>> install modules that we want. > >>>>>> > >>>>>> I named the prototype Karaf Decanter. I don't have preference about > >>>>>> the > >>>>>> name, and the location of the code (it could be as Karaf subproject > >>>>>> like > >>>>>> Cellar or Cave, or directly in the Karaf codebase). > >>>>>> > >>>>>> Thoughts ? > >>>>>> > >>>>>> Regards > >>>>>> JB > >>>>>> > >>>>>> > >>>>>> -- > >>>>> Jean-Baptiste Onofré > >>>>> [email protected] > >>>>> http://blog.nanthrax.net > >>>>> Talend - http://www.talend.com > >>>>> > >>>>> > >>>>> > >>>> -- > >>> Jean-Baptiste Onofré > >>> [email protected] > >>> http://blog.nanthrax.net > >>> Talend - http://www.talend.com > >>> > >>> > >> > >> > >> > > -- > > Jean-Baptiste Onofré > > [email protected] > > http://blog.nanthrax.net > > Talend - http://www.talend.com > > > > > > -- > Charles Moulliard > Apache Committer / Architect @RedHat > Twitter : @cmoulliard | Blog : http://cmoulliard.github.io > -- Apache Member Apache Karaf <http://karaf.apache.org/> Committer & PMC OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer & Project Lead blog <http://notizblog.nierbeck.de/> Co-Author of Apache Karaf Cookbook <http://bit.ly/1ps9rkS> Software Architect / Project Manager / Scrum Master
