Hi Christian,
first of all, thanks to have taken the time to throw a look on decanter
;) Really appreciate.
See my comments inline regarding your points:
1. Structure of the Appender data
I think the structure of the Appender data is a bit awkward:
Map<Long, Map<String, Object>> data
This means that we can not transport data if it happens at the same Date
which in practice could happen. So if we need several events we should
rather use Collection<Map<String, Object>>
I am not sure if we really need several events in one call. The
Appenders could do this internally so I would rather have the data be
just a simple Map<String, Object>
Actually, it's not a big deal to have multiple events at the same time.
The timestamp is used to index, but the String is the event key. So, if
at the same time you have multiple events with different keys, it's not
a problem.
The drawback having a Collection is that the events come from different
collector, atomic and independent each other. Imagine you have same type
of event coming from different collectors (which is possible), it would
be tricky to maintain a collection between these different collectors.
2. Use of the EventAdmin service
If we just need to transport a Map then why not simply use the OSGi
EventAdmin. It transports a Map<String, Object> on a topic. The topic is
even very similar to logging categories.
EventAdmin is standardized so we do not have to invent a new interface.
An appender would then simply subscribe to a topic and process the Events.
A collector would simply create and send EventAdmin events.
The EventAdmin already provides a nice implementation of the Dispatcher.
AFAIR, I mentioned EventAdmin in the proposal e-mail but more as an
Event Driven Collector. I like your idea of using EventAdmin as
Dispatcher, but it means that Decanter will be tight to EventAdmin (not
a big deal anyway).
3. Logging of MDC data
The current appender.log does not add the mdc Map to the log data. I
plan to add this to better cover the business logging requirement. One
question here is of we should rather flatten the MDC values using a
prefix on the key or rather put the whole MDC map into the data map
under one key "mdc". At least elastic search can process such nested maps.
+1, it's the key to add.
4. How to send to elastic search.
The current appender.elasticsearch uses the elasticsearch API. I propose
to use JestClient instead. JestClient uses the HttpRest interface of
elastic search and can also be configured for Authentication. It should
have a smaller footprint then full elastic search.
+1 for the client, but anyway, as one of the aim of Decanter is to be
"turn key" ready (very simple to install and use), I created the
elasticsearch feature in decanter. It installs a elasticsearch
instance/node in Karaf.
5. The decanter feature is version 1.3 of the xsd and seems to fail in
karaf < 4
So I propose to use the older feature definition to make sure we can
also cover at least karaf 3.
+1
WDYT?
Btw. I also plan to improve the CXF request / response logging to
extract meta data from Rest services too and map the meta data to MDC
key/values. So this would work very well with decanter (with MDC
improvement). With just very little configuration all meta data could be
transported to elastic search and anaylized nicely.
Good idea, it's what I said in the proposal: CXF Decanter interceptor,
Camel tracer for Decanter, etc, all those collectors will provide great
value to Decanter.
On my side, I'm have a kibana and Decanter Console first draft ready.
Thanks !
Regards
JB
Christian
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
