I finally managed to take a close look into decanter this weekend.
I really like the idea of managing logging events as a Map<String,
Object>. It allows a lot of flexibiltiy while giving the events a clear
key/value structure that is very near
how key/value stores work. Maybe we need to specify some key names for
typical data like the timestamp to make the format clearer but in
general I like that it can transport almost every simple data.
In general the reason why I looked into decanter is that I am searching
for a logging solution that covers technical logging (like the karaf log
file) and business logging (events that are directly in the vocabulary
of the business users). Additionally the solution should not require a
lot of configuration.
So in slf4j / log4j a nice way to implement business logging is to us
the MDC. It allows to add arbitrary key/value pairs to logging messages.
So it is easy to add for example a customer id or a contract id to
logging messages. This allows to use tools like kibana to look into
transactions of a specific user.
For the cases where you do not want to use the logging framework OSGi
events would make a perfect match to transports Map data while providing
loose coupling.
So when I looked into decanter with the above requirements / ideas I
found the following place where we could improve it:
1. Structure of the Appender data
I think the structure of the Appender data is a bit awkward:
Map<Long, Map<String, Object>> data
This means that we can not transport data if it happens at the same Date
which in practice could happen. So if we need several events we should
rather use Collection<Map<String, Object>>
I am not sure if we really need several events in one call. The
Appenders could do this internally so I would rather have the data be
just a simple Map<String, Object>
2. Use of the EventAdmin service
If we just need to transport a Map then why not simply use the OSGi
EventAdmin. It transports a Map<String, Object> on a topic. The topic is
even very similar to logging categories.
EventAdmin is standardized so we do not have to invent a new interface.
An appender would then simply subscribe to a topic and process the Events.
A collector would simply create and send EventAdmin events.
The EventAdmin already provides a nice implementation of the Dispatcher.
3. Logging of MDC data
The current appender.log does not add the mdc Map to the log data. I
plan to add this to better cover the business logging requirement. One
question here is of we should rather flatten the MDC values using a
prefix on the key or rather put the whole MDC map into the data map
under one key "mdc". At least elastic search can process such nested maps.
4. How to send to elastic search.
The current appender.elasticsearch uses the elasticsearch API. I propose
to use JestClient instead. JestClient uses the HttpRest interface of
elastic search and can also be configured for Authentication. It should
have a smaller footprint then full elastic search.
5. The decanter feature is version 1.3 of the xsd and seems to fail in
karaf < 4
So I propose to use the older feature definition to make sure we can
also cover at least karaf 3.
WDYT?
Btw. I also plan to improve the CXF request / response logging to
extract meta data from Rest services too and map the meta data to MDC
key/values. So this would work very well with decanter (with MDC
improvement). With just very little configuration all meta data could be
transported to elastic search and anaylized nicely.
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
