Hi Christian, please find my comments inline:
2015-03-09 10:45 GMT+01:00 Christian Schneider <[email protected]>: > I finally managed to take a close look into decanter this weekend. > > I really like the idea of managing logging events as a Map<String, > Object>. It allows a lot of flexibiltiy while giving the events a clear > key/value structure that is very near > how key/value stores work. Maybe we need to specify some key names for > typical data like the timestamp to make the format clearer but in general I > like that it can transport almost every simple data. > > In general the reason why I looked into decanter is that I am searching > for a logging solution that covers technical logging (like the karaf log > file) and business logging (events that are directly in the vocabulary of > the business users). Additionally the solution should not require a lot of > configuration. > > So in slf4j / log4j a nice way to implement business logging is to us the > MDC. It allows to add arbitrary key/value pairs to logging messages. So it > is easy to add for example a customer id or a contract id to logging > messages. This allows to use tools like kibana to look into transactions of > a specific user. > > For the cases where you do not want to use the logging framework OSGi > events would make a perfect match to transports Map data while providing > loose coupling. > > So when I looked into decanter with the above requirements / ideas I found > the following place where we could improve it: > > 1. Structure of the Appender data > I think the structure of the Appender data is a bit awkward: > Map<Long, Map<String, Object>> data > This means that we can not transport data if it happens at the same Date > which in practice could happen. So if we need several events we should > rather use Collection<Map<String, Object>> > I am not sure if we really need several events in one call. The Appenders > could do this internally so I would rather have the data be just a simple > Map<String, Object> > which appender did you look at? The JMX appender is approprate and the timestamp is collected by the elasticsearch trigger/appender transformed to the required timestamp field (take a look at the changes I did) > 2. Use of the EventAdmin service > If we just need to transport a Map then why not simply use the OSGi > EventAdmin. It transports a Map<String, Object> on a topic. The topic is > even very similar to logging categories. > EventAdmin is standardized so we do not have to invent a new interface. An > appender would then simply subscribe to a topic and process the Events. > A collector would simply create and send EventAdmin events. > The EventAdmin already provides a nice implementation of the Dispatcher. > > I'd rather stick to the JMX appender as the current default one and would make sure the others do log accordingly. > 3. Logging of MDC data > The current appender.log does not add the mdc Map to the log data. I plan > to add this to better cover the business logging requirement. One question > here is of we should rather flatten the MDC values using a prefix on the > key or rather put the whole MDC map into the data map under one key "mdc". > At least elastic search can process such nested maps. > > make sure the stuff looks similar to the changes I did with the JMX appender. > 4. How to send to elastic search. > The current appender.elasticsearch uses the elasticsearch API. I propose > to use JestClient instead. JestClient uses the HttpRest interface of > elastic search and can also be configured for Authentication. It should > have a smaller footprint then full elastic search. > > it's the one used for the internal one and does work appropriate. If the elastic search is external a logging appender with the standard ELK stack (elasticsearch, logstash, kibana) should be sufficient. > 5. The decanter feature is version 1.3 of the xsd and seems to fail in > karaf < 4 > So I propose to use the older feature definition to make sure we can also > cover at least karaf 3. > > worked like a charm for me on Karaf 4. > > WDYT? > > Btw. I also plan to improve the CXF request / response logging to extract > meta data from Rest services too and map the meta data to MDC key/values. > So this would work very well with decanter (with MDC improvement). With > just very little configuration all meta data could be transported to > elastic search and anaylized nicely. > > Christian > > -- > Christian Schneider > http://www.liquid-reality.de > > Open Source Architect > http://www.talend.com > > -- Apache Member Apache Karaf <http://karaf.apache.org/> Committer & PMC OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer & Project Lead blog <http://notizblog.nierbeck.de/> Co-Author of Apache Karaf Cookbook <http://bit.ly/1ps9rkS> Software Architect / Project Manager / Scrum Master
