Sorry to revive this old thread. However, we're in the process of adding
support for other "categories" of events and thus I wanted to first take a step
back and try to ensure we're not convoluting things.
There was a requirement to log a "compliance" event under certain conditions.
We did not want to write our own logging framework and instead decided to use
existing off-the-shelf logging frameworks. We have applications on both
Linux/Java, Windows/.NET and Windows/Java. Initially we chose log4net for
Windows/.NET and log4j2 for Windows/Java and Linux/Java. For these logging
frameworks we wrote artifacts, appenders basically, to help facilitate getting
these events to our system. By the way, our system will get the events
centrally, possibly put them into a relational database and also hand them off
to another system which will get them eventually to an HDFS backend. We also
exposed methods for creating this compliance event. The compliance event is
basically a map. We chose a map so that the event could also be extended by
the application team in case they needed to add additional properties which
made sense for them.
We chose to create a custom level for this "compliance" event such that we
could filter out only these events and get them into our system. The
configuration example we created had our custom unix domain socket appender
added to the root logger. It also contained a filter which filtered out any
events that weren't compliance events. The level we chose for "compliance" was
less critical than off and more critical than fatal as we wanted to ensure that
as long as logging wasn't turned off altogether our events would get logged.
I want to go over a few suggestions that were made and explain why I didn't
make use of those suggestions.
1. Since our "compliance" level didn't fit within the "vernacular" of the
existing levels we should not define this custom level. Instead we should look
at using markers.
I am not that familiar with markers but did look into them when they were
suggested. While I don't have anything against markers in general there were
some downsides as I saw it.
a. Markers are not available, as far as I know, in log4net so we'd still have
to figure out something there.
b. A bigger problem, at least I thought it was a bigger problem, was that there
would be confusion about what level to log the event at. I would certainly not
want to give an example as follows:
logger.debug(COMPLIANCE_MARKER, evnt);
or
logger.info(COMPLIANCE_MARKER, evnt);
or
logger.error(COMPLIANCE_MARKER, evnt);
...
That just screams confusion to me.
2. Use a dedicated logger to log all compliance events.
There were, as far as I could tell, a couple problems with this approach also.
a. If everyone is using a single "well known" logger to log a specific event
category then I lose the logger "context" of who's logging the event. As it
stands now we're copying the logger name into a property we call "eventSource".
b. You cannot turn on/off logging for a specific set of code. If it turns out
that we have some errant code which is using this well known logger then we
can't just turn off that code from logging as turning off the well know logger
would turn it off for everyone using it.
I did look into the EventLogger and initially that seemed promising as I guess
it logs any event you give it at the "all" level. However, as a well known
logger it suffers from the same issues above.
Now we're looking to add Business events. My initial thinking is that I can do
the same thing we did before. Add an additional custom level called "Business"
and expose methods for creating a business event. Though unlike the compliance
event, the application teams would be defining the schema more so than our
framework team. Thus any method we expose would just be used as a starting
point for setting the common properties. You would use another instance of our
unix domain socket appender for these business events and forward them to a
different location as business events would most likely have a different
retention period than compliance events. Plus you might also want them in a
different store as you may never need to query for both categories of events
and thus no need to query against a larger set of data.
In addition we're planning to capture centrally what we refer to as diagnostic
events: error, info, warn, debug, trace, etc. However, we may need to separate
these out into two different categories: critical-diagnostic and
noncritical-diagnostic. The reason is that we don't want the potential of a
critical diagnostic event, let's say an error, queued up behind potentially
thousands of non-critical diagnostic events. So you see, the category also
defines aspects on how we handle events at the source. We separate at the
source based on category as it seems a reasonable place to do so. Also, you
may want different flush times for different categories. We have a process
which buffers, compresses and sends events centrally so we have the notion of
flush time. The buffers are flushed when they become full or the flush time
elapses. Errors, since they are more critical in monitoring systems, we'll
most likely want to flush more often than say debug and trace events.
Sorry for the long winded explanation. Initially I was thinking that when we
create an event we'd set its category. However, now I'm thinking the category
should be set by the act of logging the event at a level. In some cases we
have a 1:1 mapping from level to category, eg. compliance level -> compliance
category. In some cases we have a many:1 mapping from level to category, eg.
error, info, warn -> critical-diagnostic.
We could also just define a single custom level, say "always_on", or something
like that. Then we provide some helper method to log our "new" event
categories (eg. business and compliance) at this level and have the user
specify the category, I guess similar to a marker.
logEvent(Logger logger, String category, object evnt);
I guess it's similar to the EventLogger except that we're not using a single
well known logger and thus don't have the downsides of that which I pointed out
earlier.
Any thoughts/suggestions would be appreciated.
Thanks,
Nick
________________________________
From: Mikael Ståldal <mikael.stal...@magine.com>
Sent: Wednesday, September 9, 2015 3:47 AM
To: Log4J Users List
Subject: Re: approach for defining loggers
Then perhaps you should create your own facade for doing business event
logging, which could then forward them to Log4j in an appropriate way.
On Wed, Sep 9, 2015 at 4:49 AM, Nicholas Duane <nic...@msn.com> wrote:
> I was just about to reply to your previous email about using a single
> "business" logger, or some hierarchy of business loggers, to log business
> events and say that we might go that route. However, now that you brought
> up the post from Ralph, which I just replied to, I'm thinking a logger
> won't work either for the same reason I listed in my reply to Ralph's post.
>
> You could do:
>
> logger.info("Hello");
> logger.fatal("Hello");
> logger.error("Hello");
> ...
>
> It's confusing as there are n ways to log a business event that way and
> they will all do the same thing. Which one should a developer choose.
> Should I say pick any one, it doesn't matter?
>
> Thanks,
> Nick
>
> > Date: Tue, 8 Sep 2015 19:28:21 -0700
> > Subject: Re: approach for defining loggers
> > From: garydgreg...@gmail.com
> > To: log4j-user@logging.apache.org
> >
> > Or
> > Logger logger = LogManager.getLogger("Business");
> > ...
> > logger.info("Hello");
> >
> > Gary
> >
> > On Tue, Sep 8, 2015 at 7:24 PM, Ralph Goers <ralph.go...@dslextreme.com>
> > wrote:
> >
> > > Can you please clarify, “If we had some way to know an event is a
> business
> > > event we wouldn’t need level”? I do not understand how you can code
> > > logger.log(BUSINESS, msg) but you cannot code logger.info(BUSINESS,
> msg).
> > >
> > > Ralph
> > >
> > > > On Sep 8, 2015, at 6:09 PM, Nicholas Duane <nic...@msn.com> wrote:
> > > >
> > > >
> > > >
> > > >
> > > > I looked over that stackoverflow post and I'm still not seeing a good
> > > match as a way for us to log our business events.
> > > >
> > > > A business event I guess is an event which extends whatever schema we
> > > come up with for a business event. While an instance of this schema
> could
> > > be logged at any level, that really doesn't make sense in our scenario,
> > > regardless of whether some marker was supplied. If we had some way to
> know
> > > an event is a business event we wouldn't need level. We could of
> course
> > > add some property to our schema which indicates the 'category' of the
> > > event, 'business' being one such category. Instead we were thinking we
> > > could just use level to indicate that an event is a business event.
> > > >
> > > > As I mentioned, we're looking to capture 'trace' level events to one
> > > store, 'info' - 'fatal' level events to another store, and 'business'
> > > events to yet another store. For 'trace' and 'info' - 'fatal' it seems
> > > reasonable to filter on level within the appender to get those events
> to
> > > the appropriate location. It seemed reasonable to do something
> similar for
> > > 'business'.
> > > >
> > > > I also looked into the EventLogger but not sure that's appropriate.
> For
> > > one we lose the granularity to control a specific piece of code from
> > > generating business events. This is most likely a non-issue as I have
> > > mentioned that we don't want to turn business logging off. The other
> is
> > > that we lose the name of the logger as it would be the same for
> everyone.
> > > Not sure this is that big a deal either as I guess you might be able to
> > > capture component name, though I would rather distinguish using logger
> name.
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > > >> From: ralph.go...@dslextreme.com
> > > >> Subject: Re: approach for defining loggers
> > > >> Date: Mon, 7 Sep 2015 20:39:11 -0700
> > > >> To: log4j-user@logging.apache.org
> > > >>
> > > >> I still don’t understand why you don’t want to use Markers. They
> were
> > > designed exactly for the use case you are describing.
> > > >>
> > > >> You might set retention policies for debug vs info, error and fatal,
> > > but a BUSINESS marker could cross-cut them all. That is exactly why
> it is
> > > NOT a level. IOW, it gives you a second dimension for filtering. Ceki
> > > invented Markers when he created SLF4J. For his point of view see
> > >
> http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them
[http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded&a]<http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>
What is markers in Java Logging frameworks and that is a
...<http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>
stackoverflow.com
This is a rehashed version my answer to the question "Best practices for using
Markers in SLF4J/Logback". Markers can be used to color or mark a single log
statement.
> > > <
> > >
> http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them
[http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded&a]<http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>
What is markers in Java Logging frameworks and that is a
...<http://stackoverflow.com/questions/16813032/what-is-markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>
stackoverflow.com
This is a rehashed version my answer to the question "Best practices for using
Markers in SLF4J/Logback". Markers can be used to color or mark a single log
statement.
> > > >.
> > > >>
> > > >> Ralph
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> On Sep 7, 2015, at 5:54 PM, Nicholas Duane <nic...@msn.com> wrote:
> > > >>>
> > > >>> If I'm attempting to control all the logging from the configuration
> > > and I don't know the complete set of loggers in my application as there
> > > could be 100's or 1000's, wouldn't it be hard to separate events based
> on
> > > loggers? It would seem much easier to separate events based on
> level. In
> > > addition, level might be a more reasonable approach for separating.
> For
> > > example, if I want to send all events to some big-data backend I might
> want
> > > to separate out traces and debug from info to fatal as traces and
> debug are
> > > most likely less important from a systems management aspect. My
> retention
> > > period for traces and debug might be just a couple days. The retention
> > > period for info to fatal could be 30 days. Business level might be 2
> > > years. Any system management notifications would probably be driven
> off of
> > > info to fatal events and not trace and debug events, which is another
> > > reason you might want to separate by level.
> > > >>>
> > > >>> Thanks,
> > > >>> Nick
> > > >>>
> > > >>>> Subject: Re: approach for defining loggers
> > > >>>> From: ralph.go...@dslextreme.com
> > > >>>> Date: Mon, 31 Aug 2015 08:50:58 -0700
> > > >>>> To: log4j-user@logging.apache.org
> > > >>>>
> > > >>>> A logging “Level” is a level of importance. That is why there is a
> > > hierarchy. If you want informational messages then you also would want
> > > warnings and errors.
> > > >>>>
> > > >>>> “BUSINESS” does not convey the same meaning. Rather, it is some
> sort
> > > of category, which is what Markers are for.
> > > >>>>
> > > >>>> Using the class name as the logger name is a convention. If you
> > > really want the class name, method name or line number then you should
> be
> > > specifying that you want those from the logging event, rather than the
> > > logger name. Unless location information is disabled you always have
> > > access to that information.
> > > >>>>
> > > >>>> In short, different loggers are used primarily as a way of
> grouping
> > > sets of messages - for example all org.hibernate events can be routed
> to a
> > > specific appender or turned off en masse. Levels are used to filter out
> > > noise across a set of logging events. Markers are used to categorize
> > > logging events by arbitrary attributes.
> > > >>>>
> > > >>>> Ralph
> > > >>>>
> > > >>>>
> > > >>>>> On Aug 31, 2015, at 8:10 AM, Nicholas Duane <nic...@msn.com>
> wrote:
> > > >>>>>
> > > >>>>> Thanks for the feedback. I will look into Markers and MDC.
> > > >>>>>
> > > >>>>> With respect to using a separate logger, it would seem I would
> lose
> > > the information about what application code, eg. the class logger, is
> > > sourcing the event. We would like to have this information. On top of
> > > that, it seems odd, maybe to me only, that for this new level we have
> our
> > > own logger. It seemed reasonable to me that this new event we want to
> > > capture is just a new level. Just like a DEBUG event is different
> from an
> > > INFO event. If I define a BUSINESS level why would that not follow the
> > > same design as the current levels? You wouldn't suggest having
> different
> > > loggers for TRACE DEBUG INFO WARN ERROR FATAL, would you? I think one
> of
> > > the reasons someone on our side is suggesting I have separate loggers
> is
> > > that they think the overhead of filtering at the appender is going to
> have
> > > a noticeable impact. Our plan, at least the one I have now in my
> head, is
> > > that we'll have some number of appenders in the root. We'll then
> filter x
> > > < INFO events to a tracing appender, INFO <= x <= FATAL to a logging
> > > appender, and our custom level will go to another appender. Thoughts?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Nick
> > > >>>>>
> > > >>>>>> Subject: Re: approach for defining loggers
> > > >>>>>> From: ralph.go...@dslextreme.com
> > > >>>>>> Date: Sat, 29 Aug 2015 20:59:36 -0700
> > > >>>>>> To: log4j-user@logging.apache.org
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> On Aug 29, 2015, at 7:44 PM, Nicholas Duane <nic...@msn.com>
> > > wrote:
> > > >>>>>>>
> > > >>>>>>> I'm curious if there is a prescribed approach to defining
> > > loggers. Let me state what my assumption is. I assume that normally
> if
> > > some piece of code wants to log events/messages that it should create a
> > > logger for itself. I guess a reasonable name to use is the class name
> > > itself. In terms of logger configuration I would expect that no
> loggers
> > > are specified in the log4j configuration UNLESS is needs settings other
> > > than the default. The root logger would specify the default settings,
> eg.
> > > level and appenders. If some piece of code tied to a logger needs to
> > > enable tracing in order to debug an issue then you would add that
> logger to
> > > the configuration and set the level less specific for that logger. Is
> this
> > > a typical and reasonable approach?
> > > >>>>>>
> > > >>>>>> What you describe here is the common convention. It is a
> reasonable
> > > approach.
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> I asked because we have the need for a new type of event. To
> have
> > > this event flow to where we want it to flow the plan is to have a
> custom
> > > level and have all events at that level captured by a specific
> appender.
> > > My assumption was that for existing applications we'd just need to add
> our
> > > appender to the root and add our custom level. The app would need to
> be
> > > modified to log our new event at the custom level. However, someone
> > > suggested that we could also create a separate logger for this event.
> My
> > > thinking is that while we don't ever want to turn off logging of this
> > > event, loggers represent "event sources", e.g the code raising the
> events
> > > and thus having multiple different pieces of code use the same logger
> > > wouldn't allow you to turn on/off logging from those different
> sections of
> > > code independently. I think the current configuration includes all the
> > > loggers. Normally I would expect there to be many, on the order of
> 10's or
> > > 100's, loggers within an application. However, in the case I was given
> > > there were only a handful because I think this handful is shared. So
> as I
> > > mentioned, this doesn't sound like an ideal design as you have less
> > > granularity on what you can turn on/off.
> > > >>>>>>
> > > >>>>>> You have a few options. Using a CustomLevel would not be the
> option
> > > I would choose. Creating a custom Logger will certainly work and makes
> > > routing the message to the appropriate appender rather easy. Another
> > > approach is to use Markers. Markers are somewhat hierarchical so you
> can
> > > use them for a variety of purposes. If you look at how Log4j handles
> event
> > > logging it actually does both - it specifies EventLogger as the name
> of the
> > > logger to use and it uses Markers to identify the kind of event.
> > > >>>>>>
> > > >>>>>> A third option is to use the MDC or Logger properties. If you do
> > > that then you can have information included in the actual logging event
> > > that can affect how it is routed. I also built a system that uses the
> > > RFC5424 format so that the event could have lots of key/value pairs to
> > > identify the events.
> > > >>>>>>
> > > >>>>>> Unfortunately, without knowing more details I don’t know that I
> can
> > > give you a better idea on how I would implement it.
> > > >>>>>>
> > > >>>>>> Ralph
> > > >>>>>>
> > > >>>>>>
> > > ---------------------------------------------------------------------
> > > >>>>>> To unsubscribe, e-mail:
> log4j-user-unsubscr...@logging.apache.org
> > > >>>>>> For additional commands, e-mail:
> log4j-user-h...@logging.apache.org
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> ---------------------------------------------------------------------
> > > >>>> To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
> > > >>>> For additional commands, e-mail:
> log4j-user-h...@logging.apache.org
> > > >>>>
> > > >>>
> > > >>
> > > >
> > > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
> > > For additional commands, e-mail: log4j-user-h...@logging.apache.org
> > >
> > >
> >
> >
> > --
> > E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> > Java Persistence with Hibernate, Second Edition
> > <http://www.manning.com/bauer3/>
> > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
> > Spring Batch in Action <http://www.manning.com/templier/>
> > Blog: http://garygregory.wordpress.com
[https://s0.wp.com/i/blank.jpg]<http://garygregory.wordpress.com/>
Gary Gregory<http://garygregory.wordpress.com/>
garygregory.wordpress.com
Software construction, the web, and other techs
> > Home: http://garygregory.com/
Gary Gregory<http://garygregory.com/>
garygregory.com
Rocket | Seagull . I am a Software Architect for Seagull Software, a division
of Rocket Software. Rocket Seagull specializes in tools and expertise to
modernize ...
> > Tweet! http://twitter.com/GaryGregory
Gary Gregory (@GaryGregory) | Twitter<http://twitter.com/GaryGregory>
twitter.com
The latest Tweets from Gary Gregory (@GaryGregory). Principal Software
Engineer, author: Java Persistence Hibernate https://t.co/3F8sYxc0oq, JUnit
https://t.co/yXU1DqAMDG, Spring Batch https://t.co/XwoMNoBxh7. U.S.A.
>
>
--
[image: MagineTV]
*Mikael Ståldal*
Senior software developer
*Magine TV*
mikael.stal...@magine.com
Regeringsgatan 25 | 111 53 Stockholm, Sweden |
www.magine.com<http://www.magine.com>
[https://de.magine.com/content/uploads/2016/09/magine_global_social.png]<http://www.magine.com/>
TV online with Magine TV<http://www.magine.com/>
www.magine.com
Watch the TV you love, on any device, anywhere in Germany and Sweden and find
out more about our global OTT B2B solutions. Get started today.
Privileged and/or Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such a person), you may not
copy or deliver this message to anyone. In such case,
you should destroy this message and kindly notify the sender by reply
email.
