On Mon, Oct 17, 2016 at 4:27 PM, Nicholas Duane <nic...@msn.com> wrote:

> Sorry to revive this old thread.  However, we're in the process of adding
> support for other "categories" of events and thus I wanted to first take a
> step back and try to ensure we're not convoluting things.
>
>
> There was a requirement to log a "compliance" event under certain
> conditions.  We did not want to write our own logging framework and instead
> decided to use existing off-the-shelf logging frameworks.  We have
> applications on both Linux/Java, Windows/.NET and Windows/Java.  Initially
> we chose log4net for Windows/.NET and log4j2 for Windows/Java and
> Linux/Java.  For these logging frameworks we wrote artifacts, appenders
> basically, to help facilitate getting these events to our system.  By the
> way, our system will get the events centrally, possibly put them into a
> relational database and also hand them off to another system which will get
> them eventually to an HDFS backend.  We also exposed methods for creating
> this compliance event.  The compliance event is basically a map.  We chose
> a map so that the event could also be extended by the application team in
> case they needed to add additional properties which made sense for them.
>
>
> We chose to create a custom level for this "compliance" event such that we
> could filter out only these events and get them into our system.  The
> configuration example we created had our custom unix domain socket appender
> added to the root logger.  It also contained a filter which filtered out
> any events that weren't compliance events.  The level we chose for
> "compliance" was less critical than off and more critical than fatal as we
> wanted to ensure that as long as logging wasn't turned off altogether our
> events would get logged.
>
>
> I want to go over a few suggestions that were made and explain why I
> didn't make use of those suggestions.
>
>
> 1. Since our "compliance" level didn't fit within the "vernacular" of the
> existing levels we should not define this custom level.  Instead we should
> look at using markers.
>

Yes, this is a use case for markers. The level should be used to note how
important is each compliance event.


>
> I am not that familiar with markers but did look into them when they were
> suggested.  While I don't have anything against markers in general there
> were some downsides as I saw it.
>
>
> a. Markers are not available, as far as I know, in log4net so we'd still
> have to figure out something there.
>

Indeed, we really need a port of Log4j 2 to .NET.


>
> b. A bigger problem, at least I thought it was a bigger problem, was that
> there would be confusion about what level to log the event at.  I would
> certainly not want to give an example as follows:
>
>
> logger.debug(COMPLIANCE_MARKER, evnt);
>
>
> or
>
>
> logger.info(COMPLIANCE_MARKER, evnt);
>
>
> or
>
>
> logger.error(COMPLIANCE_MARKER, evnt);
>
>
> ...
>

Think about: How important is this event? Are there different level of
importance to the audience?


>
>
> That just screams confusion to me.
>
>
> 2. Use a dedicated logger to log all compliance events.
>
>
> There were, as far as I could tell, a couple problems with this approach
> also.
>
>
> a. If everyone is using a single "well known" logger to log a specific
> event category then I lose the logger "context" of who's logging the
> event.  As it stands now we're copying the logger name into a property we
> call "eventSource".
>

A practice is to use one logger per class. Another is to use a higher-level
logger to represent higher-level abstractions like a module.


>
>
> b. You cannot turn on/off logging for a specific set of code.  If it turns
> out that we have some errant code which is using this well known logger
> then we can't just turn off that code from logging as turning off the well
> know logger would turn it off for everyone using it.
>
>
> I did look into the EventLogger and initially that seemed promising as I
> guess it logs any event you give it at the "all" level.  However, as a well
> known logger it suffers from the same issues above.
>
>
> Now we're looking to add Business events.  My initial thinking is that I
> can do the same thing we did before.  Add an additional custom level called
> "Business" and expose methods for creating a business event.


I would NOT create a custom level. Instead, I would use a Logger called
"Business".


> Though unlike the compliance event, the application teams would be
> defining the schema more so than our framework team.  Thus any method we
> expose would just be used as a starting point for setting the common
> properties.  You would use another instance of our unix domain socket
> appender for these business events and forward them to a different location
> as business events would most likely have a different retention period than
> compliance events.  Plus you might also want them in a different store as
> you may never need to query for both categories of events and thus no need
> to query against a larger set of data.
>
>
> In addition we're planning to capture centrally what we refer to as
> diagnostic events: error, info, warn, debug, trace, etc.  However, we may
> need to separate these out into two different categories:
> critical-diagnostic and noncritical-diagnostic.


This could be a user case for custom levels IF one is more important than
the other which it sure sounds like it is.



> The reason is that we don't want the potential of a critical diagnostic
> event, let's say an error, queued up behind potentially thousands of
> non-critical diagnostic events.  So you see, the category also defines
> aspects on how we handle events at the source.   We separate at the source
> based on category as it seems a reasonable place to do so.  Also, you may
> want different flush times for different categories.  We have a process
> which buffers, compresses and sends events centrally so we have the notion
> of flush time.  The buffers are flushed when they become full or the flush
> time elapses.  Errors, since they are more critical in monitoring systems,
> we'll most likely want to flush more often than say debug and trace events.
>
>
> Sorry for the long winded explanation.  Initially I was thinking that when
> we create an event we'd set its category.  However, now I'm thinking the
> category should be set by the act of logging the event at a level.  In some
> cases we have a 1:1 mapping from level to category, eg. compliance level ->
> compliance category.  In some cases we have a many:1 mapping from level to
> category, eg. error, info, warn -> critical-diagnostic.
>
>
> We could also just define a single custom level, say "always_on", or
> something like that.  Then we provide some helper method to log our "new"
> event categories (eg. business and compliance) at this level and have the
> user specify the category, I guess similar to a marker.
>

Log4j has a level called ALL.

I would really try to work hard to stay within the feature set before
thinking about anything custom.

If you can make critical-diagnostic and noncritical-diagnostic events to
stock levels, that much the better.

Gary

>
>
> logEvent(Logger logger, String category, object evnt);
>
>
> I guess it's similar to the EventLogger except that we're not using a
> single well known logger and thus don't have the downsides of that which I
> pointed out earlier.
>
>
> Any thoughts/suggestions would be appreciated.
>
>
> Thanks,
>
> Nick
>
> ________________________________
> From: Mikael Ståldal <mikael.stal...@magine.com>
> Sent: Wednesday, September 9, 2015 3:47 AM
> To: Log4J Users List
> Subject: Re: approach for defining loggers
>
> Then perhaps you should create your own facade for doing business event
> logging, which could then forward them to Log4j in an appropriate way.
>
> On Wed, Sep 9, 2015 at 4:49 AM, Nicholas Duane <nic...@msn.com> wrote:
>
> > I was just about to reply to your previous email about using a single
> > "business" logger, or some hierarchy of business loggers, to log business
> > events and say that we might go that route.  However, now that you
> brought
> > up the post from Ralph, which I just replied to, I'm thinking a logger
> > won't work either for the same reason I listed in my reply to Ralph's
> post.
> >
> > You could do:
> >
> > logger.info("Hello");
> > logger.fatal("Hello");
> > logger.error("Hello");
> > ...
> >
> > It's confusing as there are n ways to log a business event that way and
> > they will all do the same thing.  Which one should a developer choose.
> > Should I say pick any one, it doesn't matter?
> >
> > Thanks,
> > Nick
> >
> > > Date: Tue, 8 Sep 2015 19:28:21 -0700
> > > Subject: Re: approach for defining loggers
> > > From: garydgreg...@gmail.com
> > > To: log4j-user@logging.apache.org
> > >
> > > Or
> > > Logger logger = LogManager.getLogger("Business");
> > > ...
> > > logger.info("Hello");
> > >
> > > Gary
> > >
> > > On Tue, Sep 8, 2015 at 7:24 PM, Ralph Goers <
> ralph.go...@dslextreme.com>
> > > wrote:
> > >
> > > > Can you please clarify, “If we had some way to know an event is a
> > business
> > > > event we wouldn’t need level”?  I do not understand how you can code
> > > > logger.log(BUSINESS, msg)  but you cannot code logger.info(BUSINESS,
> > msg).
> > > >
> > > > Ralph
> > > >
> > > > > On Sep 8, 2015, at 6:09 PM, Nicholas Duane <nic...@msn.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > I looked over that stackoverflow post and I'm still not seeing a
> good
> > > > match as a way for us to log our business events.
> > > > >
> > > > > A business event I guess is an event which extends whatever schema
> we
> > > > come up with for a business event.  While an instance of this schema
> > could
> > > > be logged at any level, that really doesn't make sense in our
> scenario,
> > > > regardless of whether some marker was supplied.  If we had some way
> to
> > know
> > > > an event is a business event we wouldn't need level.  We could of
> > course
> > > > add some property to our schema which indicates the 'category' of the
> > > > event, 'business' being one such category.  Instead we were thinking
> we
> > > > could just use level to indicate that an event is a business event.
> > > > >
> > > > > As I mentioned, we're looking to capture 'trace' level events to
> one
> > > > store, 'info' - 'fatal' level events to another store, and 'business'
> > > > events to yet another store.  For 'trace' and 'info' - 'fatal' it
> seems
> > > > reasonable to filter on level within the appender to get those events
> > to
> > > > the appropriate location.  It seemed reasonable to do something
> > similar for
> > > > 'business'.
> > > > >
> > > > > I also looked into the EventLogger but not sure that's appropriate.
> > For
> > > > one we lose the granularity to control a specific piece of code from
> > > > generating business events.  This is most likely a non-issue as I
> have
> > > > mentioned that we don't want to turn business logging off.  The other
> > is
> > > > that we lose the name of the logger as it would be the same for
> > everyone.
> > > > Not sure this is that big a deal either as I guess you might be able
> to
> > > > capture component name, though I would rather distinguish using
> logger
> > name.
> > > > >
> > > > > Thanks,
> > > > > Nick
> > > > >
> > > > >> From: ralph.go...@dslextreme.com
> > > > >> Subject: Re: approach for defining loggers
> > > > >> Date: Mon, 7 Sep 2015 20:39:11 -0700
> > > > >> To: log4j-user@logging.apache.org
> > > > >>
> > > > >> I still don’t understand why you don’t want to use Markers. They
> > were
> > > > designed exactly for the use case you are describing.
> > > > >>
> > > > >> You might set retention policies for debug vs info, error and
> fatal,
> > > > but a BUSINESS marker could cross-cut them all.  That is exactly why
> > it is
> > > > NOT a level. IOW, it gives you a second dimension for filtering. Ceki
> > > > invented Markers when he created SLF4J. For his point of view see
> > > >
> > http://stackoverflow.com/questions/16813032/what-is-
> markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them
> [http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-
> i...@2.png?v=73d79a89bded&a]<http://stackoverflow.com/
> questions/16813032/what-is-markers-in-java-logging-
> frameworks-and-that-is-a-reason-to-use-them>
>
> What is markers in Java Logging frameworks and that is a ...<
> http://stackoverflow.com/questions/16813032/what-is-
> markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>
> stackoverflow.com
> This is a rehashed version my answer to the question "Best practices for
> using Markers in SLF4J/Logback". Markers can be used to color or mark a
> single log statement.
>
>
> > > > <
> > > >
> > http://stackoverflow.com/questions/16813032/what-is-
> markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them
> [http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-
> i...@2.png?v=73d79a89bded&a]<http://stackoverflow.com/
> questions/16813032/what-is-markers-in-java-logging-
> frameworks-and-that-is-a-reason-to-use-them>
>
> What is markers in Java Logging frameworks and that is a ...<
> http://stackoverflow.com/questions/16813032/what-is-
> markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them>
> stackoverflow.com
> This is a rehashed version my answer to the question "Best practices for
> using Markers in SLF4J/Logback". Markers can be used to color or mark a
> single log statement.
>
>
> > > > >.
> > > > >>
> > > > >> Ralph
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>> On Sep 7, 2015, at 5:54 PM, Nicholas Duane <nic...@msn.com>
> wrote:
> > > > >>>
> > > > >>> If I'm attempting to control all the logging from the
> configuration
> > > > and I don't know the complete set of loggers in my application as
> there
> > > > could be 100's or 1000's, wouldn't it be hard to separate events
> based
> > on
> > > > loggers?  It would seem much easier to separate events based on
> > level.  In
> > > > addition, level might be a more reasonable approach for separating.
> > For
> > > > example, if I want to send all events to some big-data backend I
> might
> > want
> > > > to separate out traces and debug from info to fatal as traces and
> > debug are
> > > > most likely less important from a systems management aspect.  My
> > retention
> > > > period for traces and debug might be just a couple days.  The
> retention
> > > > period for info to fatal could be 30 days.  Business level might be 2
> > > > years.  Any system management notifications would probably be driven
> > off of
> > > > info to fatal events and not trace and debug events, which is another
> > > > reason you might want to separate by level.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Nick
> > > > >>>
> > > > >>>> Subject: Re: approach for defining loggers
> > > > >>>> From: ralph.go...@dslextreme.com
> > > > >>>> Date: Mon, 31 Aug 2015 08:50:58 -0700
> > > > >>>> To: log4j-user@logging.apache.org
> > > > >>>>
> > > > >>>> A logging “Level” is a level of importance. That is why there
> is a
> > > > hierarchy. If you want informational messages then you also would
> want
> > > > warnings and errors.
> > > > >>>>
> > > > >>>> “BUSINESS” does not convey the same meaning.  Rather, it is some
> > sort
> > > > of category, which is what Markers are for.
> > > > >>>>
> > > > >>>> Using the class name as the logger name is a convention. If you
> > > > really want the class name, method name or line number then you
> should
> > be
> > > > specifying that you want those from the logging event, rather than
> the
> > > > logger name.  Unless location information is disabled you always have
> > > > access to that information.
> > > > >>>>
> > > > >>>> In short, different loggers are used primarily as a way of
> > grouping
> > > > sets of messages - for example all org.hibernate events can be routed
> > to a
> > > > specific appender or turned off en masse. Levels are used to filter
> out
> > > > noise across a set of logging events. Markers are used to categorize
> > > > logging events by arbitrary attributes.
> > > > >>>>
> > > > >>>> Ralph
> > > > >>>>
> > > > >>>>
> > > > >>>>> On Aug 31, 2015, at 8:10 AM, Nicholas Duane <nic...@msn.com>
> > wrote:
> > > > >>>>>
> > > > >>>>> Thanks for the feedback.  I will look into Markers and MDC.
> > > > >>>>>
> > > > >>>>> With respect to using a separate logger, it would seem I would
> > lose
> > > > the information about what application code, eg. the class logger, is
> > > > sourcing the event.  We would like to have this information.  On top
> of
> > > > that, it seems odd, maybe to me only, that for this new level we have
> > our
> > > > own logger.  It seemed reasonable to me that this new event we want
> to
> > > > capture is just a new level.  Just like a DEBUG event is different
> > from an
> > > > INFO event.  If I define a BUSINESS level why would that not follow
> the
> > > > same design as the current levels?  You wouldn't suggest having
> > different
> > > > loggers for TRACE DEBUG INFO WARN ERROR FATAL, would you?  I think
> one
> > of
> > > > the reasons someone on our side is suggesting I have separate loggers
> > is
> > > > that they think the overhead of filtering at the appender is going to
> > have
> > > > a noticeable impact.  Our plan, at least the one I have now in my
> > head, is
> > > > that we'll have some number of appenders in the root.  We'll then
> > filter x
> > > > < INFO events to a tracing appender, INFO <= x <= FATAL to a logging
> > > > appender, and our custom level will go to another appender.
> Thoughts?
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>> Nick
> > > > >>>>>
> > > > >>>>>> Subject: Re: approach for defining loggers
> > > > >>>>>> From: ralph.go...@dslextreme.com
> > > > >>>>>> Date: Sat, 29 Aug 2015 20:59:36 -0700
> > > > >>>>>> To: log4j-user@logging.apache.org
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>> On Aug 29, 2015, at 7:44 PM, Nicholas Duane <nic...@msn.com>
> > > > wrote:
> > > > >>>>>>>
> > > > >>>>>>> I'm curious if there is a prescribed approach to defining
> > > > loggers.  Let me state what my assumption is.  I assume that normally
> > if
> > > > some piece of code wants to log events/messages that it should
> create a
> > > > logger for itself.  I guess a reasonable name to use is the class
> name
> > > > itself.  In terms of logger configuration I would expect that no
> > loggers
> > > > are specified in the log4j configuration UNLESS is needs settings
> other
> > > > than the default.  The root logger would specify the default
> settings,
> > eg.
> > > > level and appenders.  If some piece of code tied to a logger needs to
> > > > enable tracing in order to debug an issue then you would add that
> > logger to
> > > > the configuration and set the level less specific for that logger.
> Is
> > this
> > > > a typical and reasonable approach?
> > > > >>>>>>
> > > > >>>>>> What you describe here is the common convention. It is a
> > reasonable
> > > > approach.
> > > > >>>>>>
> > > > >>>>>>>
> > > > >>>>>>> I asked because we have the need for a new type of event.  To
> > have
> > > > this event flow to where we want it to flow the plan is to have a
> > custom
> > > > level and have all events at that level captured by a specific
> > appender.
> > > > My assumption was that for existing applications we'd just need to
> add
> > our
> > > > appender to the root and add our custom level.  The app would need to
> > be
> > > > modified to log our new event at the custom level.  However, someone
> > > > suggested that we could also create a separate logger for this event.
> > My
> > > > thinking is that while we don't ever want to turn off logging of this
> > > > event, loggers represent "event sources", e.g the code raising the
> > events
> > > > and thus having multiple different pieces of code use the same logger
> > > > wouldn't allow you to turn on/off logging from those different
> > sections of
> > > > code independently.  I think the current configuration includes all
> the
> > > > loggers.  Normally I would expect there to be many, on the order of
> > 10's or
> > > > 100's, loggers within an application.  However, in the case I was
> given
> > > > there were only a handful because I think this handful is shared.  So
> > as I
> > > > mentioned, this doesn't sound like an ideal design as you have less
> > > > granularity on what you can turn on/off.
> > > > >>>>>>
> > > > >>>>>> You have a few options. Using a CustomLevel would not be the
> > option
> > > > I would choose.  Creating a custom Logger will certainly work and
> makes
> > > > routing the message to the appropriate appender rather easy.  Another
> > > > approach is to use Markers.  Markers are somewhat hierarchical so you
> > can
> > > > use them for a variety of purposes.  If you look at how Log4j handles
> > event
> > > > logging it actually does both - it specifies EventLogger as the name
> > of the
> > > > logger to use and it uses Markers to identify the kind of event.
> > > > >>>>>>
> > > > >>>>>> A third option is to use the MDC or Logger properties. If you
> do
> > > > that then you can have information included in the actual logging
> event
> > > > that can affect how it is routed. I also built a system that uses the
> > > > RFC5424 format so that the event could have lots of key/value pairs
> to
> > > > identify the events.
> > > > >>>>>>
> > > > >>>>>> Unfortunately, without knowing more details I don’t know that
> I
> > can
> > > > give you a better idea on how I would implement it.
> > > > >>>>>>
> > > > >>>>>> Ralph
> > > > >>>>>>
> > > > >>>>>>
> > > > ------------------------------------------------------------
> ---------
> > > > >>>>>> To unsubscribe, e-mail:
> > log4j-user-unsubscr...@logging.apache.org
> > > > >>>>>> For additional commands, e-mail:
> > log4j-user-h...@logging.apache.org
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > ---------------------------------------------------------------------
> > > > >>>> To unsubscribe, e-mail: log4j-user-unsubscribe@
> logging.apache.org
> > > > >>>> For additional commands, e-mail:
> > log4j-user-h...@logging.apache.org
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------
> ---------
> > > > To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
> > > > For additional commands, e-mail: log4j-user-h...@logging.apache.org
> > > >
> > > >
> > >
> > >
> > > --
> > > E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
> > > Java Persistence with Hibernate, Second Edition
> > > <http://www.manning.com/bauer3/>
> > > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
> > > Spring Batch in Action <http://www.manning.com/templier/>
> > > Blog: http://garygregory.wordpress.com
> [https://s0.wp.com/i/blank.jpg]<http://garygregory.wordpress.com/>
>
> Gary Gregory<http://garygregory.wordpress.com/>
> garygregory.wordpress.com
> Software construction, the web, and other techs
>
>
> > > Home: http://garygregory.com/
> Gary Gregory<http://garygregory.com/>
> garygregory.com
> Rocket | Seagull . I am a Software Architect for Seagull Software, a
> division of Rocket Software. Rocket Seagull specializes in tools and
> expertise to modernize ...
>
>
> > > Tweet! http://twitter.com/GaryGregory
> Gary Gregory (@GaryGregory) | Twitter<http://twitter.com/GaryGregory>
> twitter.com
> The latest Tweets from Gary Gregory (@GaryGregory). Principal Software
> Engineer, author: Java Persistence Hibernate https://t.co/3F8sYxc0oq,
> JUnit https://t.co/yXU1DqAMDG, Spring Batch https://t.co/XwoMNoBxh7.
> U.S.A.
>
>
> >
> >
>
>
>
> --
> [image: MagineTV]
>
> *Mikael Ståldal*
> Senior software developer
>
> *Magine TV*
> mikael.stal...@magine.com
> Regeringsgatan 25  | 111 53 Stockholm, Sweden  |   www.magine.com<
> http://www.magine.com>
> [https://de.magine.com/content/uploads/2016/09/magine_global_social.png]<
> http://www.magine.com/>
>
> TV online with Magine TV<http://www.magine.com/>
> www.magine.com
> Watch the TV you love, on any device, anywhere in Germany and Sweden and
> find out more about our global OTT B2B solutions. Get started today.
>
>
>
> Privileged and/or Confidential Information may be contained in this
> message. If you are not the addressee indicated in this message
> (or responsible for delivery of the message to such a person), you may not
> copy or deliver this message to anyone. In such case,
> you should destroy this message and kindly notify the sender by reply
> email.
>



-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org
Java Persistence with Hibernate, Second Edition
<https://www.amazon.com/gp/product/1617290459/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1617290459&linkCode=as2&tag=garygregory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8>

<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1617290459>
JUnit in Action, Second Edition
<https://www.amazon.com/gp/product/1935182021/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182021&linkCode=as2&tag=garygregory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22>

<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182021>
Spring Batch in Action
<https://www.amazon.com/gp/product/1935182951/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182951&linkCode=%7B%7BlinkCode%7D%7D&tag=garygregory-20&linkId=%7B%7Blink_id%7D%7D%22%3ESpring+Batch+in+Action>
<http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182951>
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

Reply via email to