On Mon, Oct 17, 2016 at 4:27 PM, Nicholas Duane <nic...@msn.com> wrote:
> Sorry to revive this old thread. However, we're in the process of adding > support for other "categories" of events and thus I wanted to first take a > step back and try to ensure we're not convoluting things. > > > There was a requirement to log a "compliance" event under certain > conditions. We did not want to write our own logging framework and instead > decided to use existing off-the-shelf logging frameworks. We have > applications on both Linux/Java, Windows/.NET and Windows/Java. Initially > we chose log4net for Windows/.NET and log4j2 for Windows/Java and > Linux/Java. For these logging frameworks we wrote artifacts, appenders > basically, to help facilitate getting these events to our system. By the > way, our system will get the events centrally, possibly put them into a > relational database and also hand them off to another system which will get > them eventually to an HDFS backend. We also exposed methods for creating > this compliance event. The compliance event is basically a map. We chose > a map so that the event could also be extended by the application team in > case they needed to add additional properties which made sense for them. > > > We chose to create a custom level for this "compliance" event such that we > could filter out only these events and get them into our system. The > configuration example we created had our custom unix domain socket appender > added to the root logger. It also contained a filter which filtered out > any events that weren't compliance events. The level we chose for > "compliance" was less critical than off and more critical than fatal as we > wanted to ensure that as long as logging wasn't turned off altogether our > events would get logged. > > > I want to go over a few suggestions that were made and explain why I > didn't make use of those suggestions. > > > 1. Since our "compliance" level didn't fit within the "vernacular" of the > existing levels we should not define this custom level. Instead we should > look at using markers. > Yes, this is a use case for markers. The level should be used to note how important is each compliance event. > > I am not that familiar with markers but did look into them when they were > suggested. While I don't have anything against markers in general there > were some downsides as I saw it. > > > a. Markers are not available, as far as I know, in log4net so we'd still > have to figure out something there. > Indeed, we really need a port of Log4j 2 to .NET. > > b. A bigger problem, at least I thought it was a bigger problem, was that > there would be confusion about what level to log the event at. I would > certainly not want to give an example as follows: > > > logger.debug(COMPLIANCE_MARKER, evnt); > > > or > > > logger.info(COMPLIANCE_MARKER, evnt); > > > or > > > logger.error(COMPLIANCE_MARKER, evnt); > > > ... > Think about: How important is this event? Are there different level of importance to the audience? > > > That just screams confusion to me. > > > 2. Use a dedicated logger to log all compliance events. > > > There were, as far as I could tell, a couple problems with this approach > also. > > > a. If everyone is using a single "well known" logger to log a specific > event category then I lose the logger "context" of who's logging the > event. As it stands now we're copying the logger name into a property we > call "eventSource". > A practice is to use one logger per class. Another is to use a higher-level logger to represent higher-level abstractions like a module. > > > b. You cannot turn on/off logging for a specific set of code. If it turns > out that we have some errant code which is using this well known logger > then we can't just turn off that code from logging as turning off the well > know logger would turn it off for everyone using it. > > > I did look into the EventLogger and initially that seemed promising as I > guess it logs any event you give it at the "all" level. However, as a well > known logger it suffers from the same issues above. > > > Now we're looking to add Business events. My initial thinking is that I > can do the same thing we did before. Add an additional custom level called > "Business" and expose methods for creating a business event. I would NOT create a custom level. Instead, I would use a Logger called "Business". > Though unlike the compliance event, the application teams would be > defining the schema more so than our framework team. Thus any method we > expose would just be used as a starting point for setting the common > properties. You would use another instance of our unix domain socket > appender for these business events and forward them to a different location > as business events would most likely have a different retention period than > compliance events. Plus you might also want them in a different store as > you may never need to query for both categories of events and thus no need > to query against a larger set of data. > > > In addition we're planning to capture centrally what we refer to as > diagnostic events: error, info, warn, debug, trace, etc. However, we may > need to separate these out into two different categories: > critical-diagnostic and noncritical-diagnostic. This could be a user case for custom levels IF one is more important than the other which it sure sounds like it is. > The reason is that we don't want the potential of a critical diagnostic > event, let's say an error, queued up behind potentially thousands of > non-critical diagnostic events. So you see, the category also defines > aspects on how we handle events at the source. We separate at the source > based on category as it seems a reasonable place to do so. Also, you may > want different flush times for different categories. We have a process > which buffers, compresses and sends events centrally so we have the notion > of flush time. The buffers are flushed when they become full or the flush > time elapses. Errors, since they are more critical in monitoring systems, > we'll most likely want to flush more often than say debug and trace events. > > > Sorry for the long winded explanation. Initially I was thinking that when > we create an event we'd set its category. However, now I'm thinking the > category should be set by the act of logging the event at a level. In some > cases we have a 1:1 mapping from level to category, eg. compliance level -> > compliance category. In some cases we have a many:1 mapping from level to > category, eg. error, info, warn -> critical-diagnostic. > > > We could also just define a single custom level, say "always_on", or > something like that. Then we provide some helper method to log our "new" > event categories (eg. business and compliance) at this level and have the > user specify the category, I guess similar to a marker. > Log4j has a level called ALL. I would really try to work hard to stay within the feature set before thinking about anything custom. If you can make critical-diagnostic and noncritical-diagnostic events to stock levels, that much the better. Gary > > > logEvent(Logger logger, String category, object evnt); > > > I guess it's similar to the EventLogger except that we're not using a > single well known logger and thus don't have the downsides of that which I > pointed out earlier. > > > Any thoughts/suggestions would be appreciated. > > > Thanks, > > Nick > > ________________________________ > From: Mikael Ståldal <mikael.stal...@magine.com> > Sent: Wednesday, September 9, 2015 3:47 AM > To: Log4J Users List > Subject: Re: approach for defining loggers > > Then perhaps you should create your own facade for doing business event > logging, which could then forward them to Log4j in an appropriate way. > > On Wed, Sep 9, 2015 at 4:49 AM, Nicholas Duane <nic...@msn.com> wrote: > > > I was just about to reply to your previous email about using a single > > "business" logger, or some hierarchy of business loggers, to log business > > events and say that we might go that route. However, now that you > brought > > up the post from Ralph, which I just replied to, I'm thinking a logger > > won't work either for the same reason I listed in my reply to Ralph's > post. > > > > You could do: > > > > logger.info("Hello"); > > logger.fatal("Hello"); > > logger.error("Hello"); > > ... > > > > It's confusing as there are n ways to log a business event that way and > > they will all do the same thing. Which one should a developer choose. > > Should I say pick any one, it doesn't matter? > > > > Thanks, > > Nick > > > > > Date: Tue, 8 Sep 2015 19:28:21 -0700 > > > Subject: Re: approach for defining loggers > > > From: garydgreg...@gmail.com > > > To: log4j-user@logging.apache.org > > > > > > Or > > > Logger logger = LogManager.getLogger("Business"); > > > ... > > > logger.info("Hello"); > > > > > > Gary > > > > > > On Tue, Sep 8, 2015 at 7:24 PM, Ralph Goers < > ralph.go...@dslextreme.com> > > > wrote: > > > > > > > Can you please clarify, “If we had some way to know an event is a > > business > > > > event we wouldn’t need level”? I do not understand how you can code > > > > logger.log(BUSINESS, msg) but you cannot code logger.info(BUSINESS, > > msg). > > > > > > > > Ralph > > > > > > > > > On Sep 8, 2015, at 6:09 PM, Nicholas Duane <nic...@msn.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > I looked over that stackoverflow post and I'm still not seeing a > good > > > > match as a way for us to log our business events. > > > > > > > > > > A business event I guess is an event which extends whatever schema > we > > > > come up with for a business event. While an instance of this schema > > could > > > > be logged at any level, that really doesn't make sense in our > scenario, > > > > regardless of whether some marker was supplied. If we had some way > to > > know > > > > an event is a business event we wouldn't need level. We could of > > course > > > > add some property to our schema which indicates the 'category' of the > > > > event, 'business' being one such category. Instead we were thinking > we > > > > could just use level to indicate that an event is a business event. > > > > > > > > > > As I mentioned, we're looking to capture 'trace' level events to > one > > > > store, 'info' - 'fatal' level events to another store, and 'business' > > > > events to yet another store. For 'trace' and 'info' - 'fatal' it > seems > > > > reasonable to filter on level within the appender to get those events > > to > > > > the appropriate location. It seemed reasonable to do something > > similar for > > > > 'business'. > > > > > > > > > > I also looked into the EventLogger but not sure that's appropriate. > > For > > > > one we lose the granularity to control a specific piece of code from > > > > generating business events. This is most likely a non-issue as I > have > > > > mentioned that we don't want to turn business logging off. The other > > is > > > > that we lose the name of the logger as it would be the same for > > everyone. > > > > Not sure this is that big a deal either as I guess you might be able > to > > > > capture component name, though I would rather distinguish using > logger > > name. > > > > > > > > > > Thanks, > > > > > Nick > > > > > > > > > >> From: ralph.go...@dslextreme.com > > > > >> Subject: Re: approach for defining loggers > > > > >> Date: Mon, 7 Sep 2015 20:39:11 -0700 > > > > >> To: log4j-user@logging.apache.org > > > > >> > > > > >> I still don’t understand why you don’t want to use Markers. They > > were > > > > designed exactly for the use case you are describing. > > > > >> > > > > >> You might set retention policies for debug vs info, error and > fatal, > > > > but a BUSINESS marker could cross-cut them all. That is exactly why > > it is > > > > NOT a level. IOW, it gives you a second dimension for filtering. Ceki > > > > invented Markers when he created SLF4J. For his point of view see > > > > > > http://stackoverflow.com/questions/16813032/what-is- > markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them > [http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch- > i...@2.png?v=73d79a89bded&a]<http://stackoverflow.com/ > questions/16813032/what-is-markers-in-java-logging- > frameworks-and-that-is-a-reason-to-use-them> > > What is markers in Java Logging frameworks and that is a ...< > http://stackoverflow.com/questions/16813032/what-is- > markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them> > stackoverflow.com > This is a rehashed version my answer to the question "Best practices for > using Markers in SLF4J/Logback". Markers can be used to color or mark a > single log statement. > > > > > > < > > > > > > http://stackoverflow.com/questions/16813032/what-is- > markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them > [http://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch- > i...@2.png?v=73d79a89bded&a]<http://stackoverflow.com/ > questions/16813032/what-is-markers-in-java-logging- > frameworks-and-that-is-a-reason-to-use-them> > > What is markers in Java Logging frameworks and that is a ...< > http://stackoverflow.com/questions/16813032/what-is- > markers-in-java-logging-frameworks-and-that-is-a-reason-to-use-them> > stackoverflow.com > This is a rehashed version my answer to the question "Best practices for > using Markers in SLF4J/Logback". Markers can be used to color or mark a > single log statement. > > > > > > >. > > > > >> > > > > >> Ralph > > > > >> > > > > >> > > > > >> > > > > >> > > > > >>> On Sep 7, 2015, at 5:54 PM, Nicholas Duane <nic...@msn.com> > wrote: > > > > >>> > > > > >>> If I'm attempting to control all the logging from the > configuration > > > > and I don't know the complete set of loggers in my application as > there > > > > could be 100's or 1000's, wouldn't it be hard to separate events > based > > on > > > > loggers? It would seem much easier to separate events based on > > level. In > > > > addition, level might be a more reasonable approach for separating. > > For > > > > example, if I want to send all events to some big-data backend I > might > > want > > > > to separate out traces and debug from info to fatal as traces and > > debug are > > > > most likely less important from a systems management aspect. My > > retention > > > > period for traces and debug might be just a couple days. The > retention > > > > period for info to fatal could be 30 days. Business level might be 2 > > > > years. Any system management notifications would probably be driven > > off of > > > > info to fatal events and not trace and debug events, which is another > > > > reason you might want to separate by level. > > > > >>> > > > > >>> Thanks, > > > > >>> Nick > > > > >>> > > > > >>>> Subject: Re: approach for defining loggers > > > > >>>> From: ralph.go...@dslextreme.com > > > > >>>> Date: Mon, 31 Aug 2015 08:50:58 -0700 > > > > >>>> To: log4j-user@logging.apache.org > > > > >>>> > > > > >>>> A logging “Level” is a level of importance. That is why there > is a > > > > hierarchy. If you want informational messages then you also would > want > > > > warnings and errors. > > > > >>>> > > > > >>>> “BUSINESS” does not convey the same meaning. Rather, it is some > > sort > > > > of category, which is what Markers are for. > > > > >>>> > > > > >>>> Using the class name as the logger name is a convention. If you > > > > really want the class name, method name or line number then you > should > > be > > > > specifying that you want those from the logging event, rather than > the > > > > logger name. Unless location information is disabled you always have > > > > access to that information. > > > > >>>> > > > > >>>> In short, different loggers are used primarily as a way of > > grouping > > > > sets of messages - for example all org.hibernate events can be routed > > to a > > > > specific appender or turned off en masse. Levels are used to filter > out > > > > noise across a set of logging events. Markers are used to categorize > > > > logging events by arbitrary attributes. > > > > >>>> > > > > >>>> Ralph > > > > >>>> > > > > >>>> > > > > >>>>> On Aug 31, 2015, at 8:10 AM, Nicholas Duane <nic...@msn.com> > > wrote: > > > > >>>>> > > > > >>>>> Thanks for the feedback. I will look into Markers and MDC. > > > > >>>>> > > > > >>>>> With respect to using a separate logger, it would seem I would > > lose > > > > the information about what application code, eg. the class logger, is > > > > sourcing the event. We would like to have this information. On top > of > > > > that, it seems odd, maybe to me only, that for this new level we have > > our > > > > own logger. It seemed reasonable to me that this new event we want > to > > > > capture is just a new level. Just like a DEBUG event is different > > from an > > > > INFO event. If I define a BUSINESS level why would that not follow > the > > > > same design as the current levels? You wouldn't suggest having > > different > > > > loggers for TRACE DEBUG INFO WARN ERROR FATAL, would you? I think > one > > of > > > > the reasons someone on our side is suggesting I have separate loggers > > is > > > > that they think the overhead of filtering at the appender is going to > > have > > > > a noticeable impact. Our plan, at least the one I have now in my > > head, is > > > > that we'll have some number of appenders in the root. We'll then > > filter x > > > > < INFO events to a tracing appender, INFO <= x <= FATAL to a logging > > > > appender, and our custom level will go to another appender. > Thoughts? > > > > >>>>> > > > > >>>>> Thanks, > > > > >>>>> Nick > > > > >>>>> > > > > >>>>>> Subject: Re: approach for defining loggers > > > > >>>>>> From: ralph.go...@dslextreme.com > > > > >>>>>> Date: Sat, 29 Aug 2015 20:59:36 -0700 > > > > >>>>>> To: log4j-user@logging.apache.org > > > > >>>>>> > > > > >>>>>> > > > > >>>>>>> On Aug 29, 2015, at 7:44 PM, Nicholas Duane <nic...@msn.com> > > > > wrote: > > > > >>>>>>> > > > > >>>>>>> I'm curious if there is a prescribed approach to defining > > > > loggers. Let me state what my assumption is. I assume that normally > > if > > > > some piece of code wants to log events/messages that it should > create a > > > > logger for itself. I guess a reasonable name to use is the class > name > > > > itself. In terms of logger configuration I would expect that no > > loggers > > > > are specified in the log4j configuration UNLESS is needs settings > other > > > > than the default. The root logger would specify the default > settings, > > eg. > > > > level and appenders. If some piece of code tied to a logger needs to > > > > enable tracing in order to debug an issue then you would add that > > logger to > > > > the configuration and set the level less specific for that logger. > Is > > this > > > > a typical and reasonable approach? > > > > >>>>>> > > > > >>>>>> What you describe here is the common convention. It is a > > reasonable > > > > approach. > > > > >>>>>> > > > > >>>>>>> > > > > >>>>>>> I asked because we have the need for a new type of event. To > > have > > > > this event flow to where we want it to flow the plan is to have a > > custom > > > > level and have all events at that level captured by a specific > > appender. > > > > My assumption was that for existing applications we'd just need to > add > > our > > > > appender to the root and add our custom level. The app would need to > > be > > > > modified to log our new event at the custom level. However, someone > > > > suggested that we could also create a separate logger for this event. > > My > > > > thinking is that while we don't ever want to turn off logging of this > > > > event, loggers represent "event sources", e.g the code raising the > > events > > > > and thus having multiple different pieces of code use the same logger > > > > wouldn't allow you to turn on/off logging from those different > > sections of > > > > code independently. I think the current configuration includes all > the > > > > loggers. Normally I would expect there to be many, on the order of > > 10's or > > > > 100's, loggers within an application. However, in the case I was > given > > > > there were only a handful because I think this handful is shared. So > > as I > > > > mentioned, this doesn't sound like an ideal design as you have less > > > > granularity on what you can turn on/off. > > > > >>>>>> > > > > >>>>>> You have a few options. Using a CustomLevel would not be the > > option > > > > I would choose. Creating a custom Logger will certainly work and > makes > > > > routing the message to the appropriate appender rather easy. Another > > > > approach is to use Markers. Markers are somewhat hierarchical so you > > can > > > > use them for a variety of purposes. If you look at how Log4j handles > > event > > > > logging it actually does both - it specifies EventLogger as the name > > of the > > > > logger to use and it uses Markers to identify the kind of event. > > > > >>>>>> > > > > >>>>>> A third option is to use the MDC or Logger properties. If you > do > > > > that then you can have information included in the actual logging > event > > > > that can affect how it is routed. I also built a system that uses the > > > > RFC5424 format so that the event could have lots of key/value pairs > to > > > > identify the events. > > > > >>>>>> > > > > >>>>>> Unfortunately, without knowing more details I don’t know that > I > > can > > > > give you a better idea on how I would implement it. > > > > >>>>>> > > > > >>>>>> Ralph > > > > >>>>>> > > > > >>>>>> > > > > ------------------------------------------------------------ > --------- > > > > >>>>>> To unsubscribe, e-mail: > > log4j-user-unsubscr...@logging.apache.org > > > > >>>>>> For additional commands, e-mail: > > log4j-user-h...@logging.apache.org > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > --------------------------------------------------------------------- > > > > >>>> To unsubscribe, e-mail: log4j-user-unsubscribe@ > logging.apache.org > > > > >>>> For additional commands, e-mail: > > log4j-user-h...@logging.apache.org > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------ > --------- > > > > To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org > > > > For additional commands, e-mail: log4j-user-h...@logging.apache.org > > > > > > > > > > > > > > > > > -- > > > E-Mail: garydgreg...@gmail.com | ggreg...@apache.org > > > Java Persistence with Hibernate, Second Edition > > > <http://www.manning.com/bauer3/> > > > JUnit in Action, Second Edition <http://www.manning.com/tahchiev/> > > > Spring Batch in Action <http://www.manning.com/templier/> > > > Blog: http://garygregory.wordpress.com > [https://s0.wp.com/i/blank.jpg]<http://garygregory.wordpress.com/> > > Gary Gregory<http://garygregory.wordpress.com/> > garygregory.wordpress.com > Software construction, the web, and other techs > > > > > Home: http://garygregory.com/ > Gary Gregory<http://garygregory.com/> > garygregory.com > Rocket | Seagull . I am a Software Architect for Seagull Software, a > division of Rocket Software. Rocket Seagull specializes in tools and > expertise to modernize ... > > > > > Tweet! http://twitter.com/GaryGregory > Gary Gregory (@GaryGregory) | Twitter<http://twitter.com/GaryGregory> > twitter.com > The latest Tweets from Gary Gregory (@GaryGregory). Principal Software > Engineer, author: Java Persistence Hibernate https://t.co/3F8sYxc0oq, > JUnit https://t.co/yXU1DqAMDG, Spring Batch https://t.co/XwoMNoBxh7. > U.S.A. > > > > > > > > > > -- > [image: MagineTV] > > *Mikael Ståldal* > Senior software developer > > *Magine TV* > mikael.stal...@magine.com > Regeringsgatan 25 | 111 53 Stockholm, Sweden | www.magine.com< > http://www.magine.com> > [https://de.magine.com/content/uploads/2016/09/magine_global_social.png]< > http://www.magine.com/> > > TV online with Magine TV<http://www.magine.com/> > www.magine.com > Watch the TV you love, on any device, anywhere in Germany and Sweden and > find out more about our global OTT B2B solutions. Get started today. > > > > Privileged and/or Confidential Information may be contained in this > message. If you are not the addressee indicated in this message > (or responsible for delivery of the message to such a person), you may not > copy or deliver this message to anyone. In such case, > you should destroy this message and kindly notify the sender by reply > email. > -- E-Mail: garydgreg...@gmail.com | ggreg...@apache.org Java Persistence with Hibernate, Second Edition <https://www.amazon.com/gp/product/1617290459/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1617290459&linkCode=as2&tag=garygregory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1617290459> JUnit in Action, Second Edition <https://www.amazon.com/gp/product/1935182021/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182021&linkCode=as2&tag=garygregory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182021> Spring Batch in Action <https://www.amazon.com/gp/product/1935182951/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1935182951&linkCode=%7B%7BlinkCode%7D%7D&tag=garygregory-20&linkId=%7B%7Blink_id%7D%7D%22%3ESpring+Batch+in+Action> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=1935182951> Blog: http://garygregory.wordpress.com Home: http://garygregory.com/ Tweet! http://twitter.com/GaryGregory