Hello log4j users/developers,
I have created a new appender called *EventConsolidatingAppender*. The
purpose of this appender is to consolidate multiple events that are received
by a single logger within a specified number of seconds into a single event;
this single consolidated event is then forwarded to a "downstream" appender.
EventConsolidatingAppender is probably most useful when the downstream
appender is SMTPAppender. In this case, you can use
EventConsolidatingAppender so that if, say, 200 errors are raised to a given
logger within 30 seconds, instead of receiving 200 emails, you'll receive *
one* email that contains the messages from all 200 errors.
Example use case: Our system makes high-volume, real-time requests to a data
provider's web services. When we have a problem contacting a provider's
service, we'll raise a warning or error event to a logger. When a data
provider's service goes down for a several minutes, we may make multiple
hundreds of requests to that service that will all fail. We like using
SMTPAppender so that we receive immediate notification of any warnings and
errors via email, but if a data provider is down and we're sending an email
per error, our email system may get overwhelmed and/or disabled by our
hosting provider (Gmail). Using EventConsolidatingAppender with SMTPAppender
as the downstream appender solves this problem.
This problem probably could also have been solved via the configuration of
more sophisticated operations/monitoring software (Nagios, Zenoss, etc.), or
via the delivery of events to a JMS queue, but in our case it wasn't
desirable to install, configure and maintain something so heavyweight.
This appender is thread-safe and implements internal "per-logger"
synchronization (since it performs "per-logger" caching).
Sample configuration (includes SMTPAppender configuration just for
completeness):
log4j.appender.ConsolidatingAppender=
org.mattklein.log4j.EventConsolidatingAppender
log4j.appender.ConsolidatingAppender.threshold=WARN
log4j.appender.ConsolidatingAppender.downstreamAppender=EmailAppender
log4j.appender.ConsolidatingAppender.delaySecs=30
log4j.appender.EmailAppender=org.apache.log4j.net.SMTPAppender
log4j.appender.EmailAppender.threshold=WARN
log4j.appender.EmailAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.EmailAppender.layout.ConversionPattern=[%d{ISO8601}] [%t]
[%c] [%l] %n%p -%m%n
log4j.appender.EmailAppender.BufferSize=1
log4j.appender.EmailAppender.SMTPHost=smtp.gmail.com
log4j.appender.EmailAppender.SMTPUsername=o...@mattklein.org
log4j.appender.EmailAppender.SMTPPassword=password
log4j.appender.EmailAppender.From=Ops <o...@mattklein.org>
log4j.appender.EmailAppender.To=o...@mattklein.org
log4j.appender.EmailAppender.Subject=[ALERT] Warning or error raised
# Because we want the SMTPAppender to send emails on WARNINGs as well as
ERRORs, we need to provide a custom evaluatorClass
log4j.appender.EmailAppender.evaluatorClass=
org.mattklein.log4j.MyTriggeringEventEvaluator
Sample "consolidated" event:
[2011-02-21 10:55:15,901]
ERROR - The following 6 events were consolidated since they occurred within
30 seconds of each other
Event 1:
[2011-02-21 10:54:45,896]
WARN - Warn message
Event 2:
[2011-02-21 10:54:45,898]
ERROR - Error message
Event 3:
[2011-02-21 10:54:57,903]
WARN - Warn message
Event 4:
[2011-02-21 10:54:57,903]
ERROR - Error message
Event 5:
[2011-02-21 10:55:09,905]
WARN - Warn message
Event 6:
[2011-02-21 10:55:09,905]
ERROR - Error message
The priority of the consolidated event is set to the highest priority of any
of the constituent events (i.e., WARN and ERROR events consolidate into an
ERROR event). If only a single event is raised within the delaySecs
parameter, it will be delivered unmodified to the downstream appender.
Note that the delivery of events to your downstream appender will be delayed
by up to delaySecs seconds. So, if only a single event is raised and
delaySecs is 30, the email for it will be sent with a 30 second lag from the
time the event was raised (as the EventConsolidatingAppender waits 30
seconds to see if any other events are raised). We consider this delay to be
a fine tradeoff though.
If anyone else is interested in this code I'm happy to provide it. Feel free
to contact me at mpkl...@gmail.com.
Matt Klein
