ppkarwasz commented on code in PR #2717:
URL: https://github.com/apache/logging-log4j2/pull/2717#discussion_r1668298843
##########
src/site/antora/modules/ROOT/pages/manual/filters.adoc:
##########
@@ -17,1038 +17,1407 @@
[id=filters]
= Filters
+:stem:
-Log4j supports filtering of log events at each level of the logging pipeline
using two features:
+Filters are Log4j plugins that evaluate the parameters of a logging call or a
log event and return one of three results:
-* the `level` attributes that can be set on loggers and appender references,
-* filter components that can be attached to loggers, appenders, appender
references or the global configuration object.
+ACCEPT:: The log event is accepted by the filter and goes to the next stage of
the logging pipeline.
-Filters evaluate the parameters of a logging call (context-wide filter) or a
log event and return one of three results:
+DENY:: The log event is unconditionally dropped.
-ACCEPT:: The log event is accepted by the filter and goes to the next stage of
the logging pipeline,
+NEUTRAL:: Log4j behaves as if the filter was not present.
-DENY:: The log event is unconditionally dropped,
+Filters can be used at each level of the
+xref:manual/architecture.adoc#architecture-diagram[logging pipeline]:
-NEUTRAL:: Log4j behaves as if the filter was not present.
+* the global configuration element can contain a
xref:manual/configuration.adoc#global-filters[global filter].
+* each xref:manual/configuration.adoc#configuring-loggers[logger]
configuration element can contain a
xref:manual/configuration.adoc#logger-elements-filters[logger filter].
+* each xref:manual/configuration.adoc#configuring-appenderrefs[appender
reference] configuration element can contain an
xref:manual/configuration.adoc#appenderrefs-elements-filters[appender reference
filter].
+* each xref:manual/appenders.adoc[appender] configuration element can contain
an xref:manual/appenders.adoc[appender filter].
+
+Additionally, the following configuration attributes take part in the
filtering process:
+
+* the xref:manual/configuration.adoc#logger-attributes-level[`level`
attribute] of logger configuration elements.
+* the xref:manual/configuration.adoc#appenderref-attributes-level[`level`
attribute] of appender reference configuration elements.
+
+[#filtering-process]
+== Filtering process
+
+Due to the interaction of many elements,
+the filtering process in Log4j is quite complex and can be divided in four
stages:
+
+. <<logger-stage,`Logger` stage>>
+. <<logger-config-stage,`LoggerConfig` stage>>
+. <<appender-control-stage,`AppenderControl` stage>>
+. <<appender-stage,`Appender` stage>>
+
+[IMPORTANT]
+====
+For performance reasons, log events should be filtered at the earliest
possible stage.
+This reduces the cost of disabled log events:
+e.g., log event creation, population of context data, formatting, transfer
through an asynchronous barrier.
+====
+
+[#logger-stage]
+=== 1. `Logger` stage
+
+[plantuml]
+....
+@startuml
+start
+group Logger
+
+:A Logger method;
+
+switch (Apply global filter)
+case (DENY)
+ #pink:Discard;
+ detach
+case (ACCEPT)
+case (NEUTRAL)
+ if (Is less severe than logger level?) then (yes)
+ #pink:Discard;
+ detach
+ else (no)
+ ' The label improves spacing
+ label a1
+ endif
+endswitch
+end group
+:Create LogEvent;
+stop
+....
+
+The parameters of the logging call are passed to the global filter.
+If the global filter returns:
+
+DENY:: The log message is immediately discarded.
+NEUTRAL:: If the level of the log message is less severe than the configured
logger threshold, the message is discarded.
+Otherwise, a
+link:../javadoc/log4j-core/org/apache/logging/log4j/core/LogEvent.html[`LogEvent`]
is created and processing continues.
+ACCEPT:: A `LogEvent` is created and processing continues in the next stage.
+
+[TIP]
+====
+Filtering logging calls at this stage provides the best performance:
+
+* this stage precedes the creation of log events, therefore operations like the
+xref:manual/thread-context.adoc[injection of context data],
+xref:manual/layouts.adoc#LocationInformation[computation of location
information]
+will not be performed for disabled log statements.
+* this stage precedes the asynchronous calls performed by either
+xref:manual/async.adoc[asynchronous loggers]
+or
+xref:manual/appenders.adoc#AsyncAppender[asynchronous appenders].
+====
+
+[#logger-config-stage]
+=== 2. `LoggerConfig` stage
+
+[plantuml]
+....
+@startuml
+start
+group LoggerConfig
+repeat
+
+:LoggerConfig.log();
+
+if (Apply logger filter) then (DENY)
+ #pink:Discard;
+ detach
+else (not DENY)
+ ' The label improves spacing
+ label a1
+endif
+repeat while (Go to parent logger?) is (yes)
+-> no;
+end group
+stop
+@enduml
+....
+
+In this stage, log events are evaluated by all the
+xref:manual/configuration.adoc#logger-elements-filters[logger filters]
+that stand on the path from the logger to an appender.
+Due to the
+xref:manual/configuration.adoc#logger-attributes-additivity[additivity of
logger configurations],
+this means that a log event must also pass the filters of all the parent
loggers,
+until it reaches the logger that references the chosen appender.
+
+[#appender-control-stage]
+=== 3. `AppenderControl` stage
+
+[plantuml]
+....
+@startuml
+!pragma useVerticalIf on
+start
+group AppenderControl
+
+:AppenderControl.callAppender();
+
+if (Is less severe then appender reference level?) then (yes)
+ #pink:Discard;
+ detach
+else (no)
+ ' The label improves spacing
+ label a2
+endif
+if (Apply appender reference filter) then (DENY)
+ #pink:Discard;
+ detach
+else (not DENY)
+ ' The label improves spacing
+ label a1
+endif
+end group
+stop
+@enduml
+....
+
+To pass this stage, log events must satisfy both conditions:
+
+* the log event must be at least as severe as the
+xref:manual/configuration.adoc#appenderref-attributes-level[`level` attribute]
+of the appender reference.
+* the xref:manual/configuration.adoc#appenderrefs-elements-filters[appender
reference filter] must return `ACCEPT` or `NEUTRAL`,
+
+[#appender-stage]
+=== 4. `Appender` stage
+
+[plantuml]
+....
+@startuml
+start
+group Appender
+
+:Appender.append();
+
+if (Apply appender filter) then (DENY)
+ #pink:Discard;
+ detach
+else (not DENY)
+ ' The label improves spacing
+ label a1
+endif
+end group
+#palegreen:Log message;
+@enduml
+....
+
+When log events reach an appender,
+the filter attached to an appender is evaluated and if the result is `DENY`,
+the log event is discarded.
+
+[NOTE]
+====
+Some appenders like the
+xref:manual/appenders.adoc#AsyncAppender[asynchronous appender]
+use appender references to transfer log events to other appenders.
+In such a case, the filtering process goes back to the
<<appender-control-stage,`AppenderControl` stage>>.
+====
+
+[TIP]
+====
+Users migrating from Log4j 1 often replace the `threshold` property of a Log4j
1 appender with a <<ThresholdFilter>> on the equivalent Log4j 2 appender.
+
+Using the `level` property of appender references will give a better
performance.
+====
+
+[WARNING]
+====
+Configuring filters is a measure of last resort,
+since it adds a large overhead to disabled log events.
+You should rather configure the filtering in one of the previous stages.
+====
-To decide whether a log event from a certain logger is delivered to a specific
appender, the following procedure is followed:
+[#example-configuration-file]
+=== Example configuration file
+
+The following example shows the order in which filters are evaluated:
[tabs]
====
XML::
+
-.Snippet from an example
{antora-examples-url}/manual/filters/filters.xml[`log4j2.xml`]
[source,xml]
----
-include::example$manual/filters/filters.xml[lines=23..41,indent=0]
+include::example$manual/filters/filters.xml[lines=1;18..-1]
----
JSON::
+
-.Snippet from an example
{antora-examples-url}/manual/filters/filters.json[`log4j2.json`]
[source,json]
----
-include::example$manual/filters/filters.json[lines=3..35,indent=0]
+include::example$manual/filters/filters.json[]
----
YAML::
+
-.Snippet from an example
{antora-examples-url}/manual/filters/filters.yaml[`log4j2.yaml`]
[source,yaml]
----
-include::example$manual/filters/filters.yaml[lines=18..-1]
+include::example$manual/filters/filters.yaml[lines=17..-1]
----
Properties::
+
-.Snippet from an example
{antora-examples-url}/manual/filters/filters.properties[`log4j2.properties`]
[source,properties]
----
include::example$manual/filters/filters.properties[lines=17..-1]
----
====
-<1> First the context-wide filter is consulted.
-If it returns `ACCEPT` the log message goes directly to point 3.
-<2> Then Log4j checks the message against the configured logger level.
-<3> The filter configured on a logger is applied next.
-If the logger is additive, the filter on the parent logger is applied
recursively until we end up on the logger that references the given appender.
-<4> Next comes the turn of the filter configured on an appender reference,
-<5> followed by a level check against the configured level of the reference.
-<6> The process ends with the filter attached to an appender.
+<1> Global filter
+<2> Logger `level` attribute (it is skipped if the event matches the global
filter)
+<3> Filter of the "org.example" logger
+<4> Filter of the root logger (it is the parent of the "org.example" logger)
+<5> Appender reference `level` attribute
+<6> Filter of the appender reference
+<7> Filter of the appender
-[WARNING]
-====
-For performance reasons, log events should be filtered as soon as possible in
the logging pipeline.
-This reduces the costs (formatting, transfer through an asynchronous barrier)
of disabled log events.
-====
+[#common-configuration]
+== Common configuration
-[TIP]
-====
-Users migrating from Log4j 1 often replace the `threshold` property of a Log4j
1 appender with a <<ThresholdFilter>> on the equivalent Log4j 2 appender.
+[#common-configuration-attributes]
+=== Common configuration attributes
-Using the `level` property of appender references will give a better
performance.
-====
+The default behavior of filters is in line with the `filter()` methods of
functional interfaces, such as
+https://docs.oracle.com/javase/{java-target-version}/docs/api/java/util/Optional.html#filter-java.util.function.Predicate-[`Optional.filter()`]
+or
+https://docs.oracle.com/javase/{java-target-version}/docs/api/java/util/stream/Stream.html#filter-java.util.function.Predicate-[`Stream.filter()`]:
+filters pass matching events to the next filter and drop those that do not
match.
+
+To allow for a larger spectrum of behaviors,
+all standard filters, except `CompositeFilter` and `DenyAllFilter`, accept the
following configuration attributes:
+
+.Common filter configuration attributes
+[cols="1m,1,1,4"]
+|===
+|Attribute |Type | Default value |Description
+
+| [[onMatch]]onMatch
+|
link:../javadoc/log4j-core/org/apache/logging/log4j/core/Filter.Result.html[`Result`]
+|
link:../javadoc/log4j-core/org/apache/logging/log4j/core/Filter.Result.html#NEUTRAL[`NEUTRAL`]
+| Result returned if the condition matches.
+
+| [[onMismatch]]onMismatch
+|
link:../javadoc/log4j-core/org/apache/logging/log4j/core/Filter.Result.html[`Result`]
+|
link:../javadoc/log4j-core/org/apache/logging/log4j/core/Filter.Result.html#DENY[`DENY`]
+| Result returned if the condition does not match.
+
+|===
+
+[#CompositeFilter]
+=== Composing filters
+
+Filters usually test for a single condition.
+To express a more complex filtering logic, Log4j provides a
+xref:plugin-reference.adoc#org-apache-logging-log4j_log4j-core_org-apache-logging-log4j-core-filter-CompositeFilter[`Filters`]
+plugin.
+This plugin can contain a sequence of filters and has no other configuration
option.
+
+The `Filters` plugin sequentially evaluates each sub-filter and:
+
+* if the sub-filter returns `ACCEPT` (resp. `DENY`), the `Filters` plugin
returns `ACCEPT` (resp. `DENY`).
+* if the sub-filter return `NEUTRAL`, the `Filters` plugin evaluates the next
sub-filter in the chain.
+* if the last sub-filter returns `NEUTRAL`, the `Filters` plugin returns
`NEUTRAL`.
+
+The `Filters` plugin together with the ternary logic of filters, can be used
to express all the boolean operators.
+In the following examples `A` and `B` are two filters.
+
+`NOT A`::
+You can invert the functionality of filter `A` by swapping the `onMatch` and
`onMismatch`:
++
+[source,xml]
+----
+<A onMatch="DENY" onMismatch="NEUTRAL"/>
+----
+
+`A AND B`::
+To select the events that match both `A` and `B` you can use:
++
+[source,xml]
+----
+<Filters>
+ <A/>
+ <B/>
+</Filters>
+----
+
+`A OR B`::
+To select the events that match `A` or `B` we can replace `NEUTRAL` with
`ACCEPT` in the `onMatch` attribute:
++
+[source,xml]
+----
+<Filters>
+ <A onMatch="ACCEPT"/>
+ <B onMatch="ACCEPT"/>
+</Filters>
+----
+
+xref:plugin-reference.adoc#org-apache-logging-log4j_log4j-core_org-apache-logging-log4j-core-filter-CompositeFilter[📖
Plugin reference for `Filters`]
[#collection]
== Collection
-Log4j bundles several predefined filters to assist in several common
deployment use cases.
-Following sections explain all these in detail.
+Log4j Core provides the following filters out-of-the-box.
+
+[#timestamp-filters]
+=== Timestamp filters
+
+Timestamp filters use the timestamp of log events to decide whether to log
them or not.
[#BurstFilter]
-=== BurstFilter
+==== `BurstFilter`
-The BurstFilter provides a mechanism to control the rate at which LogEvents
are processed by silently discarding events after the maximum limit has been
reached.
+The `BurstFilter` uses the _sliding window log_ algorithm to limit the rate of
log events.
+The rate limit is only applied to log events less severe than a configured log
level.
-.Burst Filter Parameters
-[cols="1m,1,4"]
+Besides the common configuration attributes,
+the `BurstFilter` supports the following parameters:
+
+.`BurstFilter` -- configuration attributes
+[cols="1m,1,1,4"]
|===
-|Parameter Name |Type |Description
-
-|level
-|String
-|Level of messages to be filtered. Anything at or below
-this level will be filtered out if `maxBurst` has been exceeded. The
-default is WARN meaning any messages that are higher than warn will be
-logged regardless of the size of a burst.
-
-|rate
-|float
-|The average number of events per second to allow.
-
-|maxBurst
-|integer
-|The maximum number of events that can occur before
-events are filtered for exceeding the average rate. The default is 10
-times the rate.
-
-|onMatch
-|String
-|Action to take when the filter matches. Can be ACCEPT,
-DENY or NEUTRAL. The default value is NEUTRAL.
-
-|onMismatch
-|String
-|Action to take when the filter does not match. Can
-be ACCEPT, DENY or NEUTRAL. The default value is DENY.
+|Attribute | Type | Default value | Description
+
+| level
+| link:../javadoc/log4j-api/org/apache/logging/log4j/Level.html[`Level`]
+| link:../javadoc/log4j-api/org/apache/logging/log4j/Level.html#WARN[`WARN`]
+| The rate limit only applies to log events less severe than this level.
+Events at least as severe as this level will always match.
+
+| rate
+| `float`
+| `10`
+| The average number of events per second to allow.
+
+| maxBurst
+| `long`
+| `10 × rate`
+| The maximum number of events in a sliding window.
+
|===
-A configuration containing the BurstFilter might look like:
+The sliding window can be easily computed as:
Review Comment:
Some users (cf. [this SO
question](https://stackoverflow.com/q/72000886/11748454)) want to have more
control on the frequency used to log events.
In the case of the `SMTP` appender it really makes sense to have 1 e-mail
per hour instead of 24 e-mail at once and then silence.
Fixed in 2cde19bec61964536b33d11635712c9ae8d21fd8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
