Re: Monitoring discards from async logging

[Thomas, Adam]

This setup should work for us. While we don’t control logging initialization, 
we will reliably be present on the classpath at initialization time, and 
ServiceLoader should have no problems finding our implementation.

> You probably want to implement a "push" counter for the discarded messages, 
> so you get notified immediately whenever a discard event occurs?

For my use-case, no, periodic reporting is fine, but I'm happy to do the 
batching on my end.

-Adam

On 4/12/24, 2:50 PM, "Piotr P. Karwasz" mailto:piotr.karw...@gmail.com>> wrote:

Hi Adam,


On Fri, 12 Apr 2024 at 21:42, Thomas, Adam mailto:adamt...@amazon.com.inva>lid> wrote:
> That sounds appealing, but do you have to have this hook registered by the 
> time the disruptor is set up? That's my fundamental concern with a lot of 
> these interception strategies - if your library/framework is called (which 
> ours is), instead of being the entry point (as many others often are), 
> logging will most likely already be initialized by the time control is handed 
> to you. By the time my code gets a chance to inspect the LoggerContext, log4j 
> has already decided which policy to use, instantiated it, and installed it.


My draft proposal[1] uses `ServiceLoader` to load all available
`InstrumentationService` implementations, so it happens before the
creation of the first `Logger` and the `AsyncLoggerDisruptor`.
We could obviously develop a system to allow late registrations of
`InstrumentationService`. Most of the components I expose in the PR
are tied to a `Configuration` so triggering a reconfiguration could
allow your code to instrument most of them.


Unfortunately the list does not include the `MessageFactory` used by
already created loggers or the disruptor attached to
`AsyncLoggerContext` (but it includes the disruptor used by
`AsyncLoggerConfig`s). You probably want to implement a "push" counter
for the discarded messages, so you get notified immediately whenever a
discard event occurs?


Feel free to comment on the PR or branch the code to correct it.


Piotr


[1] https://github.com/apache/logging-log4j2/pull/2469 

Re: Monitoring discards from async logging

[Piotr P. Karwasz]

Hi Adam,

On Fri, 12 Apr 2024 at 21:42, Thomas, Adam  wrote:
> That sounds appealing, but do you have to have this hook registered by the 
> time the disruptor is set up? That's my fundamental concern with a lot of 
> these interception strategies - if your library/framework is called (which 
> ours is), instead of being the entry point (as many others often are), 
> logging will most likely already be initialized by the time control is handed 
> to you. By the time my code gets a chance to inspect the LoggerContext, log4j 
> has already decided which policy to use, instantiated it, and installed it.

My draft proposal[1] uses `ServiceLoader` to load all available
`InstrumentationService` implementations, so it happens before the
creation of the first `Logger` and the `AsyncLoggerDisruptor`.
We could obviously develop a system to allow late registrations of
`InstrumentationService`. Most of the components I expose in the PR
are tied to a `Configuration` so triggering a reconfiguration could
allow your code to instrument most of them.

Unfortunately the list does not include the `MessageFactory` used by
already created loggers or the disruptor attached to
`AsyncLoggerContext` (but it includes the disruptor used by
`AsyncLoggerConfig`s). You probably want to implement a "push" counter
for the discarded messages, so you get notified immediately whenever a
discard event occurs?

Feel free to comment on the PR or branch the code to correct it.

Piotr

[1] https://github.com/apache/logging-log4j2/pull/2469

Re: Monitoring discards from async logging

[Thomas, Adam]

Matt,

That sounds appealing, but do you have to have this hook registered by the time 
the disruptor is set up? That's my fundamental concern with a lot of these 
interception strategies - if your library/framework is called (which ours is), 
instead of being the entry point (as many others often are), logging will most 
likely already be initialized by the time control is handed to you. By the time 
my code gets a chance to inspect the LoggerContext, log4j has already decided 
which policy to use, instantiated it, and installed it.

-Adam

On 4/12/24, 11:49 AM, "Matt Sicker" mailto:m...@musigma.org>> wrote:

We’ve got a new proposal from Piotr on this front now which sort of emulates 
his suggestion for 3.x but applied to 2.x (since that doesn’t have the DI 
system). I like the approach there as it provides enough of a hook for users to 
customize whatever metrics library they wish to enhance some components however 
they see fit (similar to Spring’s BeanPostProcessor functionality).

> On Apr 12, 2024, at 06:33, Piotr P. Karwasz  > wrote:
>
> Hi Volkan,
>
> On Thu, 11 Apr 2024 at 21:26, Volkan Yazıcı  > wrote:
>> *Slightly longer answer:* Log4j has never had a holistic observability view
>> to its internals. We need something new (if not, from scratch), and
>> preferably, applicable to not only async. logging, but all components
>> wherever this can be needed. This is time consuming hard work.
>> Piotr and I, sort of, the only active Log4j maintainers nowadays, are
>> swamped with other priorities. All that said, you can contract a maintainer
>>  
>> ;> to get 
>> this
>> rolling.
>
> I think you meant "you can contribute the functionality or contract
> **any** developer to do it". ;-) The way you wrote it sounds horrible.
>
> Adam, to be entirely clear:
>
> * Volkan and I are **not** the only maintainers. If a majority of
> maintainers support your PR, personally I will not block it, although
> I prefer to keep implementation details encapsulated as much as
> possible, so we can refactor them at any time,
>
> * We would like to have a holistic solution to instrumenting Log4j
> Core. Even though Volkan and I undeniably work a lot on Log4j as part
> of our job, adding metrics to Log4j is not covered by our clients. So
> we cannot afford to deliver it ourselves in our spare time,
>
> * Volkan probably meant that a Log4j maintainer could deliver this
> work faster, but we really don't care who contributes it. If you can
> contribute such a PR, we would be happy to review it. However before
> starting such a major endeavour you should either discuss the details
> on this mailing list or you can contact us directly and we can
> organize a video call with the rest of the PMC,
>
> * Let me stress that we treat all PRs equally, whether coming from a
> project founder, PMC member or external contributors. My
> co-maintainers could probably tell you more about it. ;-)
>
> Piotr

Re: Monitoring discards from async logging

[Matt Sicker]

We’ve got a new proposal from Piotr on this front now which sort of emulates 
his suggestion for 3.x but applied to 2.x (since that doesn’t have the DI 
system). I like the approach there as it provides enough of a hook for users to 
customize whatever metrics library they wish to enhance some components however 
they see fit (similar to Spring’s BeanPostProcessor functionality).

> On Apr 12, 2024, at 06:33, Piotr P. Karwasz  wrote:
> 
> Hi Volkan,
> 
> On Thu, 11 Apr 2024 at 21:26, Volkan Yazıcı  wrote:
>> *Slightly longer answer:* Log4j has never had a holistic observability view
>> to its internals. We need something new (if not, from scratch), and
>> preferably, applicable to not only async. logging, but all components
>> wherever this can be needed. This is time consuming hard work.
>> Piotr and I, sort of, the only active Log4j maintainers nowadays, are
>> swamped with other priorities. All that said, you can contract a maintainer
>>  to get this
>> rolling.
> 
> I think you meant "you can contribute the functionality or contract
> **any** developer to do it". ;-) The way you wrote it sounds horrible.
> 
> Adam, to be entirely clear:
> 
> * Volkan and I are **not** the only maintainers. If a majority of
> maintainers support your PR, personally I will not block it, although
> I prefer to keep implementation details encapsulated as much as
> possible, so we can refactor them at any time,
> 
> * We would like to have a holistic solution to instrumenting Log4j
> Core. Even though Volkan and I undeniably work a lot on Log4j as part
> of our job, adding metrics to Log4j is not covered by our clients. So
> we cannot afford to deliver it ourselves in our spare time,
> 
> * Volkan probably meant that a Log4j maintainer could deliver this
> work faster, but we really don't care who contributes it. If you can
> contribute such a PR, we would be happy to review it. However before
> starting such a major endeavour you should either discuss the details
> on this mailing list or you can contact us directly and we can
> organize a video call with the rest of the PMC,
> 
> * Let me stress that we treat all PRs equally, whether coming from a
> project founder, PMC member or external contributors. My
> co-maintainers could probably tell you more about it. ;-)
> 
> Piotr

Re: Monitoring discards from async logging

[Piotr P. Karwasz]

Hi Volkan,

On Thu, 11 Apr 2024 at 21:26, Volkan Yazıcı  wrote:
> *Slightly longer answer:* Log4j has never had a holistic observability view
> to its internals. We need something new (if not, from scratch), and
> preferably, applicable to not only async. logging, but all components
> wherever this can be needed. This is time consuming hard work.
> Piotr and I, sort of, the only active Log4j maintainers nowadays, are
> swamped with other priorities. All that said, you can contract a maintainer
>  to get this
> rolling.

I think you meant "you can contribute the functionality or contract
**any** developer to do it". ;-) The way you wrote it sounds horrible.

Adam, to be entirely clear:

* Volkan and I are **not** the only maintainers. If a majority of
maintainers support your PR, personally I will not block it, although
I prefer to keep implementation details encapsulated as much as
possible, so we can refactor them at any time,

* We would like to have a holistic solution to instrumenting Log4j
Core. Even though Volkan and I undeniably work a lot on Log4j as part
of our job, adding metrics to Log4j is not covered by our clients. So
we cannot afford to deliver it ourselves in our spare time,

* Volkan probably meant that a Log4j maintainer could deliver this
work faster, but we really don't care who contributes it. If you can
contribute such a PR, we would be happy to review it. However before
starting such a major endeavour you should either discuss the details
on this mailing list or you can contact us directly and we can
organize a video call with the rest of the PMC,

* Let me stress that we treat all PRs equally, whether coming from a
project founder, PMC member or external contributors. My
co-maintainers could probably tell you more about it. ;-)

Piotr

Re: Monitoring discards from async logging

[Volkan Yazıcı]

*Short answer:* I would suggest using reflection for the time being. We
don't anticipate any changes in that part of the code base in the
foreseeable future.

*Slightly longer answer:* Log4j has never had a holistic observability view
to its internals. We need something new (if not, from scratch), and
preferably, applicable to not only async. logging, but all components
wherever this can be needed. This is time consuming hard work.
Piotr and I, sort of, the only active Log4j maintainers nowadays, are
swamped with other priorities. All that said, you can contract a maintainer
<https://logging.apache.org/log4j/2.x/support.html#commercial> to get this
rolling.

On Thu, Apr 11, 2024 at 6:36 PM Thomas, Adam 
wrote:

> Volkan,
>
> Not frustrating at all, I'm coming into this without any real experience
> with how log4j approaches API design.
>
> > Thinking naively: Assuming you already thought about it, why doesn't the
> following work for you?
> > 1. Cast the `LoggerContext` to `AsyncLoggerContext`
> > 2. Reflectively extract the `AsyncLoggerContext#loggerDisruptor`
> > 3. Reflectively extract the `AsyncLoggerDisruptor#asyncQueueFullPolicy`
> > 4. Call `DiscardingAsyncQueueFullPolicy.getDiscardCount()` on the policy
> instance
>
> It's something we considered a last resort. I'm always hesitant to couple
> to a non-public API (such as reflecting into private fields), and doubly so
> when it is code I do not control.
>
> -Adam
>
>
> From: Volkan Yazıcı 
> Date: Thursday, April 11, 2024 at 4:09 AM
> To: Adam Thomas 
> Cc: "dev@logging.apache.org" 
> Subject: RE: Monitoring discards from async logging
>
>
> Hello Adam,
>
> I am very sorry for the frustrating experience. I can certainly see you do
> an excellent job in getting engaged with the project to get this feature
> request rolling. Thank you so much for that.
>
> I read your conversation with Piotr and I get your point – yes, neither
> migration to Log4j 3 (which still has an unknown release date), nor
> augmenting the configuration is an option for you. I agree with your
> assessments there.
>
> Thinking naively: Assuming you already thought about it, why doesn't the
> following work for you?
> 1. Cast the `LoggerContext` to `AsyncLoggerContext`
> 2. Reflectively extract the `AsyncLoggerContext#loggerDisruptor`
> 3. Reflectively extract the `AsyncLoggerDisruptor#asyncQueueFullPolicy`
> 4. Call `DiscardingAsyncQueueFullPolicy.getDiscardCount()` on the policy
> instance
> Kind regards.
>
> On Wed, Apr 10, 2024 at 12:11 AM Thomas, Adam 
> wrote:
> I created a pull request[1] late last year for this, and was encouraged to
> start a discussion on the dev list at the time.
>
> We run log4j2 on a lot of hosts that are operated by different teams. By
> default, our users use async logging and use the Discard
> asyncQueueFullPolicy. As far as I can tell, the status logger gets one WARN
> message the first time an event is discarded, and during shutdown a TRACE
> message is emitted if any discards happened. This leaves us with
> essentially no insight as to whether or not events are actually being
> discarded, when they are being discarded, or how many we are discarding.
>
> My pull request is pretty simple; the discarding policy already tracks
> discards, and I expose a getter for this counter from
> LoggerContext/AsyncLoggerContext. Polling the counter and pushing It to our
> telemetry system is an exercise left up to the user (me). I don’t have any
> real concerns about the safety of the change, but I have no idea how
> acceptable it is as a change to the API surface.
>
> This also spawned a metrics proof-of-concept that was recently closed [2].
> I’m fine with a more complete metrics solution, but I’m not sure of the
> timeline of such a solution, if any.
>
> How can I move forward on this pull request to make it acceptable, or is
> there another way to accomplish this that would be more palatable?
>
> -Adam
>
> [1] https://github.com/apache/logging-log4j2/pull/1927
> [2] https://github.com/apache/logging-log4j2/pull/1943
>
>

Re: Monitoring discards from async logging

[Thomas, Adam]

Volkan,

Not frustrating at all, I'm coming into this without any real experience with 
how log4j approaches API design.

> Thinking naively: Assuming you already thought about it, why doesn't the 
> following work for you?
> 1. Cast the `LoggerContext` to `AsyncLoggerContext`
> 2. Reflectively extract the `AsyncLoggerContext#loggerDisruptor`
> 3. Reflectively extract the `AsyncLoggerDisruptor#asyncQueueFullPolicy`
> 4. Call `DiscardingAsyncQueueFullPolicy.getDiscardCount()` on the policy 
> instance

It's something we considered a last resort. I'm always hesitant to couple to a 
non-public API (such as reflecting into private fields), and doubly so when it 
is code I do not control.

-Adam


From: Volkan Yazıcı 
Date: Thursday, April 11, 2024 at 4:09 AM
To: Adam Thomas 
Cc: "dev@logging.apache.org" 
Subject: RE: Monitoring discards from async logging


Hello Adam, 

I am very sorry for the frustrating experience. I can certainly see you do an 
excellent job in getting engaged with the project to get this feature request 
rolling. Thank you so much for that.

I read your conversation with Piotr and I get your point – yes, neither 
migration to Log4j 3 (which still has an unknown release date), nor augmenting 
the configuration is an option for you. I agree with your assessments there.

Thinking naively: Assuming you already thought about it, why doesn't the 
following work for you?
1. Cast the `LoggerContext` to `AsyncLoggerContext`
2. Reflectively extract the `AsyncLoggerContext#loggerDisruptor`
3. Reflectively extract the `AsyncLoggerDisruptor#asyncQueueFullPolicy`
4. Call `DiscardingAsyncQueueFullPolicy.getDiscardCount()` on the policy 
instance
Kind regards.

On Wed, Apr 10, 2024 at 12:11 AM Thomas, Adam  
wrote:
I created a pull request[1] late last year for this, and was encouraged to 
start a discussion on the dev list at the time.

We run log4j2 on a lot of hosts that are operated by different teams. By 
default, our users use async logging and use the Discard asyncQueueFullPolicy. 
As far as I can tell, the status logger gets one WARN message the first time an 
event is discarded, and during shutdown a TRACE message is emitted if any 
discards happened. This leaves us with essentially no insight as to whether or 
not events are actually being discarded, when they are being discarded, or how 
many we are discarding.

My pull request is pretty simple; the discarding policy already tracks 
discards, and I expose a getter for this counter from 
LoggerContext/AsyncLoggerContext. Polling the counter and pushing It to our 
telemetry system is an exercise left up to the user (me). I don’t have any real 
concerns about the safety of the change, but I have no idea how acceptable it 
is as a change to the API surface.

This also spawned a metrics proof-of-concept that was recently closed [2]. I’m 
fine with a more complete metrics solution, but I’m not sure of the timeline of 
such a solution, if any.

How can I move forward on this pull request to make it acceptable, or is there 
another way to accomplish this that would be more palatable?

-Adam

[1] https://github.com/apache/logging-log4j2/pull/1927
[2] https://github.com/apache/logging-log4j2/pull/1943

Re: Monitoring discards from async logging

[Volkan Yazıcı]

Hello Adam,

I am very sorry for the frustrating experience. I can certainly see you do
an excellent job in getting engaged with the project to get this feature
request rolling. Thank you so much for that.

I read your conversation with Piotr and I get your point – yes, neither
migration to Log4j 3 (which still has an unknown release date), nor
augmenting the configuration is an option for you. I agree with your
assessments there.

Thinking naively: Assuming you already thought about it, why doesn't the
following work for you?

   1. Cast the `LoggerContext` to `AsyncLoggerContext`
   2. Reflectively extract the `AsyncLoggerContext#loggerDisruptor`
   3. Reflectively extract the `AsyncLoggerDisruptor#asyncQueueFullPolicy`
   4. Call `DiscardingAsyncQueueFullPolicy.getDiscardCount()` on the policy
   instance

Kind regards.

On Wed, Apr 10, 2024 at 12:11 AM Thomas, Adam 
wrote:

> I created a pull request[1] late last year for this, and was encouraged to
> start a discussion on the dev list at the time.
>
> We run log4j2 on a lot of hosts that are operated by different teams. By
> default, our users use async logging and use the Discard
> asyncQueueFullPolicy. As far as I can tell, the status logger gets one WARN
> message the first time an event is discarded, and during shutdown a TRACE
> message is emitted if any discards happened. This leaves us with
> essentially no insight as to whether or not events are actually being
> discarded, when they are being discarded, or how many we are discarding.
>
> My pull request is pretty simple; the discarding policy already tracks
> discards, and I expose a getter for this counter from
> LoggerContext/AsyncLoggerContext. Polling the counter and pushing It to our
> telemetry system is an exercise left up to the user (me). I don’t have any
> real concerns about the safety of the change, but I have no idea how
> acceptable it is as a change to the API surface.
>
> This also spawned a metrics proof-of-concept that was recently closed [2].
> I’m fine with a more complete metrics solution, but I’m not sure of the
> timeline of such a solution, if any.
>
> How can I move forward on this pull request to make it acceptable, or is
> there another way to accomplish this that would be more palatable?
>
> -Adam
>
> [1] https://github.com/apache/logging-log4j2/pull/1927
> [2] https://github.com/apache/logging-log4j2/pull/1943
>

Re: Monitoring discards from async logging

[Thomas, Adam]

> All you need is a `log4j2.component.properties` file in your library.

It is very common practice here to explicitly specify the discard policy as a 
system property, as it was recommended to teams moving from log4j1 to log4j2. 
Adding this would have very little impact.

> Until then I am afraid that all libraries that propose metrics and traces in 
> Log4j Core will need to have specific version for each release.

I can cope with different log4j versions having different ways to discover 
metrics, I understand a major version increase is likely to be disruptive. I 
also completely understand not wanting to add this as a first-class property of 
`LoggerContext`. Would publishing this value via JMX be more palatable? I 
believe you've removed JMX from 3.x, so this hopefully would not collide with 
whatever instrumentation you decided to backport to 2.x.

-Adam

On 4/10/24, 10:42 AM, "Piotr P. Karwasz" mailto:piotr.karw...@gmail.com>> wrote:

Hi Adam,


On Wed, 10 Apr 2024 at 19:03, Thomas, Adam mailto:adamt...@amazon.com.inva>lid> wrote:
> I would do this if I had control of the end user's logging setup. 
> Unfortunately, I vend an internal library that is used by a lot of different 
> teams. By the time my code is invoked, AsyncLoggerDisruptor has already read 
> the system property and initialized the queue full policy. For other log4j 
> telemetry I just programmatically inspect the LoggerContext when my code 
> initializes, and install a property change listener to listen for 
> configuration changes. Even if I vend my own custom AsyncQueueFullPolicy, 
> practically speaking I will never get very good coverage, because I would 
> have to go convince thousands of developers to change their system properties.


Log4j supports multiple property sources, not only Java system properties:


https://logging.apache.org/log4j/2.x/manual/configuration#SystemProperties 



All you need is a `log4j2.component.properties` file in your library.


Regarding your original PR, we are working hard to reach a definitive
Log4j Core 3.x version. Once that is achieved, we can propose a way to
get metrics that will work in both 2.x and 3.x. Until then I am afraid
that all libraries that propose metrics and traces in Log4j Core will
need to have specific version for each release.


Piotr

Re: Monitoring discards from async logging

[Piotr P. Karwasz]

Hi Adam,

On Wed, 10 Apr 2024 at 19:03, Thomas, Adam  wrote:
> I would do this if I had control of the end user's logging setup. 
> Unfortunately, I vend an internal library that is used by a lot of different 
> teams. By the time my code is invoked, AsyncLoggerDisruptor has already read 
> the system property and initialized the queue full policy. For other log4j 
> telemetry I just programmatically inspect the LoggerContext when my code 
> initializes, and install a property change listener to listen for 
> configuration changes. Even if I vend my own custom AsyncQueueFullPolicy, 
> practically speaking I will never get very good coverage, because I would 
> have to go convince thousands of developers to change their system properties.

Log4j supports multiple property sources, not only Java system properties:

https://logging.apache.org/log4j/2.x/manual/configuration#SystemProperties

All you need is a `log4j2.component.properties` file in your library.

Regarding your original PR, we are working hard to reach a definitive
Log4j Core 3.x version. Once that is achieved, we can propose a way to
get metrics that will work in both 2.x and 3.x. Until then I am afraid
that all libraries that propose metrics and traces in Log4j Core will
need to have specific version for each release.

Piotr

Re: Monitoring discards from async logging

[Thomas, Adam]

> What version of Java are you using? If it is Java 17

Unfortunately, I need to support Java 8 and 11.

> you can still configure Log4j Core to use your own implementation of 
> `AsyncQueueFullPolicy` by setting the `log4j2.asyncQueueFullPolicy` system 
> property

I would do this if I had control of the end user's logging setup. 
Unfortunately, I vend an internal library that is used by a lot of different 
teams. By the time my code is invoked, AsyncLoggerDisruptor has already read 
the system property and initialized the queue full policy. For other log4j 
telemetry I just programmatically inspect the LoggerContext when my code 
initializes, and install a property change listener to listen for configuration 
changes. Even if I vend my own custom AsyncQueueFullPolicy, practically 
speaking I will never get very good coverage, because I would have to go 
convince thousands of developers to change their system properties.

-Adam

On 4/9/24, 4:02 PM, "Piotr P. Karwasz" mailto:piotr.karw...@gmail.com>> wrote:

Hi Adam,


On Wed, 10 Apr 2024 at 00:11, Thomas, Adam mailto:adamt...@amazon.com.inva>lid> wrote:
> I created a pull request[1] late last year for this, and was encouraged to 
> start a discussion on the dev list at the time.


Be reassured that we didn't forget about your PR, we just have a lot
of changes going on to publish Log4j Core 3.x.


> We run log4j2 on a lot of hosts that are operated by different teams. By 
> default, our users use async logging and use the Discard 
> asyncQueueFullPolicy. As far as I can tell, the status logger gets one WARN 
> message the first time an event is discarded, and during shutdown a TRACE 
> message is emitted if any discards happened. This leaves us with essentially 
> no insight as to whether or not events are actually being discarded, when 
> they are being discarded, or how many we are discarding.


What version of Java are you using? If it is Java 17 or later, you
could use Log4j Core 3.x that is based on a small dependency injection
implementation. While this technical detail is of no importance to
most users, in your case you could use an `InstancePostProcessor`[1]
to intercept the creation of each service class (AsyncQueueFullPolicy
in your case) and instrument it.


If you are running Java 8 or Java 11, you can still configure Log4j
Core to use your own implementation of `AsyncQueueFullPolicy` by
setting the `log4j2.asyncQueueFullPolicy` system property.


Piotr


[1] 
https://github.com/apache/logging-log4j2/blob/main/log4j-plugins/src/main/java/org/apache/logging/log4j/plugins/di/spi/InstancePostProcessor.java
 

Re: Monitoring discards from async logging

[Piotr P. Karwasz]

Hi Adam,

On Wed, 10 Apr 2024 at 00:11, Thomas, Adam  wrote:
> I created a pull request[1] late last year for this, and was encouraged to 
> start a discussion on the dev list at the time.

Be reassured that we didn't forget about your PR, we just have a lot
of changes going on to publish Log4j Core 3.x.

> We run log4j2 on a lot of hosts that are operated by different teams. By 
> default, our users use async logging and use the Discard 
> asyncQueueFullPolicy. As far as I can tell, the status logger gets one WARN 
> message the first time an event is discarded, and during shutdown a TRACE 
> message is emitted if any discards happened. This leaves us with essentially 
> no insight as to whether or not events are actually being discarded, when 
> they are being discarded, or how many we are discarding.

What version of Java are you using? If it is Java 17 or later, you
could use Log4j Core 3.x that is based on a small dependency injection
implementation. While this technical detail is of no importance to
most users, in your case you could use an `InstancePostProcessor`[1]
to intercept the creation of each service class (AsyncQueueFullPolicy
in your case) and instrument it.

If you are running Java 8 or Java 11, you can still configure Log4j
Core to use your own implementation of `AsyncQueueFullPolicy` by
setting the `log4j2.asyncQueueFullPolicy` system property.

Piotr

[1] 
https://github.com/apache/logging-log4j2/blob/main/log4j-plugins/src/main/java/org/apache/logging/log4j/plugins/di/spi/InstancePostProcessor.java

Monitoring discards from async logging

[Thomas, Adam]

I created a pull request[1] late last year for this, and was encouraged to 
start a discussion on the dev list at the time.

We run log4j2 on a lot of hosts that are operated by different teams. By 
default, our users use async logging and use the Discard asyncQueueFullPolicy. 
As far as I can tell, the status logger gets one WARN message the first time an 
event is discarded, and during shutdown a TRACE message is emitted if any 
discards happened. This leaves us with essentially no insight as to whether or 
not events are actually being discarded, when they are being discarded, or how 
many we are discarding.

My pull request is pretty simple; the discarding policy already tracks 
discards, and I expose a getter for this counter from 
LoggerContext/AsyncLoggerContext. Polling the counter and pushing It to our 
telemetry system is an exercise left up to the user (me). I don’t have any real 
concerns about the safety of the change, but I have no idea how acceptable it 
is as a change to the API surface.

This also spawned a metrics proof-of-concept that was recently closed [2]. I’m 
fine with a more complete metrics solution, but I’m not sure of the timeline of 
such a solution, if any.

How can I move forward on this pull request to make it acceptable, or is there 
another way to accomplish this that would be more palatable?

-Adam

[1] https://github.com/apache/logging-log4j2/pull/1927
[2] https://github.com/apache/logging-log4j2/pull/1943