Re: [RESULT] Making Log4j 2 API "the Log4j API"

[Piotr P. Karwasz]

Hi Matt,

On Mon, 22 Jan 2024 at 19:35, Matt Sicker  wrote:
>
> There was also the idea that if we introduce some form of a v3 API, it’ll be 
> alongside the existing v2 API, not a breaking change.

Yes, that idea has also a PoC:
https://github.com/apache/logging-log4j2/pull/2215

However right now I prefer Volkan's idea since:

 1. Log4j 3.x **must** come out soon. I think everyone is tired of
discussions on 3.x. Let's publish it and move onto 4.x.
 2. Since the lifecycle of Log4j API and Log4j Core will be
independent, we can switch to Log4j API 3.x at some point in the
future. I would prefer to discuss Log4j API 3.x, based on some
statistical data on how other logging APIs are used (e.g. how many
parameters do people use) and on expert and user opinions. Right now
we risk to design a Log4j API that we will need to patch multiple
times in the future.

Piotr

Re: [log4j] Bump default status logger level to WARN

[Matt Sicker]

I like this idea, Piotr.

> On Jan 22, 2024, at 12:28 PM, Piotr P. Karwasz  
> wrote:
> 
> Hi Volkan,
> 
> On Mon, 22 Jan 2024 at 15:15, Volkan Yazıcı  wrote:
>> Shall we mention this issue (that is, ineffective configurations as the one
>> you shared about `bufferSize`-vs-`bufferedIo`) to Łukasz and see if he
>> would be willing to carry out that clean up? ... granted PMC agrees to
>> raise the default status logger level to WARN.
> 
> The task to silence excessively verbose warnings can require quite a
> lot of knowledge of Log4j code internals.
> I prefer to do it myself and ask another PMC member to proofread it.
> 
> Piotr

Re: interesting writeup of some nice engineering

[Matt Sicker]

I’m not sure if it’s a binary layout, but OpenTelemetry does define some 
standard formats and protocols around transferring and collecting log data. Any 
sort of binary layout would require some level of tooling support to be useful.

> On Jan 19, 2024, at 3:12 AM, Volkan Yazıcı  wrote:
> 
> I knew about lossless-compressions with navigation capabilities, but CLP
>  is new to me. Thanks for the pointer.
> 
> For those interested, I want to add some more context into "binary
> serialization" subject:
> 
>   - LOG4J2-3023 
>   proposes a layout wrapper compressing the output of the delegate
>   - LOG4J2-1305  (and
>   LOG4J2-836 ) proposes
>   a binary layout – AFAIC, there is a proliferation of serialization options
>   (Avro, Protocol Buffers, CBOR, CLP, etc.) and this makes any specialized
>   solution opinionated. I have my doubts if just picking one is the right way
>   forward.
>   - OpenHFT Chronicle Logger 
>   implements an efficient file appender with built-in binary layout for
>   SLF4J, Logback, Log4j 1, Log4j 2, JUL, and JCL.
> 
> 
> On Thu, Jan 18, 2024 at 11:16 PM Remko Popma  wrote:
> 
>> 
>> https://www.uber.com/en-JP/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp/
>> 
>> tldr:
>> Uber created a CLP
>> <
>> https://github.com/y-scope/clp?uclick_id=e2bbfdc9-f79f-4d7a-999b-6e3c3ea19364
>>> 
>> appender for log4j that compresses on the fly.
>> Impressive 169x compression rate.
>> Resulting compressed files are searchable because of embedded dictionaries.
>> 

Re: [RESULT] Making Log4j 2 API "the Log4j API"

[Matt Sicker]

There was also the idea that if we introduce some form of a v3 API, it’ll be 
alongside the existing v2 API, not a breaking change.

> On Jan 21, 2024, at 1:32 PM, Volkan Yazıcı  wrote:
> 
> *Abstract:* There will not be a Log4j 3 API. Both Log4j 2 and Log4j 3 will
> implement the Log4j 2 API.
> 
> In the video call we had with Ralph, Matt, Piotr, Christian, Robert, and I
> on 2024-01-19 (last Friday), we decided to proceed with this plan. In a
> nutshell,
> 
>   1. We will *make the business logic of Log4j 2 API pluggable* in a way
>   seamless to existing Log4j 2 users. Hence, everything that used to work for
>   Log4j 2 API, will work intact.
>   2. We will use this (new) pluggability to *support Log4j 2 API in Log4j
>   3*. Hence, there will not be a Log4j 3 API.
>   3. Preferably, we will *move Log4j 2 API to a separate repository* with
>   its own release lifecycle and further explore options on how to evolve it.
> 
> Let me know if you have any questions or concerns.
> 
> On Wed, Jan 17, 2024 at 5:11 PM Volkan Yazıcı  wrote:
> 
>> Given Ralph and Piotr are strongly opinionated about keeping
>> `log4j-api-3.x` binary compatible to `log4j-api-2.x`, can we not release
>> `log4j-api-3.x` in `main` and make `main` only depend on `log4j-api-2.x`
>> instead? (We can move the contents of the `spi` package in `log4j-api-3.x`
>> to a separate `log4j-spi` module in `main`.) This will make everything
>> crystal clear:
>> 
>>   - Log4j 3 is just a major improvement over the backend
>>   - Log4j 3 still supports Log4j 2 API
>>   - We can move the Log4j 2 API to a separate repository with its own
>>   release life cycle (ala SLF4J)
>>   - When time comes to make a new Log4j API where PMC agrees to make
>>   breaking changes, we can call that one Log4j 3 API
>> 
>> I would appreciate it if you can help me to understand if I am
>> missing something. Otherwise, I would like to know why we need to make a
>> major release for a project that is identical to its previous version.
>> 

Re: [log4j] Bump default status logger level to WARN

[Piotr P. Karwasz]

Hi Volkan,

On Mon, 22 Jan 2024 at 15:15, Volkan Yazıcı  wrote:
> Shall we mention this issue (that is, ineffective configurations as the one
> you shared about `bufferSize`-vs-`bufferedIo`) to Łukasz and see if he
> would be willing to carry out that clean up? ... granted PMC agrees to
> raise the default status logger level to WARN.

The task to silence excessively verbose warnings can require quite a
lot of knowledge of Log4j code internals.
I prefer to do it myself and ask another PMC member to proofread it.

Piotr

Re: [log4j] Bump default status logger level to WARN

[Volkan Yazıcı]

I am in favor of reporting unrecognized/ignored properties, otherwise there
is no incentive for users to fix their configurations. Those who don't want
to be bothered with them, can still do so via
`-Dlog4j.StatusLogger.level=off`.

Shall we mention this issue (that is, ineffective configurations as the one
you shared about `bufferSize`-vs-`bufferedIo`) to Łukasz and see if he
would be willing to carry out that clean up? ... granted PMC agrees to
raise the default status logger level to WARN.

On Mon, Jan 22, 2024 at 2:34 PM Piotr P. Karwasz 
wrote:

> Hi Volkan,
>
> On Mon, 22 Jan 2024 at 13:31, Volkan Yazıcı  wrote:
> > Piotr, could you share some examples of typical working configurations
> that
> > will start reporting errors? What kind of errors will these reports
> > contain? A message or a fully-blown stack trace? Will there be a
> multitude
> > of these? Or 1-2 occasional appearances?
>
> There are about 100 warnings in `log4j-core`. Most of them only occur
> if an error condition occurs, but it is fixed by the code.
> For example if a user has a properties configuration with a lot of
> unrecognized/ignored properties, a warning will be issued for each one
> of them.
>
> Some of these warnings are not fixable by the user, e.g. the "The
> bufferSize is set to {} but bufferedIo is false: {}" (bufferSize has
> always a value), but I am willing to go through the list of warnings
> and fix them.
>
> Piotr
>

Re: [log4j] Bump default status logger level to WARN

[Piotr P. Karwasz]

Hi Volkan,

On Mon, 22 Jan 2024 at 13:31, Volkan Yazıcı  wrote:
> Piotr, could you share some examples of typical working configurations that
> will start reporting errors? What kind of errors will these reports
> contain? A message or a fully-blown stack trace? Will there be a multitude
> of these? Or 1-2 occasional appearances?

There are about 100 warnings in `log4j-core`. Most of them only occur
if an error condition occurs, but it is fixed by the code.
For example if a user has a properties configuration with a lot of
unrecognized/ignored properties, a warning will be issued for each one
of them.

Some of these warnings are not fixable by the user, e.g. the "The
bufferSize is set to {} but bufferedIo is false: {}" (bufferSize has
always a value), but I am willing to go through the list of warnings
and fix them.

Piotr

Re: [log4j] Bump default status logger level to WARN

[Volkan Yazıcı]

Piotr, could you share some examples of typical working configurations that
will start reporting errors? What kind of errors will these reports
contain? A message or a fully-blown stack trace? Will there be a multitude
of these? Or 1-2 occasional appearances?

I know answers will vary dependending on the configuration. I am interested
in, let's say, typical Spring, Java application users with simple
`log4j2.xml`, `log4j2.yaml` files.

On Wed, Jul 26, 2023 at 12:14 PM Piotr P. Karwasz 
wrote:

> Hi all,
>
> In [1] Łukasz implemented the long-awaited bump of the default status
> logger level from ERROR to WARN.
>
> From my perspective this is a good change, since most configuration
> errors can be caught without setting `-Dlog4j2.debug=true`. On the
> other hand some users will complain that "working" configurations will
> start reporting errors on stderr now.
>
> What do you think about this?
>
> Piotr
>
> [1] https://github.com/apache/logging-log4j2/pull/1592
>