Re: Context data propagation and logger-bound context data

[Matt Sicker]

https://docs.micrometer.io/context-propagation/reference/

This is more in line with what I was thinking we should support. I’m not 
suggesting that we add Micrometer as a dependency to the API or Core, but we 
can replicate the minimal API here or figure out a smaller API for integration 
purposes.

Note that similar projects like the OpenTelemetry context propagation API is 
geared toward cross-process context propagation, not cross-thread or similar 
lower level primitives. The low level primitives belong in Log4j IMO.


> On Mar 27, 2024, at 16:56, Ralph Goers  wrote:
> 
> Volkan,
> 
> No more hand waving. Please see 
> https://github.com/apache/logging-log4j2/pull/2419.
> 
> I should note that while implementing the classes I added to support this 
> makes it easier I did not have to make any changes to the logging internals 
> to make this work.
> 
> Ralph
> 
>> On Mar 22, 2024, at 1:45 AM, Volkan Yazıcı  wrote:
>> 
>> No, it is not the same thing Matt. Let me be as explicit as I can:
>> 
>> var logger0 = getLogger();  // MDC: {}
>> var logger1 = logger0.withContextData("x", 1);  // MDC: {x: 1}
>> var logger2 = logger1.withContextData("y", 2);  // MDC: {x: 1, y: 2}
>> 
>> This is the functionality being requested. Whoever claims this can be done 
>> using a `MessageFactory`, they need to share a working [pseudo] code, 
>> instead of hand waving. So far, nobody responded to this. Piotr, speculated 
>> on a non-existing `Logger#withMessageFactory(MessageFactory)`, but there is 
>> not one single working example shared. Hence, unless you can prove me wrong 
>> with a working practical[1] example, the requested feature is currently 
>> known to be not practically possible in Log4j.
>> 
>> [1] Implementing `logger.withContextData("x", 1)` with 50 LoC Java code 
>> using the existing Log4j feature set is not a "practical example".
>> 
>> P.S. For Log4j 3 API Javadocs, you can browse to 
>> https://logging.apache.org/log4j/3.x and search for "Javadoc" in the menu. 
>> (Obviously, same works for Log4j 2 too.)
>> 
>> On Thu, Mar 21, 2024 at 6:10 PM Matt Sicker  wrote:
>> LogManager - log4j-api 3.0.0-alpha1 javadoc
>> javadoc.io
>> 
>> Pass your custom MessageFactory here. It’s an optional argument when 
>> creating the Logger.
>> 
>> Also, I’m not sure where to even find the current javadocs for the API. 
>> javadoc.io only seems to have this alpha release.
>> 
>> 
>>> On Mar 21, 2024, at 04:34, Volkan Yazıcı  wrote:
>>> 
>>> Ralph, could you show how those two users can use a `MessageFactory` to
>>> create `Logger`s with predefined additional context data?
>>> 
>>> On Thu, Mar 21, 2024 at 7:25 AM Ralph Goers  wrote:
>>> 
>>>> Unfortunately this is another message I somehow didn't get in my inbox.
>>>> Replying to it via lists.a.o is not a great experience but is the best I
>>>> can do.
>>>> 
>>>> On 2024/03/20 13:51:56 Volkan Yazıcı wrote:
>>>>> I agree with the way Piotr dissects the problem. I think `ScopedContext`,
>>>>> even though it has its own merits, doesn't address the problem reported
>>>> by
>>>>> users. They simply want a new logger associated with some additional
>>>>> context data.
>>>> 
>>>> Two users do.  I have personally been asked for something like
>>>> ScopedContext several times.
>>>> As I replied to Piotr, we already solved the problem of adding data to
>>>> Loggers. That is what MessageFactories are intended for.
>>>> 
>>>>> 
>>>>> *[See my comments below.]*
>>>>> 
>>>>> On Mon, Mar 18, 2024 at 10:40 AM Piotr P. Karwasz <
>>>> piotr.karw...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> * we can create a `Logger` wrapper "bound" to context data as Mikko
>>>>>> does. This wrapper will take care of setting the `ThreadContext`
>>>>>> before the logger call and restore it after it.
>>>>> 
>>>>> Creating a wrapper `Logger` can work without needing to deal with
>>>>> `ThreadContext`. I can think of two different ways to carry this out:
>>>>> 
>>>>>  1. Currently, `AbstractLogger` only creates `Message`s. We can rework
>>>> it
>>>>>  to create `LogEvent`s too. Once `AbstractLogger` gets its hand on a
>>>>>  `LogEvent`, it can enrich its context data as it wishes.

Re: Better Javadoc/Impl for org.apache.logging.log4j.core.Layout.getContentType()

[Matt Sicker]

We’ve got nullability annotations now, so we should be defining them. For use 
cases like this, I’d examine how existing things work and then document the 
nullability as such. For new code, I prefer things to be non-null except for 
cases where it makes sense to use null rather than, say, passing Optional 
parameters to methods or things like that.

> On Apr 12, 2024, at 08:55, Gary D. Gregory  wrote:
> 
> Hi All,
> 
> Our current Javadoc says:
> 
>/**
> * Returns the content type output by this layout. The base class returns 
> "text/plain".
> *
> * @return the content type.
> */
>String getContentType();
> 
> I'd like to clarify if returning null should be legal/allowed for 
> implementation in general.
> 
> IOW, I'd like to say either:
> - Call sites should be expected to handle null, or
> - Implementations MUST not return null
> 
> WDYT?
> Gary

Re: Monitoring discards from async logging

[Matt Sicker]

We’ve got a new proposal from Piotr on this front now which sort of emulates 
his suggestion for 3.x but applied to 2.x (since that doesn’t have the DI 
system). I like the approach there as it provides enough of a hook for users to 
customize whatever metrics library they wish to enhance some components however 
they see fit (similar to Spring’s BeanPostProcessor functionality).

> On Apr 12, 2024, at 06:33, Piotr P. Karwasz  wrote:
> 
> Hi Volkan,
> 
> On Thu, 11 Apr 2024 at 21:26, Volkan Yazıcı  wrote:
>> *Slightly longer answer:* Log4j has never had a holistic observability view
>> to its internals. We need something new (if not, from scratch), and
>> preferably, applicable to not only async. logging, but all components
>> wherever this can be needed. This is time consuming hard work.
>> Piotr and I, sort of, the only active Log4j maintainers nowadays, are
>> swamped with other priorities. All that said, you can contract a maintainer
>>  to get this
>> rolling.
> 
> I think you meant "you can contribute the functionality or contract
> **any** developer to do it". ;-) The way you wrote it sounds horrible.
> 
> Adam, to be entirely clear:
> 
> * Volkan and I are **not** the only maintainers. If a majority of
> maintainers support your PR, personally I will not block it, although
> I prefer to keep implementation details encapsulated as much as
> possible, so we can refactor them at any time,
> 
> * We would like to have a holistic solution to instrumenting Log4j
> Core. Even though Volkan and I undeniably work a lot on Log4j as part
> of our job, adding metrics to Log4j is not covered by our clients. So
> we cannot afford to deliver it ourselves in our spare time,
> 
> * Volkan probably meant that a Log4j maintainer could deliver this
> work faster, but we really don't care who contributes it. If you can
> contribute such a PR, we would be happy to review it. However before
> starting such a major endeavour you should either discuss the details
> on this mailing list or you can contact us directly and we can
> organize a video call with the rest of the PMC,
> 
> * Let me stress that we treat all PRs equally, whether coming from a
> project founder, PMC member or external contributors. My
> co-maintainers could probably tell you more about it. ;-)
> 
> Piotr

Re: [VOTE] Release Apache Log4j Tools 0.8.0 (RC3)

[Matt Sicker]

+1

Validated checksums, signatures, reproducible build.

Small note: NOTICE.txt should get an updated copyright year, but not a blocker.

> On Mar 21, 2024, at 10:42, Volkan Yazıcı  wrote:
> 
> This is a vote to release the Apache Log4j Tools 0.8.0 (RC3).
> 
> Website: https://logging.staged.apache.org/log4j/tools
> GitHub: https://github.com/apache/logging-log4j-tools
> Commit: 7d157b61e198e3baca816129d8d406b300436e2f
> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j-tools
> Nexus:
> https://repository.apache.org/content/repositories/orgapachelogging-1266
> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
> 
> Please download, test, and cast your votes on this mailing list.
> 
> [ ] +1, release the artifacts
> [ ] -1, don't release, because...
> 
> This vote is open until 2024-03-22 12:30 (CET) and will pass unless
> getting a net negative vote count. All votes are welcome and we
> encourage everyone to test the release, but only the Logging
> Services PMC votes are officially counted. At least 3 +1 votes and
> more positive than negative votes are required.
> 
> === Timing & Differences
> 
> I will keep this vote open until 2024-03-22 12:30 (CET), that is, the
> official end date of the RC2, since RC3 contains a minor difference:
> 
> https://github.com/apache/logging-log4j-tools/compare/2bb07037bbbfe14fe1c224d46a3e4135b48ffde6...7d157b61e198e3baca816129d8d406b300436e2f
> 
> === Review kit
> 
> The minimum set of steps needed to review the uploaded distribution
> files in the Subversion repository can be summarized as follows:
> 
># Check out the distribution
>svn co https://dist.apache.org/repos/... && cd $_
> 
># Verify checksums
>shasum --check *.sha512
> 
># Verify signatures
>wget -O - https://downloads.apache.org/logging/KEYS | gpg --import
>for sigFile in *.asc; do gpg --verify $sigFile; done
> 
># Verify reproduciblity
>umask 0022
>unzip *-src.zip -d src
>cd src
>export NEXUS_REPO=https://repository.apache.org/content/...
>sh mvnw -Prelease verify artifact:compare -Dreference.repo=$NEXUS_REPO
># If preferred, augment `mvnw` with `-DskipTests` to speed things up
> 
> === Release notes
> 
> This release delivers the first version of Log4j Docgen (Documentation
> Generator).
> It is a set of tools to auto-generate the Log4j plugin documentation (to be
> integrated into the website) and the Log4j configuration XSD file (for
> assisting the configuration of the Log4j runtime, i.e., `log4j2.xml`) from
> the Log4j source code. See the project website for details.
> 
>  Added
> 
> * Add the `log4j-docgen` et al. containing a universal XML model to
> document Log4j plugins
> 
>  Changed
> 
> * Move Log4j Changelog XML namespace and schema location to `
> https://logging.apache.org/xml/ns` and `
> https://logging.apache.org/xml/ns/log4j-changelog-0.xsd`, respectively
> 
>  Removed
> 
> * Remove `author` from Log4j Changelog. It was yet another bit to maintain
> and created role-related (who did what) problems. Many modern software
> projects use a VCS (e.g., Git) and support services (e.g., GitHub) which
> make it trivial to trace back the origin of a change using commit and issue
> IDs.
> 
>  Updated
> 
> * Update `org.apache.logging:logging-parent` to version `10.6.0`
> * Update `jakarta.inject:jakarta.inject-api` to version `2.0.1` (#94)
> * Update `org.apache.logging.log4j:log4j-core` to version `2.23.1` (#108)
> * Update `org.apache.logging.log4j:log4j-plugins` to version `3.0.0-beta2`
> (#107)
> * Update `org.apache.maven.plugin-tools:maven-plugin-annotations` to
> version `3.11.0` (#98)
> * Update `org.assertj:assertj-core` to version `3.25.3` (#104)
> * Update `org.codehaus.modello:modello-maven-plugin` to version `2.3.0`
> (#105)
> * Update `org.junit:junit-bom` to version `5.10.2` (#103)

Re: Context data propagation and logger-bound context data

[Matt Sicker]

https://www.javadoc.io/doc/org.apache.logging.log4j/log4j-api/3.0.0-alpha1/org.apache.logging.log4j/org/apache/logging/log4j/LogManager.html

Pass your custom MessageFactory here. It’s an optional argument when creating 
the Logger.

Also, I’m not sure where to even find the current javadocs for the API. 
javadoc.io  only seems to have this alpha release.


> On Mar 21, 2024, at 04:34, Volkan Yazıcı  wrote:
> 
> Ralph, could you show how those two users can use a `MessageFactory` to
> create `Logger`s with predefined additional context data?
> 
> On Thu, Mar 21, 2024 at 7:25 AM Ralph Goers  wrote:
> 
>> Unfortunately this is another message I somehow didn't get in my inbox.
>> Replying to it via lists.a.o is not a great experience but is the best I
>> can do.
>> 
>> On 2024/03/20 13:51:56 Volkan Yazıcı wrote:
>>> I agree with the way Piotr dissects the problem. I think `ScopedContext`,
>>> even though it has its own merits, doesn't address the problem reported
>> by
>>> users. They simply want a new logger associated with some additional
>>> context data.
>> 
>> Two users do.  I have personally been asked for something like
>> ScopedContext several times.
>> As I replied to Piotr, we already solved the problem of adding data to
>> Loggers. That is what MessageFactories are intended for.
>> 
>>> 
>>> *[See my comments below.]*
>>> 
>>> On Mon, Mar 18, 2024 at 10:40 AM Piotr P. Karwasz <
>> piotr.karw...@gmail.com>
>>> wrote:
>>> 
 * we can create a `Logger` wrapper "bound" to context data as Mikko
 does. This wrapper will take care of setting the `ThreadContext`
 before the logger call and restore it after it.
>>> 
>>> Creating a wrapper `Logger` can work without needing to deal with
>>> `ThreadContext`. I can think of two different ways to carry this out:
>>> 
>>>   1. Currently, `AbstractLogger` only creates `Message`s. We can rework
>> it
>>>   to create `LogEvent`s too. Once `AbstractLogger` gets its hand on a
>>>   `LogEvent`, it can enrich its context data as it wishes.
>>>   2. We can extend `ContextDataInjector` with a new `void
>>>   injectContextData(Logger logger, StringMap target)` method, provide a
>>>   `ContextDataInjector` implementation that branches on `logger
>> instanceof
>>>   ContextDataProvider`, and call `ContextDataInjector` with the
>> associated
>>>   `Logger` in `LogEventFactory`.
>> 
>> We can do lots of things, most of which I wouldn't recommend. As to yours:
>> 1. Logger/AbstractLogger got very complex with Async, Garbage Free,
>> Reliablity Strategies, etc. Trying to move creating the LogEvent sooner is
>> likely to be a major PITA and could seriously impact performance. While we
>> could add a context map to AbstractLogger we would have to pass that on the
>> logging calls to LoggerConfig and deal with all that that means - remember,
>> a LoggerConfig can be handling multiple Loggers.
>> 2. I don't recommend extending ContextDataInjector. That proved difficult
>> to work with which is why we now recommend using ContextDataProviders. You
>> really can only have one ContextDataInjector. Also, please note that
>> ContextDataInjector is called while constructing the LogEvent. The LogEvent
>> isn't passed the Logger, only the LoggerName. Looking up the Logger to do
>> this is yet another way to slow down logging.
>> 
>>> 
>>> On Tue, Mar 19, 2024 at 7:45 AM Ralph Goers 
>>> wrote:
 In the meantime, I provided
>>> https://github.com/apache/logging-log4j2/pull/2385 which I very loosely
>>> modeled after ScopedValues.
>>> 
>>> The fact that `ScopedContext` tries to imitate `ScopedValue` using
>>> `ThreadLocal`s is extremely confusing (from a user's pov) and risky
>>> liability (from a maintainer's pov). I guess you wanted to implement *a*
>>> `ScopedValue` without using *the* `ScopedValue` to be compatible with
>> Java
>>> 8. If so, that really sounds like the `o.a.l.log4j.util.Supplier`
>> downward
>>> spiral. We can rather have an *internal* `Log4jScopedValue` interface and
>>> provide Java 8 (using `InheritableThreadLocal`) and Java 21+ (using
>>> `ScopedValue`) compatible solutions in an MRJ (Multi-Release JAR).
>> 
>> I am NOT trying to make ScopedContext compatible with ScopedValue. I am
>> trying to make it conceptually close enough to ScopedValue that users will
>> understand what it is doing.
>> We can argue about naming if you want. Gary has already expressed his
>> opinion.
>>> 
>>> We can integrate `ScopedContext` to the `LogEventFactory` by providing a
>>> specialized `ContextDataInjector` plugin – assuming `LogEventFactory`
>>> employs all available `ContextDataInjector` plugins.
>> 
>> ScopedContext is integrated with a ContextDataProvider, which is the
>> supported way to do this. Again, you cannot have more than one
>> ContextDataInjector so providing "specialized versions" is a pipe dream.
>> You will simply have to enhance the one we already have.
>> ContextDataInjector is NOT a plugin.
>> 
>>> 
>>> I find the current ceremony 

Re: Clean site repo

[Matt Sicker]

Must be a relic from when we used Subversion to publish the website instead of 
Git. Maybe ask Infra?

> On Feb 26, 2024, at 12:05, Piotr P. Karwasz  wrote:
> 
> Hi Matt,
> 
> On Mon, 26 Feb 2024 at 18:49, Matt Sicker  wrote:
>> 
>> The /log4j/2.0/ link is because 2.0 is a symbolic link to 2.x, and 2.x is a 
>> symbolic link to the latest release (or at least that’s how it was set up 
>> originally). The links can probably use .htaccess rewrite rules instead.
> 
> I really don't see it! I see the website:
> 
> https://logging.apache.org/log4j/2.0/
> 
> but I don't see any '2.0' symlink in the repo (nor any rewrite rule
> that matches 2.0)
> 
> https://github.com/apache/logging-log4j-site/tree/asf-site
> 
> Piotr

Re: Clean site repo

[Matt Sicker]

The /log4j/2.0/ link is because 2.0 is a symbolic link to 2.x, and 2.x is a 
symbolic link to the latest release (or at least that’s how it was set up 
originally). The links can probably use .htaccess rewrite rules instead.

> On Feb 21, 2024, at 04:44, Piotr P. Karwasz  wrote:
> 
> Hi,
> 
> On Mon, 19 Feb 2024 at 10:34, Volkan Yazıcı  > wrote:
>> 
>> If there are no objections, after Log4j `2.23.0` and `3.0.0-beta2`
>> releases, I want to proceed with replacing the contents of
>> `asf-{site,staging}` branches with `clean-staging`. Effectively, this
>> operation has the following outcomes:
>> 
>>   - The `logging-log4j-site` repository will contain only following files:
>>  - `1.x`
>>  - `2.12.x`
>>  - `2.3.x`
>>  - `2.x`
>>  - `3.x`
>>  - `.asf.yaml`
>>  - `changelog-*.xsd`
>>  - `extras`
>>  - `.htaccess`
>>   - `.htaccess` will take care of redirections for backward compatibility
> 
> I have replaced `asf-staging` with `clean-staging`, but `asf-site`
> remains for now untouched.
> 
> Please peruse the `https://logging.staged.apache.org/log4j/` 
>  pages to
> see if something is wrong.
> 
> As I stated before, the main rationale behind this change is SEO. What
> do you get if you perform the following search:
> 
> https://www.google.com/search?client=firefox-b-e=DefaultThreadContextMap+log4j+javadoc
> 
> The result depends on the location, but I find results from:
> 
> * /log4j/2.0/: (sic!) I didn't even know it existed, can someone tell
> me from which repo is this served?
> * /log4j/log4j-2.3.2/
> * Eventually there is a result from `/log4j/2.x/` but pointing at the
> wrong class.
> 
> Remark: in the renewed staging site there are **still** a
> /log4j/2.3.x/ and /log4j/2.12.x/ folders and the rewrite rules
> "round-up" the `/log4j/log4j-` folders to one of
> /log4j/2.3.x/, `/log4j/2.12.x/` or `/log4j/2.x/`.
> We could probably also drop the documentation of those versions too,
> but this is another discussion.
> 
> Piotr
> 
> PS: right now INFRA didn't update
> https://logging.staged.apache.org/log4j/ yet, but you can see the
> intended content on https://logging-clean.staged.apache.org/log4j/

Re: Migrate *all* Issue Tracking and Discussions to GitHub

[Matt Sicker]

I think it’s a good idea, especially since Jira registrations are closed.

> On Feb 13, 2024, at 02:20, Volkan Yazıcı  wrote:
> 
> Log4j has deprecated JIRA in favor of GitHub Issues and enabled GitHub
> Discussions as an alternative (not replacement!) to mailing lists. So far
> it has been a great success[1]. I suggest doing the same for Log4cxx and
> Log4net too. Thoughts? Objections?
> 
> Note that I am not talking about only enabling these features. But to
> actively promote them on the website too. Check out the Log4j support page
>  for an example.
> 
> [1] Code was already on GitHub. Now we can refer to PRs, issues, commits,
> code blocks, etc. while having conversations on any GitHub text input. I
> find this quite convenient. IMO, as a result of this convenience, I see way
> more maintainer engagement in PRs and issues. Next to that, GitHub
> Discussions clearly enabled more user interactions. It works around the
> mailing list subscription barrier.

Re: [log4j] Remove `` in `main`

[Matt Sicker]

If there were a way to allow for multiple StatusLogger configurations 
concurrently, then I’d support keeping it in the config file. Otherwise, I’m 
somewhat ambivalent about this.

> On Feb 12, 2024, at 13:29, Volkan Yazıcı  wrote:
> 
> *Abstract:* I want to remove from `main` the feature that a `Configuration`
> (e.g., `` in a `log4j2.xml`)
> configuring the `StatusLogger`, and instead, only allow configuration using
> properties.
> 
> `StatusLogger` can be configured in following ways:
> 
>   1. Passing system properties to the Java process (e.g.,
>   `-Dlog4j2.StatusLogger.level=INFO`)
>   2. Providing properties in a `log4j2.StatusLogger.properties` file in
>   the classpath
>   3. Using Log4j configuration (i.e., `   dest="out">`) in a `log4j2.xml` in the classpath. This is facilitated by
>   `StatusConfiguration`.
>   4. Programmatically (i.e., `StatusLogger.getLogger().setLevel(...)`)
> 
> `StatusConfiguration`-based configuration has certain caveats:
> 
>   - There is a time between the first `StatusLogger` access and a
>   configuration file (e.g., `log4j2.xml`) read. Hence, there is a time window
>   that only the defaults will be effective.
>   - `StatusConfiguration` is created per `LoggerContext` and each LC
>   mutates the (globally shared!) `StatusLogger`. Hence, the last loaded
>   configuration always becomes the effective one.
> 
> Given the purpose of `StatusLogger` (i.e., the logger of the logging
> system) and above shortcomings related to `StatusConfiguration`, I want to
> remove the `StatusConfiguration`-based configuration in `main`. Thoughts?
> Objections?

Re: New committer: Fred am Nil (`freeandnil`)

[Matt Sicker]

Welcome, Fred!

> On Feb 9, 2024, at 13:06, Volkan Yazıcı  wrote:
> 
> The Project Management Committee (PMC) for Apache Logging Services has
> invited Fred am Nil to become a committer and we are pleased to announce
> that they have accepted! 
> 
> They have been a long time Log4Net contributor and I am very happy to see
> them stepping up to become a maintainer for it. Logging Services
>  is more than Log4Net; we have Log4j, Log4cxx,
> and several other projects. I am looking forward to their contributions to
> all these projects! 

Re: Revamp of asynchronous logging

[Matt Sicker]

For #3, I agree that we should perform formatting on the application thread 
before transferring control to an asynchronous thread.

> On Feb 5, 2024, at 14:49, Piotr P. Karwasz  wrote:
> 
> Hi all (and especially Remko),
> 
> Lately I spent some time looking at the asynchronous logger/logger
> config and I think it could profit from some refactoring in 3.x.
> 
> The problems I am seeing are:
> 
> 1. The disruptors of `AsyncLogger` and `AsyncLoggerConfig` use
> different structures. The disruptor of `AsyncLoggerConfig` contains
> basically a `Pair`, while `AsyncLogger` uses
> `RingBufferLogEvent`, which is basically `MutableLogEvent` with an
> additional field of type `AsyncLogger`.
> 
> The design of the `AsyncLoggerConfigDisruptor` seems better: it always
> uses an allocation-free `EventTranslatorTwoArg`[1], while
> `AsyncLoggerDisruptor` users must choose between a
> `EventTranslatorVarArg` which allocates a temporary array and an
> `EventTranslator` that copies the log event into a thread-bound
> structure first.
> 
> Can we remove `RingBufferLogEvent` and use `Pair MutableLogEvent>` instead?
> 
> 2. I am not convinced by the utility of having multiple
> implementations of `ReusableMessage`. Since the `ReusableMessage`
> interface does not specify how to format messages, transferring data
> between a `ReusableParameterizedMessage` and `MutableLogEvent`
> requires a call to `ReusableMessage#formatTo` to account for the
> possibility of different formatting algorithms.
> 
> We could remove `ReusableObjectMessage`, `ReusableSimpleMessage`,
> `ReusableParameterizedMessage` and merge `ReusableMessageFactory` with
> `ReusableLogEventFactory`.
> 
> Instead of 4 thread locals for each type of `ReusableMessage` we would
> just have a single `Recycler` and the performance of
> `MutableLogEvent#setMessage` could be improved:
> 
> public void setMessage(final Message message) {
>if (message instanceof MutableLogEvent) {
>// NOP
>} else ...
> }
> 
> 3. Before a thread jump, three values must be computed: the source
> location (optional), the context map and the formatted message.
> Currently the location were these are computed varies and can be as
> early as the `ReusableLogEventFactory`. I would propose to move them
> to the last moment before we pass control to another thread, i.e.:
> 
> * in `EventTranslator#translateTo` (this way nothing is computed if
> the ring buffer is full and the `AsyncQueueFullPolicy` chooses to
> discard the message) for `AsyncLogger`/`AsyncLoggerConfig`,
> * just before  the call to `Queue#offer` for the `AsyncAppender`.
> 
> What do you think?
> 
> Piotr
> 
> [1] 
> https://lmax-exchange.github.io/disruptor/javadoc/com.lmax.disruptor/com/lmax/disruptor/EventTranslatorTwoArg.html

Re: [Log4j] If and how to document potential OOME

[Matt Sicker]

Perhaps mention of StringBuilderFormattable? It helps work around the “load the 
string all at once” problem.

> On Jan 26, 2024, at 06:36, Gary Gregory  wrote:
> 
> Oh, I don't want to suggest this is a Lo4j issue at all. I am wondering if
> there is a pattern we should document aside from "know your third party
> objects" which is not helpful for us to say. It sounds like there is
> nothing for us to do.
> 
> Gary
> 
> On Fri, Jan 26, 2024, 7:16 AM Apache  wrote:
> 
>> That is kind of my point. Anyone who logs an object that behaves this way
>> is asking for trouble. It really sounds like an esoteric edge case to me. I
>> don’t view this as a Log4J problem as the error wouldn’t even have Log4J in
>> the stack trace.
>> 
>> Ralph
>> 
>>> On Jan 25, 2024, at 9:52 PM, Gary Gregory 
>> wrote:
>>> 
>>> I have no control over this. I think this is a general problem with
>> third
>>> party objects and toString(). There is no toString(fromHereToThere) or
>>> toStringReader() so I can't see a general way to deal with it. Even if I
>>> created a wrapper for the Object and called toString(), truncated and
>>> cached the result and have the wrapper return the shorter string in its
>>> toString(), I still would have toString()'d the original...
>>> 
>>> Gary
>>> 
>>>> On Thu, Jan 25, 2024, 11:33 PM Ralph Goers 
>>>> wrote:
>>>> 
>>>> OK. The only good way to handle that is to parse the YAML/JSON file
>> while
>>>> streaming it and extract just the fields you want to include in the
>> logs.
>>>> 
>>>> Ralph
>>>> 
>>>>> On Jan 25, 2024, at 6:40 PM, Gary Gregory 
>>>> wrote:
>>>>> 
>>>>> Well, it's worse than that because the object is an object created by
>>>>> parsing a YAML (or JSON) file, then the toString() of that object
>>>>> renders a String in some other format.
>>>>> 
>>>>> Gary
>>>>> 
>>>>> On Thu, Jan 25, 2024 at 7:45 PM Ralph Goers <
>> ralph.go...@dslextreme.com>
>>>> wrote:
>>>>>> 
>>>>>> Volkan & Matt,
>>>>>> 
>>>>>> Neither of those is going to help. The issue is that when the toString
>>>> method is called it reads a whole file in and stores it as a String.
>> This
>>>> could cause the OOM error. Truncating it in a layout simply limits how
>> much
>>>> of the String is printed. Even Gary’s proposal of calling substring() is
>>>> still going to operate on the whole String. He would really need a
>> method
>>>> that accepts the max number of characters to read from the file.
>>>>>> 
>>>>>> Ralph
>>>>>> 
>>>>>>> On Jan 25, 2024, at 2:49 PM, Volkan Yazıcı  wrote:
>>>>>>> 
>>>>>>> *[Just responding to Matt. I don't have an answer for Gary.]*
>>>>>>> 
>>>>>>> `JsonTemplateLayout` has `maxStringLength`, and related with it,
>>>>>>> `truncatedStringSuffix`.
>>>>>>> 
>>>>>>> On Thu, Jan 25, 2024 at 9:45 PM Matt Sicker 
>> wrote:
>>>>>>> 
>>>>>>>> You can use the %maxLength{…}{N} pattern converter with
>> PatternLayout
>>>> at
>>>>>>>> least. Unfortunately, I don’t think any other layouts support a
>>>> similar
>>>>>>>> option.
>>>>>>>> 
>>>>>>>>> On Jan 25, 2024, at 10:55, Gary D. Gregory 
>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hi All,
>>>>>>>>> 
>>>>>>>>> I'd like to ask how to if we can devise advice around an issue I
>> ran
>>>>>>>> into this week.
>>>>>>>>> 
>>>>>>>>> One of our test suites processes about 40K files of test fixtures.
>>>> These
>>>>>>>> inputs are parsed, processed, and asserted. This randomly fails on a
>>>> call
>>>>>>>> to Logger#debug(), randomly in that it happens usually once per
>> build,
>>>>>>>> somewhere in a logging call. But it usually fails with a call that
>>>> looks
>>>>>>>> like this:
>>>>>>>>> 
>>

Re: [Log4j] If and how to document potential OOME

[Matt Sicker]

You can use the %maxLength{…}{N} pattern converter with PatternLayout at least. 
Unfortunately, I don’t think any other layouts support a similar option.

> On Jan 25, 2024, at 10:55, Gary D. Gregory  wrote:
> 
> Hi All,
> 
> I'd like to ask how to if we can devise advice around an issue I ran into 
> this week.
> 
> One of our test suites processes about 40K files of test fixtures. These 
> inputs are parsed, processed, and asserted. This randomly fails on a call to 
> Logger#debug(), randomly in that it happens usually once per build, somewhere 
> in a logging call. But it usually fails with a call that looks like this:
> 
> logger.debug("This is fun" + myFunObject);
> 
> To simplify things, let's say that it turns out that after an underlying 
> third party jar file version upgrade the call to myFunObject#toString() no 
> longer returns Object#toString() but rather (again to simplify) the contents 
> of the file that was parsed to create myFunObject. This toString() can be 
> megabytes. The solution is obvious:
> 
> logger.debug("This is fun", myFunObject::toString)
> 
> And our CI builds no longer randomly fail since our default logging does not 
> log at the debug level.
> 
> A better solution could be:
> 
> logger.debug("This is fun", () -> myFunObject.toString().substring(0, 100))
> 
> where I still want _some_ information better than making my own toString() 
> with System#identityHashCode(Object) or somesuch. Sure, .toString() is still 
> called but it does not make it down into logging. In my case the OOME 
> happened in myFunObject::toString so the substring() example would not have 
> worked.
> 
> My question is: Should we document some general advice on this pattern and 
> what should the advice be? Would it make sense to have some features where we 
> truncate/reject Strings above a threshold. And yes, calling 
> myFunObject.toString() can still still get me in trouble.
> 
> Gary
> 

Re: [log4j] Bump default status logger level to WARN

[Matt Sicker]

I like this idea, Piotr.

> On Jan 22, 2024, at 12:28 PM, Piotr P. Karwasz  
> wrote:
> 
> Hi Volkan,
> 
> On Mon, 22 Jan 2024 at 15:15, Volkan Yazıcı  wrote:
>> Shall we mention this issue (that is, ineffective configurations as the one
>> you shared about `bufferSize`-vs-`bufferedIo`) to Łukasz and see if he
>> would be willing to carry out that clean up? ... granted PMC agrees to
>> raise the default status logger level to WARN.
> 
> The task to silence excessively verbose warnings can require quite a
> lot of knowledge of Log4j code internals.
> I prefer to do it myself and ask another PMC member to proofread it.
> 
> Piotr

Re: interesting writeup of some nice engineering

[Matt Sicker]

I’m not sure if it’s a binary layout, but OpenTelemetry does define some 
standard formats and protocols around transferring and collecting log data. Any 
sort of binary layout would require some level of tooling support to be useful.

> On Jan 19, 2024, at 3:12 AM, Volkan Yazıcı  wrote:
> 
> I knew about lossless-compressions with navigation capabilities, but CLP
>  is new to me. Thanks for the pointer.
> 
> For those interested, I want to add some more context into "binary
> serialization" subject:
> 
>   - LOG4J2-3023 
>   proposes a layout wrapper compressing the output of the delegate
>   - LOG4J2-1305  (and
>   LOG4J2-836 ) proposes
>   a binary layout – AFAIC, there is a proliferation of serialization options
>   (Avro, Protocol Buffers, CBOR, CLP, etc.) and this makes any specialized
>   solution opinionated. I have my doubts if just picking one is the right way
>   forward.
>   - OpenHFT Chronicle Logger 
>   implements an efficient file appender with built-in binary layout for
>   SLF4J, Logback, Log4j 1, Log4j 2, JUL, and JCL.
> 
> 
> On Thu, Jan 18, 2024 at 11:16 PM Remko Popma  wrote:
> 
>> 
>> https://www.uber.com/en-JP/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp/
>> 
>> tldr:
>> Uber created a CLP
>> <
>> https://github.com/y-scope/clp?uclick_id=e2bbfdc9-f79f-4d7a-999b-6e3c3ea19364
>>> 
>> appender for log4j that compresses on the fly.
>> Impressive 169x compression rate.
>> Resulting compressed files are searchable because of embedded dictionaries.
>> 

Re: [RESULT] Making Log4j 2 API "the Log4j API"

[Matt Sicker]

There was also the idea that if we introduce some form of a v3 API, it’ll be 
alongside the existing v2 API, not a breaking change.

> On Jan 21, 2024, at 1:32 PM, Volkan Yazıcı  wrote:
> 
> *Abstract:* There will not be a Log4j 3 API. Both Log4j 2 and Log4j 3 will
> implement the Log4j 2 API.
> 
> In the video call we had with Ralph, Matt, Piotr, Christian, Robert, and I
> on 2024-01-19 (last Friday), we decided to proceed with this plan. In a
> nutshell,
> 
>   1. We will *make the business logic of Log4j 2 API pluggable* in a way
>   seamless to existing Log4j 2 users. Hence, everything that used to work for
>   Log4j 2 API, will work intact.
>   2. We will use this (new) pluggability to *support Log4j 2 API in Log4j
>   3*. Hence, there will not be a Log4j 3 API.
>   3. Preferably, we will *move Log4j 2 API to a separate repository* with
>   its own release lifecycle and further explore options on how to evolve it.
> 
> Let me know if you have any questions or concerns.
> 
> On Wed, Jan 17, 2024 at 5:11 PM Volkan Yazıcı  wrote:
> 
>> Given Ralph and Piotr are strongly opinionated about keeping
>> `log4j-api-3.x` binary compatible to `log4j-api-2.x`, can we not release
>> `log4j-api-3.x` in `main` and make `main` only depend on `log4j-api-2.x`
>> instead? (We can move the contents of the `spi` package in `log4j-api-3.x`
>> to a separate `log4j-spi` module in `main`.) This will make everything
>> crystal clear:
>> 
>>   - Log4j 3 is just a major improvement over the backend
>>   - Log4j 3 still supports Log4j 2 API
>>   - We can move the Log4j 2 API to a separate repository with its own
>>   release life cycle (ala SLF4J)
>>   - When time comes to make a new Log4j API where PMC agrees to make
>>   breaking changes, we can call that one Log4j 3 API
>> 
>> I would appreciate it if you can help me to understand if I am
>> missing something. Otherwise, I would like to know why we need to make a
>> major release for a project that is identical to its previous version.
>> 

Re: [log4j] Making Log4j 2 API "the Log4j API"

[Matt Sicker]

That might work, yeah.

> On Jan 17, 2024, at 12:46 PM, Volkan Yazıcı  wrote:
> 
> We can move the spi package content in main to a separate module in main.
> SPI problem is solved?
> 
> On Wed, 17 Jan 2024 at 18:33, Matt Sicker  wrote:
> 
>> I suspect this won’t work that well once I’ve implemented
>> https://github.com/apache/logging-log4j2/issues/1977 as the current
>> provider SPI is fairly lacking. It might make more sense to release the
>> main API as 3.0.0 and have 2.x depend on the updated API.
>> 
>>> On Jan 17, 2024, at 10:11 AM, Volkan Yazıcı  wrote:
>>> 
>>> Given Ralph and Piotr are strongly opinionated about keeping
>>> `log4j-api-3.x` binary compatible to `log4j-api-2.x`, can we not release
>>> `log4j-api-3.x` in `main` and make `main` only depend on `log4j-api-2.x`
>>> instead? (We can move the contents of the `spi` package in
>> `log4j-api-3.x`
>>> to a separate `log4j-spi` module in `main`.) This will make everything
>>> crystal clear:
>>> 
>>>  - Log4j 3 is just a major improvement over the backend
>>>  - Log4j 3 still supports Log4j 2 API
>>>  - We can move the Log4j 2 API to a separate repository with its own
>>>  release life cycle (ala SLF4J)
>>>  - When time comes to make a new Log4j API where PMC agrees to make
>>>  breaking changes, we can call that one Log4j 3 API
>>> 
>>> I would appreciate it if you can help me to understand if I am
>>> missing something. Otherwise, I would like to know why we need to make a
>>> major release for a project that is identical to its previous version.
>> 
>> 

Re: [log4j] Making Log4j 2 API "the Log4j API"

[Matt Sicker]

Or if we back port any of those changes I’ll propose, then perhaps we can 
continue with the API at 2.x. That does require that the API target Java 8, 
though.

> On Jan 17, 2024, at 11:32 AM, Matt Sicker  wrote:
> 
> I suspect this won’t work that well once I’ve implemented 
> https://github.com/apache/logging-log4j2/issues/1977 as the current provider 
> SPI is fairly lacking. It might make more sense to release the main API as 
> 3.0.0 and have 2.x depend on the updated API.
> 
>> On Jan 17, 2024, at 10:11 AM, Volkan Yazıcı  wrote:
>> 
>> Given Ralph and Piotr are strongly opinionated about keeping
>> `log4j-api-3.x` binary compatible to `log4j-api-2.x`, can we not release
>> `log4j-api-3.x` in `main` and make `main` only depend on `log4j-api-2.x`
>> instead? (We can move the contents of the `spi` package in `log4j-api-3.x`
>> to a separate `log4j-spi` module in `main`.) This will make everything
>> crystal clear:
>> 
>>  - Log4j 3 is just a major improvement over the backend
>>  - Log4j 3 still supports Log4j 2 API
>>  - We can move the Log4j 2 API to a separate repository with its own
>>  release life cycle (ala SLF4J)
>>  - When time comes to make a new Log4j API where PMC agrees to make
>>  breaking changes, we can call that one Log4j 3 API
>> 
>> I would appreciate it if you can help me to understand if I am
>> missing something. Otherwise, I would like to know why we need to make a
>> major release for a project that is identical to its previous version.
> 

Re: [log4j] Making Log4j 2 API "the Log4j API"

[Matt Sicker]

I suspect this won’t work that well once I’ve implemented 
https://github.com/apache/logging-log4j2/issues/1977 as the current provider 
SPI is fairly lacking. It might make more sense to release the main API as 
3.0.0 and have 2.x depend on the updated API.

> On Jan 17, 2024, at 10:11 AM, Volkan Yazıcı  wrote:
> 
> Given Ralph and Piotr are strongly opinionated about keeping
> `log4j-api-3.x` binary compatible to `log4j-api-2.x`, can we not release
> `log4j-api-3.x` in `main` and make `main` only depend on `log4j-api-2.x`
> instead? (We can move the contents of the `spi` package in `log4j-api-3.x`
> to a separate `log4j-spi` module in `main`.) This will make everything
> crystal clear:
> 
>   - Log4j 3 is just a major improvement over the backend
>   - Log4j 3 still supports Log4j 2 API
>   - We can move the Log4j 2 API to a separate repository with its own
>   release life cycle (ala SLF4J)
>   - When time comes to make a new Log4j API where PMC agrees to make
>   breaking changes, we can call that one Log4j 3 API
> 
> I would appreciate it if you can help me to understand if I am
> missing something. Otherwise, I would like to know why we need to make a
> major release for a project that is identical to its previous version.

Re: Should we keep `log4j-appserver` and `log4j-jcl` in 3.x?

[Matt Sicker]

Agreed that we can add things back later if needed. +1

> On Jan 15, 2024, at 3:02 PM, Volkan Yazıcı  wrote:
> 
> +1
> 
> Allow me to remind everyone that anything that is removed from `3.0.0` can
> be added again in a follow up minor release, if need arises. Hence, no need
> to be overcautious.
> 
> On Mon, Jan 15, 2024 at 9:50 PM Piotr P. Karwasz 
> wrote:
> 
>> Hi all,
>> 
>> While 3.x is approaching its first stable release as a meteor burning
>> through the sky, its core is getting smaller as it approaches earth.
>> 
>> That is why I would propose to remove two further modules from 3.x:
>> 
>> * The functionality of `log4j-jcl` is included in `commons-logging`
>> version 1.3.0 (and previously it was also provided by `spring-jcl`). I
>> assume that users migrating to Log4j 3.x can also migrate to
>> `commons-logging` 1.3.0.
>> 
>> * The Jetty 9.x logger, which is half of the functionality of
>> `log4j-appserver`, can be replaced by `log4j-slf4j-impl` if the users
>> are willing to migrate to Jetty 10.x. Since community support for
>> Jetty 9.x ended two years ago, I guess user will migrate to a
>> supported Jetty version first and only then think about Log4j 3.x.
>> 
>> * The Tomcat logger (the other half of `log4j-appserver`) is only
>> part of the puzzle of artifacts required to have a fully functional
>> logging system in Tomcat. I usually also needed my own artifacts from
>> copernik-eu/log4j-plugins[1] to have something comparable (but still
>> worse) than what WildFly has. I can adopt this part of
>> `log4j-appserver` into copernik-eu/log4j-plugins and make a release
>> before Log4j 3.0.0 is out. If there is any user interest in it, I can
>> "contribute back" the Tomcat logger and other components in a future
>> release.
>> 
>> What do you think?
>> 
>> [1] https://github.com/copernik-eu/log4j-plugins
>> 

Re: Website repos/branches (again)

[Matt Sicker]

I love this idea, Volkan!

> On Jan 15, 2024, at 3:40 AM, Volkan Yazıcı  wrote:
> 
> -1 on merging multiple websites to a single repository.
> 
> I think documentation should reside in the same repository where sources
> are. I already implemented this for *almost* every repository:
> 
> logging-parent
> logging-log4j-tools
> logging-log4j-transform
> logging-log4j-kotlin
> logging-log4j-scala
> logging-log4j-jakarta
> 
> There is only one left to migrate to this scheme: `logging-log4j2`, which I
> intend to do on February 9 during STF MS12. `logging-site`, `cyclonedx`,
> `activity-monitor` are exceptions, since they are only websites, not source
> code repositories. Hence, your statement of *"the way our website is
> created from multiple repos and branches is somehow incoherent"* is untrue.
> 
> Using *"site next to sources"* scheme, when one checks out a repository, it
> is crystal clear what goes where:
> 
> - the `asf-site` branch for `logging.apache.org`
> - the `asf-staging` branch for `logging.staged.apache.org`
> - the rest of the branches for the source code
> 
> This scheme would enable many benefits:
> 
>   1. Contributors, maintainers, users, etc. can easily locate the site and
>   submit changes with zero a priori knowledge. Cloned the sources? Ready to
>   go.
>   2. Since all Maven-based projects would be sharing the same site scheme,
>   we can automate the website publication in the CI-based release process.
> 
> Those who are concerned about folder structure deviation between
> `afs-{site,staging}` branches and the rest, all you need is a Git worktree:
> 
> $ git clone g...@github.com:
> apache/logging-log4j-tools.git logging-log4j-tools~main
> $ cd logging-log4j-tools~main
> $ git checkout -B asf-site origin/asf-site
> $ git checkout -B asf-staging origin/asf-staging
> $ git worktree add ../logging-log4j-tools~site asf-site
> 
> On Mon, Jan 15, 2024 at 9:34 AM Piotr P. Karwasz 
> wrote:
> 
>> Hi,
>> 
>> As we discussed yesterday the way our website is created from multiple
>> repos and branches is somehow incoherent: some parts of the website
>> have separate repos, other parts have a branch in a code repo, other
>> parts have a branch in a website repo.
>> 
>> For example:
>> 
>> * the `/log4j/jakarta` directory is published from the `asf-site`
>> branch of the `logging-log4j-jakarta` repo (which also contains the
>> code of Log4j Jakarta),
>> * the `/cyclonedx` directory is published from the `cyclonedx` branch
>> of the `logging-site` repo,
>> * the `/log4j` directory is published from the `asf-site` branch of
>> the `logging-log4j-site` repo (dedicated site repo).
>> 
>> This is getting confusing and we need some kind of list to find the
>> repo/branch combination responsible for each part of the site.
>> 
>> I think we should bring some order to it. Since not all the parts of
>> the website are connected to a code repo, my personal preference would
>> be to have everything in `logging-site`, with branches named like:
>> 
>> asf-site
>> cyclonedx/asf-site
>> log4j/asf-site
>> log4j/2.x/asf-site
>> log4j/jakarta/asf-site
>> 
>> Putting an `asf-site` (or another suffix) is a Git requirement since
>> Git does not allow us to have both `asf-site` and `asf-site/log4j`
>> branches.
>> 
>> Anyway, any other convention is good for me, as long as the rules to
>> find the correct branch are not too convoluted.
>> 
>> Piotr
>> 

Re: Spring 3 and Log4j 3

[Matt Sicker]

Oh, in that case, assuming it’s a bug in log4j 2.x as well, then yes, seems 
worth fixing.

> On Jan 10, 2024, at 12:29 PM, Piotr P. Karwasz  
> wrote:
> 
> Hi Matt,
> 
> On Wed, 10 Jan 2024 at 18:45, Matt Sicker  wrote:
>> 
>> This might affect 2.x, though it’s largely in the Spring environment 
>> property source. Given the application lifecycle there, Spring doesn’t seem 
>> to remove its own closed property sources at shutdown, hence the exception. 
>> The issue was only reported against Spring 3, though.
> 
> A lot of 2.x and 3.x here. ;-)
> What I meant is the patch should be ported to Log4j 2.x, so it can
> work with Spring 3.x. After all Spring is currently using the 2.x
> branch of Log4j.
> 
> Piotr

Re: Spring 3 and Log4j 3

[Matt Sicker]

This might affect 2.x, though it’s largely in the Spring environment property 
source. Given the application lifecycle there, Spring doesn’t seem to remove 
its own closed property sources at shutdown, hence the exception. The issue was 
only reported against Spring 3, though.

> On Jan 10, 2024, at 4:04 AM, Piotr P. Karwasz  wrote:
> 
> Hi Ralph,
> 
> On Tue, 9 Jan 2024 at 22:45, Ralph Goers  wrote:
>> 
>> FYI - in 
>> https://github.com/spring-projects/spring-boot/issues/33450#issuecomment-1883014368
>>  has confirmed that Log4j 3.0.0-beta1 now works correctly with Spring 3.x.
> 
> Are you planning to backport that patch or is it specific to 3.x?
> 
> Piotr

Re: Dependency management style

[Matt Sicker]

I like this idea.

> On Jan 8, 2024, at 4:09 AM, Piotr P. Karwasz  wrote:
> 
> Hi all,
> 
> Following the discussion in PR#2166, I would like to change the dep
> management convention I mentioned in 2022.
> 
> On Mon, 12 Sept 2022 at 09:11, Piotr P. Karwasz  
> wrote:
>> It would be also nice to synchronise the `pom.xml` of `release-2.x`
>> and `master`. Since the main `pom.xml` has about a hundred
>> dependencies, what do you think about normalizing them by:
>> 
>> * using BOMs if available (e.g. Jackson),
>> * removing the scope from ``: this way there
>> will be no difference between BOMs and explicit dependencies. It's
>> more verbose, but we won't risk having JUnit in the compile scope.
>> * removing exclusions from ``: AFAIK they are
>> ignored by Maven. Or we can keep the exclusions as a template for the
>> projects.
>> * adding a property in the main pom.xml for *each* dependency used
>> (e.g. even `slf4j-api:2.0.0` used in a single module). A convention on
>> how to name these properties would be nice too...
>> * sorting dependencies by scope (provided > compile > runtime >
>> test), artifactId and groupId.
>> 
>> Since POM style is as personal as code style, I would agree to all
>> possible conventions as long as they are coherent.
> 
> This convention seems unnatural to many of our contributors,
> especially regarding dependencies that are used by a single module.
> 
> Alternative Maven resolvers have also problems with a centralized dep
> management (cf. Issue#1983 for example).
> 
> Therefore I would like to amend the proposition above to state:
> 
> * if a dependency is used by a single module, it is only declared in
> that module,
> * if a dependency is used by more than one module, it should be
> managed in the parent.
> 
> What do you think?
> 
> Piotr
> 
> [1] https://github.com/apache/logging-log4j2/pull/2166
> [2] https://github.com/apache/logging-log4j2/issues/1983

Re: (logging-log4j2) branch main updated: Fix compilation error

[Matt Sicker]

Thanks for fixing this. This formatting change got swept up in a branch of mine 
you already reviewed.

> On Jan 2, 2024, at 10:10 AM, pkarw...@apache.org wrote:
> 
> This is an automated email from the ASF dual-hosted git repository.
> 
> pkarwasz pushed a commit to branch main
> in repository https://gitbox.apache.org/repos/asf/logging-log4j2.git
> 
> 
> The following commit(s) were added to refs/heads/main by this push:
> new 4789503a4e Fix compilation error
> 4789503a4e is described below
> 
> commit 4789503a4e31268c7a523cea7be09a53a836a9c8
> Author: Piotr P. Karwasz 
> AuthorDate: Tue Jan 2 17:09:50 2024 +0100
> 
>Fix compilation error
> ---
> .../test/java/org/apache/logging/log4j/mongodb4/MongoDb4Resolver.java   | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git 
> a/log4j-mongodb4/src/test/java/org/apache/logging/log4j/mongodb4/MongoDb4Resolver.java
>  
> b/log4j-mongodb4/src/test/java/org/apache/logging/log4j/mongodb4/MongoDb4Resolver.java
> index 03ca4188b4..640b1e8489 100644
> --- 
> a/log4j-mongodb4/src/test/java/org/apache/logging/log4j/mongodb4/MongoDb4Resolver.java
> +++ 
> b/log4j-mongodb4/src/test/java/org/apache/logging/log4j/mongodb4/MongoDb4Resolver.java
> @@ -121,7 +121,7 @@ public class MongoDb4Resolver extends 
> TypeBasedParameterResolver im
> @Override
> public MongoClient resolveParameter(ParameterContext parameterContext, 
> ExtensionContext extensionContext)
> throws ParameterResolutionException {
> -return 
> ExtensionContextAnchor.getRequiredAttribute(MongoClientHolder.class, 
> MongoClientHolder.class, extensionContext)
> +return ExtensionContextAnchor.getAttribute(MongoClientHolder.class, 
> MongoClientHolder.class, extensionContext)
> .get();
> }
> 
> 

Re: Nullability annotations

[Matt Sicker]

Nullability annotations are trivial to remove. I’ve added some basic aliases 
for them in main. As it stands, they’re copies of the four JSpecify annotations 
with those annotations applied to them along with equivalent JSR 305 
meta-annotations to make the annotations function the same in existing tooling. 
The annotations are in log4j-api (in a new package that hasn’t been published 
before), and the JSR and JSpecify annotations are explicitly excluded from the 
OSGi and JPMS module imports to ensure they’re optional (being annotations 
after all). I’ve annotated a few things, but before bothering with more 
widespread updates, I figure we should discuss what to do there before spending 
a lot of time updating things. The annotation dependencies are all “provided” 
scope (we have some of these already there for SpotBugs-related things). If we 
want to reuse the same thing across components, though, then we’d likely want a 
separate jar for this (or to directly use JSpecify as said jar).

I personally like and use nullability annotations wherever possible in 
Spinnaker as we use a combination of Java, Kotlin, and other languages, and 
Kotlin has built-in support for various nullability annotations to bridge over 
to its nullable-based type system (i.e., Kotlin can differentiate between types 
like `List?`, `List`, and `List?`, which would correspond to Java 
versions like `@Nullable List`, `List<@Nullable T>`, and `@Nullable 
List<@Nullable T>` respectively).

By adopting the JSpecify convention, it’s easy to mark either a package or 
class as `@NullUnmarked` which means everything in scope is non-null unless 
otherwise annotated as `@Nullable`.

The branch approach to adding this is extremely likely to end up in merge hell 
after short order. I’ve had this happen to my own branches where I tried to 
incrementally add nullability annotations. What I’d suggest is ensuring 
everything in log4j-api is properly annotated first, then the major plugin and 
core APIs/SPIs, and then the rest. Then we can take advantage of the static 
analysis frameworks as a regular check to ensure no new NPEs are introduced.

> On Jan 2, 2024, at 4:17 AM, Piotr P. Karwasz  wrote:
> 
> Hi all,
> 
> Matt made an interesting proposal to use JSpecify nullability
> annotations in Log4j:
> 
> https://github.com/apache/logging-parent/issues/88
> 
> I am a big fan of nullability annotations but in my professional
> experience they are worthless, unless the whole team agrees to use
> them and how to introduce them.
> 
> IMHO we should:
> 
> * create a branch that will differ from `main` only for the present
> of nullability annotations,
> * depend directly (scope `provided`) on JSpecify, although this might
> cause issue reports like this one [1],
> * only merge the changes of one module at a time (starting with `log4j-api`),
> * start by marking all fields and return types as `@Nullable` and all
> parameters as `@NonNull`,
> * only merge the change if a tool (NullAway/Checker Framework)
> guarantees that the code is NPE-free.
> * target 3.1.x as the first release with nullability annotations.
> 
> What do you think?
> 
> Piotr
> 
> [1] https://github.com/apache/logging-log4j2/issues/2144

Re: [VOTE] Release log4cxx 1.2.0-RC1

[Matt Sicker]

+1

Same checks as Volkan. Platform build info:

cmake version 3.28.1
GNU Make 3.81
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
pkg-config: 0.29.2
apr-1: 1.5.2
apr-util-1: 1.5.4
EXPAT: 2.5.0

—
Matt Sicker

> On Dec 28, 2023, at 18:00, Robert Middleton  wrote:
> 
> This is a vote to release log4cxx 1.2.0-RC1.
> 
> Please download, test, and cast your votes on the log4j developers list.
> [] +1, release the artifacts
> [] -1, don't release because...
> 
> This vote will remain open for 72 hours(or more if required).
> 
> All votes are welcome and we encourage everyone to test the release,
> but only Logging PMC votes are “officially” counted. As always, at
> least 3 +1 votes and more positive than negative votes are required.
> 
> A quick changelog is below:
> * Various build failures have been fixed
> * Added a new Hexdump utility method to dump arbitrary memory
> * Fixed a segfault when shutting down and not stopping the library
> * QStrings may now be logged directly
> * The main namespace is now configurable from log4cxx to any value
> that is desired
> 
> Tag:
> For a new copy do "git clone
> https://github.com/apache/logging-log4cxx.git; and then "git checkout
> tags/v1.2.0-RC1"
> For an existing working copy, do "git pull" and then "git checkout
> tags/v1.2.0-RC1"
> 
> Web site: https://logging.staged.apache.org/log4cxx/latest_stable/
> 
> Distribution archives: https://dist.apache.org/repos/dist/dev/logging/log4cxx/
> 
> Building directions are on the website(under 'Development').  Note
> that APR is required to build(as well as boost if your compiler does
> not support C++17).
> 
> -Robert Middleton

Re: Reproducibility checks

[Matt Sicker]

I’ve been using the reproducibility check provided in the vote email, though I 
haven’t been able to reproduce a build 100% so far due to some 
dependency-convergence issues in the Cassandra plugin (go figure, complex 
dependency tree there), but I’ve mentioned something about this in the vote 
emails.
—
Matt Sicker

> On Dec 27, 2023, at 08:10, Piotr P. Karwasz  wrote:
> 
> Hi Gary,
> 
> On Wed, 27 Dec 2023 at 13:58, Gary Gregory  wrote:
>> Please include whatever instructions you want folks to run in the vote
>> email to prove reproducibility. Then at least we can agree on what it
>> means to do the reproducibility check and when it passes or fails,
>> assuming it's a binary property.
> 
> The steps to check reproducibility are in the vote e-mail:
> 
># Verify reproduciblity
>umask 0022
>unzip *-src.zip -d src
>cd src
>export 
> NEXUS_REPO=https://repository.apache.org/content/repositories/orgapachelogging-1254
>sh mvnw -Prelease verify artifact:compare -Dreference.repo=$NEXUS_REPO
> 
>> A long-standing pet peeve of mine is PMC members (in many projects,
>> I'm not singling out Log4j here) that vote on a release candidate
>> without stating _what_ they did to check the viability of said
>> release.
>> 
>> If this matters, it should be an Apache requirement, which it is not ATM 
>> AFAIK.
> 
> I agree, there should be some minimal best practices for release
> verification. If Apache Security does not want ATM to set some
> guidelines, I wouldn't mind if Apache Commons did.
> 
> BTW I cited your vote mail in this thread, mostly because you always
> describe what you are checking.
> From the votes of some PMC members it is impossible to deduce what was 
> checked.
> 
> Piotr

Re: [VOTE] Release Apache Log4j 2.22.1 RC1

[Matt Sicker]

While there appears to be some issues with the reproducibility of the build, it 
appears to be inconsequential.

+1
—
Matt Sicker

> On Dec 22, 2023, at 12:00, Piotr P. Karwasz  wrote:
> 
> This is a vote to release the Apache Log4j 2.22.1.
> 
> Website: https://logging.staged.apache.org/log4j/2.x/
> GitHub: https://github.com/apache/logging-log4j2
> Commit: 8469975a4f2b1f8f1bd4f25ca6d1989a52aefc1b
> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j
> Nexus: 
> https://repository.apache.org/content/repositories/orgapachelogging-1254
> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
> 
> Please download, test, and cast your votes on this mailing list.
> 
> [ ] +1, release the artifacts
> [ ] -1, don't release, because...
> 
> This vote is open for 72 hours and will pass unless getting a
> net negative vote count. All votes are welcome and we encourage
> everyone to test the release, but only the Logging Services PMC
> votes are officially counted.
> 
> == Review Kit
> 
> The minimum set of steps needed to review the uploaded distribution
> files in the Subversion repository can be summarized as follows:
> 
># Check out the distribution
>wget --recursive --no-parent --no-host-directories --cut-dirs=5
> https://dist.apache.org/repos/dist/dev/logging/log4j
> 
># Verify checksums
>sha512sum --check *.sha512
> 
># Verify signatures
>wget -O - https://downloads.apache.org/logging/KEYS | gpg --import
>for sigFile in *.asc; do gpg --verify $sigFile; done
> 
># Verify reproduciblity
>umask 0022
>unzip *-src.zip -d src
>cd src
>export 
> NEXUS_REPO=https://repository.apache.org/content/repositories/orgapachelogging-1254
>sh mvnw -Prelease verify artifact:compare -Dreference.repo=$NEXUS_REPO
> 
> == Release Notes
> 
> This release contains only dependency upgrades and bug fixes, which do
> not change the behavior of the artifacts.
> 
> While maintaining compatibility with Java 8, the artifacts in this
> release where generated using JDK 17, unlike version `2.22.0` that
> used JDK 11.
> 
> 
> [#release-notes-2-22-1-fixed]
> === Fixed
> 
> * Mark `JdkMapAdapterStringMap` as frozen if map is immutable. (#2098)
> * Fix NPE in `CloseableThreadContext`. (#1426)
> * Use the module name of Conversant Media Disruptor from version
> `1.2.16+` of the library.
> * Fix NPE in `RollingFileManager`. (#1645)
> * Fix `log4j-to-slf4j` JPMS and OSGi descriptors. (#1983)
> * Workaround a Coursier/Ivy dependency resolution bug affecting
> `log4j-slf4j-impl` and `log4j-mongodb3`. (#2065)
> 
> [#release-notes-2-22-1-updated]
> === Updated
> 
> * Bumped the minimum Java version required for the build to Java 17.
> Runtime requirements remain unchanged. (#2021)
> * Update `com.github.luben:zstd-jni` to version `1.5.5-11` (#2030)
> * Update `com.google.guava:guava` to version `33.0.0-jre` (#2110)
> * Update `commons-codec:commons-codec` to version `1.16.0` (#2042)
> * Update `commons-io:commons-io` to version `2.15.1` (#2034)
> * Update `commons-logging:commons-logging` to version `1.3.0` (#2050)
> * Update `io.netty:netty-bom` to version `4.1.104.Final` (#2095)
> * Update `org.apache.commons:commons-compress` to version `1.25.0` (#2045)
> * Update `org.apache.commons:commons-dbcp2` to version `2.11.0` (#2048)
> * Update `org.apache.commons:commons-lang3` to version `3.14.0` (#2047)
> * Update `org.apache.commons:commons-pool2` to version `2.12.0` (#2057)
> * Update `org.apache.kafka:kafka-clients` to version `3.6.1` (#2068)
> * Update `org.apache.logging:logging-parent` to version `10.5.0` (#2119)
> * Update `org.jctools:jctools-core` to version `4.0.2` (#1984)
> * Update `org.springframework.boot:spring-boot` to version `2.7.18` (#1998)
> * Update `org.springframework.cloud:spring-cloud-dependencies` to
> version `2021.0.9` (#2109)

Re: [VOTE] Release Apache Log4j Kotlin API 1.4.0

[Matt Sicker]

And my +1.

With this, the vote passes with 4 +1 votes from myself, Volkan, Piotr, and Gary.
—
Matt Sicker

> On Dec 24, 2023, at 09:57, Gary Gregory  wrote:
> 
> +1
> - Tested src zip file
> - ASC OK
> - SHA512 OK
> - Building 'mvn clean verify' OK
> - Using:
> Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
> Maven home: /usr/local/Cellar/maven/3.9.6/libexec
> Java version: 17.0.9, vendor: Homebrew, runtime:
> /usr/local/Cellar/openjdk@17/17.0.9/libexec/openjdk.jdk/Contents/Home
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "14.2.1", arch: "x86_64", family: "mac"
> Darwin  23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:10
> PST 2023; root:xnu-10002.61.3~2/RELEASE_X86_64 x86_64
> 
> Gary
> 
> On Wed, Dec 20, 2023 at 1:08 PM Matt Sicker  wrote:
>> 
>> This is a vote to release the Apache Log4j Kotlin API 1.4.0.
>> 
>> Website: https://logging.staged.apache.org/log4j/kotlin
>> GitHub: https://github.com/apache/logging-log4j-kotlin
>> Commit: ee2d4a8efa16bb2223252329531c94e98ed6d1e6
>> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j-kotlin
>> Nexus: 
>> https://repository.apache.org/content/repositories/orgapachelogging-1113
>> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
>> 
>> Please download, test, and cast your votes on this mailing list.
>> 
>> [ ] +1, release the artifacts
>> [ ] -1, don't release, because...
>> 
>> This vote is open for 72 hours and will pass unless getting a
>> net negative vote count. All votes are welcome and we encourage
>> everyone to test the release, but only the Logging Services PMC
>> votes are officially counted. At least 3 +1 votes and more
>> positive than negative votes are required.
>> 
>> === Review kit
>> 
>> The minimum set of steps needed to review the uploaded distribution
>> files in the Subversion repository can be summarized as follows:
>> 
>># Check out the distribution
>>svn co https://dist.apache.org/repos/... && cd $_
>> 
>># Verify checksums
>>shasum --check *.sha512
>> 
>># Verify signatures
>>wget -O - https://downloads.apache.org/logging/KEYS | gpg --import
>>for sigFile in *.asc; do gpg --verify $sigFile; done
>> 
>># Verify reproduciblity
>>umask 0022
>>unzip *-src.zip -d src
>>cd src
>>export NEXUS_REPO=https://repository.apache.org/content/...
>>sh mvnw -Prelease verify artifact:compare -Dreference.repo=$NEXUS_REPO
>> 
>> === Release notes
>> 
>> 
>> This minor release fixes incorrect coroutine context map and stack.
>> 
>> 
>>  Added
>> 
>> * Started generating CycloneDX SBOM with the recent update of 
>> `logging-parent` to version `10.2.0`
>> 
>>  Changed
>> 
>> * Coroutine context is not cleared properly, only appended to (#54)
>> * Update `org.apache.logging:logging-parent` to version `10.2.0`
>> * Update `org.apache.logging.log4j:log4j-bom` to version `2.22.0` (#52)
>> * Update `org.apache.logging:logging-parent` to version `10.4.0` (#53)
>> * Update `org.codehaus.mojo:build-helper-maven-plugin` to version `3.5.0` 
>> (#51)
>> * Update `org.codehaus.mojo:exec-maven-plugin` to version `3.1.1` (#50)
>> * Update `org.junit:junit-bom` to version `5.10.1` (#49)

Re: [VOTE] Release Apache Log4j 3.0.0-beta1 (RC2)

[Matt Sicker]

+1

I’ll note that the reproducibility check fails on log4j-bom this time, but not 
a blocker.

> On Dec 19, 2023, at 3:00 PM, Volkan Yazıcı  wrote:
> 
> This is a vote to release the Apache Log4j 3.0.0-beta1 RC2.
> 
> Website: https://logging.staged.apache.org/log4j/3.x
> GitHub: https://github.com/apache/logging-log4j2
> Commit: 416cd4dcf419b59c88054d2001d34c7fec010560
> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j
> Nexus: 
> https://repository.apache.org/content/repositories/orgapachelogging-1252
> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
> 
> Please download, test, and cast your votes on this mailing list.
> 
> [ ] +1, release the artifacts
> [ ] -1, don't release, because...
> 
> This vote is open for 72 hours and will pass unless getting a
> net negative vote count. All votes are welcome and we encourage
> everyone to test the release, but only the Logging Services PMC
> votes are officially counted.
> 
> PLEASE USE THIS THREAD ONLY FOR VOTING +1 OR -1. IF YOU HAVE THOUGHTS,
> CONCERNS, QUESTIONS, ETC. SHARE THEM ELSEWHERE. THIS IS A BETA
> RELEASE. WE INTEND TO HAVE SEVERAL OTHER BETA RELEASES. THIS IS NOT
> THE CONCLUSIVE `3.0.0` RELEASE.
> 
> == Review Kit
> 
> The minimum set of steps needed to review the uploaded distribution
> files in the Subversion repository can be summarized as follows:
> 
># Check out the distribution
>svn co https://dist.apache.org/repos/... && cd $_
> 
># Verify checksums
>shasum --check *.sha512
> 
># Verify signatures
>wget -O - https://downloads.apache.org/logging/KEYS | gpg --import
>for sigFile in *.asc; do gpg --verify $sigFile; done
> 
># Verify reproduciblity
>umask 0022
>unzip *-src.zip -d src
>cd src
>export NEXUS_REPO=https://repository.apache.org/content/...
>sh mvnw -Prelease verify artifact:compare -Dreference.repo=$NEXUS_REPO
> 
> == Release Notes
> 
> This is the first beta release of the upcoming major release, i.e., `3.0.0`.
> 
> === Added
> 
> * Add annotations for nullability. (LOG4J2-1477)
> * Remove deprecated code. (LOG4J2-2493)
> * Add a more generalized dependency injection system to plugins
> inspired by JSR 330. (LOG4J2-2803)
> * Add and enhance structured properties for per-context settings
> outside configuration files. (LOG4J2-3299[LOG4J2-3299], #1473)
> * Automate artifact publishing and release preparation. (LOG4J2-3466)
> * Add support for dependency injection of plugins into container types
> such as `Optional`, `Collection`, `Set`, `Stream`,
> `List`, and `Map`. (LOG4J2-3496)
> * Add support for `ConstraintValidator` in plugin classes. (LOG4J2-3497)
> 
> === Changed
> 
> * Remove liquibase-log4j2 maven module (#1193)
> * Make the output of annotation processing reproducible. (#1520)
> * Replace `synchronized` blocks with locks for improved performance
> with virtual threads. (#1532)
> * Removes additional `isFiltered` checks in `AsyncLoggerConfig`. (#1550)
> * Ignore exceptions thrown by PropertySources. Eliminate
> ClassCastException when SimpleLoggerContext is used.
> (spring-projects/spring-boot#33450, #1799)
> * Update `com.lmax:disruptor` to version `4.0.0` (#1829)
> * Migrate most tests to JUnit 5. This includes a more powerful set of
> test extensions. (LOG4J2-2653)
> * Make Log4j use its own BOM. (LOG4J2-3511)
> * Change encoding of HTTP Basic Authentication to UTF-8. (#1970)
> * Upgraded the required compiler version to Java 17
> * Upgraded the required runtime version to Java 17
> * Update `actions/checkout` to version `4.1.1` (#1869)
> * Update `actions/setup-java` to version `3.13.0` (#1809)
> * Update `actions/setup-python` to version `4.7.1` (#1831)
> * Update `ch.qos.logback:logback-classic` to version `1.4.14` (#2028)
> * Update `com.datastax.cassandra:cassandra-driver-core` to version
> `3.11.5` (#1889)
> * Update `com.fasterxml.jackson:jackson-bom` to version `2.16.0` (#1974)
> * Update `com.github.luben:zstd-jni` to version `1.5.5-11` (#2032)
> * Update `com.github.spotbugs:spotbugs-maven-plugin` to version
> `4.7.3.6` (#1879)
> * Update `com.github.tomakehurst:wiremock-jre8` to version `2.35.1` (#1765)
> * Update 
> `com.google.code.java-allocation-instrumenter:java-allocation-instrumenter`
> to version `3.3.4` (#2102)
> * Update `com.google.errorprone:error_prone_core` to version `2.23.0` (#1871)
> * Update `com.google.guava:guava-testlib` to version `32.1.3-jre` (#1934)
> * Update `com.h2database:h2` to version `2.2.224` (#1917)
> * Update `commons-codec:commons-codec` to version `1.16.0` (#2054)
> * Update `commons-io:commons-io` to version `2.15.1` (#2035)
> * Update `commons-logging:commons-logging` to version `1.3.0` (#2046)
> * Update `de.flapdoodle.reverse:de.flapdoodle.reverse` to version
> `1.7.2` (#2000)
> * Update `io.netty:netty-bom` to version `4.1.104.Final` (#2097)
> * Update `net.java.dev.jna:jna` to version `5.14.0` (#2082)
> * Update `org.apache.aries.spifly:org.apache.aries.spifly.dynamic.bundle`
> to 

Re: [VOTE] Release Apache Log4j Kotlin API 1.4.0

[Matt Sicker]

Oops, thanks for the correction!

> On Dec 20, 2023, at 3:20 PM, Volkan Yazıcı  wrote:
> 
> Before Piotr and others start voting -1, let me make a correction: Nexus
> URL is incorrect. The correct one is:
> https://repository.apache.org/content/repositories/orgapachelogging-1251
> (Matt, there is a warning about this in the release instructions. )
> 
> On Wed, Dec 20, 2023 at 7:08 PM Matt Sicker  wrote:
> 
>> This is a vote to release the Apache Log4j Kotlin API 1.4.0.
>> 
>> Website: https://logging.staged.apache.org/log4j/kotlin
>> GitHub: https://github.com/apache/logging-log4j-kotlin
>> Commit: ee2d4a8efa16bb2223252329531c94e98ed6d1e6
>> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j-kotlin
>> Nexus:
>> https://repository.apache.org/content/repositories/orgapachelogging-1113
>> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
>> 
>> Please download, test, and cast your votes on this mailing list.
>> 
>> [ ] +1, release the artifacts
>> [ ] -1, don't release, because...
>> 
>> This vote is open for 72 hours and will pass unless getting a
>> net negative vote count. All votes are welcome and we encourage
>> everyone to test the release, but only the Logging Services PMC
>> votes are officially counted. At least 3 +1 votes and more
>> positive than negative votes are required.
>> 
>> === Review kit
>> 
>> The minimum set of steps needed to review the uploaded distribution
>> files in the Subversion repository can be summarized as follows:
>> 
>># Check out the distribution
>>svn co https://dist.apache.org/repos/... && cd $_
>> 
>># Verify checksums
>>shasum --check *.sha512
>> 
>># Verify signatures
>>wget -O - https://downloads.apache.org/logging/KEYS | gpg --import
>>for sigFile in *.asc; do gpg --verify $sigFile; done
>> 
>># Verify reproduciblity
>>umask 0022
>>unzip *-src.zip -d src
>>cd src
>>export NEXUS_REPO=https://repository.apache.org/content/...
>>sh mvnw -Prelease verify artifact:compare -Dreference.repo=$NEXUS_REPO
>> 
>> === Release notes
>> 
>> 
>> This minor release fixes incorrect coroutine context map and stack.
>> 
>> 
>>  Added
>> 
>> * Started generating CycloneDX SBOM with the recent update of
>> `logging-parent` to version `10.2.0`
>> 
>>  Changed
>> 
>> * Coroutine context is not cleared properly, only appended to (#54)
>> * Update `org.apache.logging:logging-parent` to version `10.2.0`
>> * Update `org.apache.logging.log4j:log4j-bom` to version `2.22.0` (#52)
>> * Update `org.apache.logging:logging-parent` to version `10.4.0` (#53)
>> * Update `org.codehaus.mojo:build-helper-maven-plugin` to version `3.5.0`
>> (#51)
>> * Update `org.codehaus.mojo:exec-maven-plugin` to version `3.1.1` (#50)
>> * Update `org.junit:junit-bom` to version `5.10.1` (#49)

[VOTE] Release Apache Log4j Kotlin API 1.4.0

[Matt Sicker]

This is a vote to release the Apache Log4j Kotlin API 1.4.0.

Website: https://logging.staged.apache.org/log4j/kotlin
GitHub: https://github.com/apache/logging-log4j-kotlin
Commit: ee2d4a8efa16bb2223252329531c94e98ed6d1e6
Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j-kotlin
Nexus: https://repository.apache.org/content/repositories/orgapachelogging-1113
Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0

Please download, test, and cast your votes on this mailing list.

[ ] +1, release the artifacts
[ ] -1, don't release, because...

This vote is open for 72 hours and will pass unless getting a
net negative vote count. All votes are welcome and we encourage
everyone to test the release, but only the Logging Services PMC
votes are officially counted. At least 3 +1 votes and more
positive than negative votes are required.

=== Review kit

The minimum set of steps needed to review the uploaded distribution
files in the Subversion repository can be summarized as follows:

# Check out the distribution
svn co https://dist.apache.org/repos/... && cd $_

# Verify checksums
shasum --check *.sha512

# Verify signatures
wget -O - https://downloads.apache.org/logging/KEYS | gpg --import
for sigFile in *.asc; do gpg --verify $sigFile; done

# Verify reproduciblity
umask 0022
unzip *-src.zip -d src
cd src
export NEXUS_REPO=https://repository.apache.org/content/...
sh mvnw -Prelease verify artifact:compare -Dreference.repo=$NEXUS_REPO

=== Release notes


This minor release fixes incorrect coroutine context map and stack.


 Added

* Started generating CycloneDX SBOM with the recent update of `logging-parent` 
to version `10.2.0`

 Changed

* Coroutine context is not cleared properly, only appended to (#54)
* Update `org.apache.logging:logging-parent` to version `10.2.0`
* Update `org.apache.logging.log4j:log4j-bom` to version `2.22.0` (#52)
* Update `org.apache.logging:logging-parent` to version `10.4.0` (#53)
* Update `org.codehaus.mojo:build-helper-maven-plugin` to version `3.5.0` (#51)
* Update `org.codehaus.mojo:exec-maven-plugin` to version `3.1.1` (#50)
* Update `org.junit:junit-bom` to version `5.10.1` (#49)

Re: [VOTE] Release Apache Log4j 3.0.0-beta1

[Matt Sicker]

+1

> On Dec 18, 2023, at 2:45 PM, Matt Sicker  wrote:
> 
> We can review this without doing a reproducibility check. The signatures of 
> everything look good; just doing some final validations before voting.
> 
>> On Dec 18, 2023, at 6:38 AM, Gary Gregory  wrote:
>> 
>> Since Piotr asked for a fix and Matt can reproduce the issue, I'm not going
>> to take the time.to review this RC.
>> 
>> Gary
>> 
>> On Mon, Dec 18, 2023, 3:21 AM Volkan Yazıcı  wrote:
>> 
>>> Even though the vote was intended for 72 hours, it has been 5 days and
>>> there hasn't been any official votes on the release. I will wait for
>>> another 24 hours and cancel the release. I will appreciate PMC members'
>>> participation in the voting.
>>> 
>>> On Wed, Dec 13, 2023 at 4:26 PM Volkan Yazıcı  wrote:
>>> 
>>>> This is a vote to release the Apache Log4j 3.0.0-beta1.
>>>> 
>>>> Website: https://logging.staged.apache.org/log4j
>>>> GitHub: https://github.com/apache/logging-log4j2
>>>> Commit: c5dbdcfeb0216e1e3e333436e9b4d04cc3b8e6fd
>>>> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j
>>>> Nexus:
>>>> https://repository.apache.org/content/repositories/orgapachelogging-1246
>>>> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
>>>> 
>>>> Please download, test, and cast your votes on this mailing list.
>>>> 
>>>> [ ] +1, release the artifacts
>>>> [ ] -1, don't release, because...
>>>> 
>>>> This vote is open for 72 hours and will pass unless getting a
>>>> net negative vote count. All votes are welcome and we encourage
>>>> everyone to test the release, but only the Logging Services PMC
>>>> votes are officially counted.
>>>> 
>>>> == Review Kit
>>>> 
>>>> The minimum set of steps needed to review the uploaded distribution
>>>> files in the Subversion repository can be summarized as follows:
>>>> 
>>>>   # Check out the distribution
>>>>   svn co https://dist.apache.org/repos/... && cd $_
>>>> 
>>>>   # Verify checksums
>>>>   shasum --check *.sha512
>>>> 
>>>>   # Verify signatures
>>>>   wget -O - https://downloads.apache.org/logging/KEYS | gpg --import
>>>>   for sigFile in *.asc; do gpg --verify $sigFile; done
>>>> 
>>>>   # Verify reproduciblity
>>>>   umask 0022
>>>>   unzip *-src.zip -d src
>>>>   cd src
>>>>   export NEXUS_REPO=https://repository.apache.org/content/...
>>>>   sh mvnw -Prelease \
>>>>   verify artifact:compare \
>>>>   -Dreference.repo=$NEXUS_REPO \
>>>>   -Dcyclonedx.skip
>>>> 
>>>> Some SBOM discrepancy is causing reproducibility mismatch, hence the
>>>> `-Dcyclonedx.skip`. Since `2.x` and `main` are greatly diverged, I
>>> couldn't
>>>> figure out the missing piece yet.
>>>> 
>>>> == Release Notes
>>>> 
>>>> This is the first beta release of the upcoming major release, i.e.,
>>>> `3.0.0`.
>>>> 
>>>> === Added
>>>> 
>>>> * Add annotations for nullability. (LOG4J2-1477)
>>>> * Remove deprecated code. (LOG4J2-2493)
>>>> * Add a more generalized dependency injection system to plugins inspired
>>>> by JSR 330. (LOG4J2-2803)
>>>> * Add and enhance structured properties for per-context settings outside
>>>> configuration files. (1473)
>>>> * Automate artifact publishing and release preparation. (LOG4J2-3466)
>>>> * Add support for dependency injection of plugins into container types
>>>> such as `Optional`, `Collection`, `Set`, `Stream`, `List`,
>>>> and `Map`. (LOG4J2-3496)
>>>> * Add support for `ConstraintValidator` in plugin classes. (LOG4J2-3497)
>>>> 
>>>> === Changed
>>>> 
>>>> * Remove liquibase-log4j2 maven module (#1193)
>>>> * Make the output of annotation processing reproducible. (#1520)
>>>> * Replace `synchronized` blocks with locks for improved performance with
>>>> virtual threads. (#1532)
>>>> * Removes additional `isFiltered` checks in `AsyncLoggerConfig`. (#1550)
>>>> * Ignore exceptions thrown by PropertySources. Eliminate
>>>> ClassCastException when SimpleLoggerContext is used.
>>>> (spring-pro

Re: [VOTE] Release Apache Log4j 3.0.0-beta1

[Matt Sicker]

We can review this without doing a reproducibility check. The signatures of 
everything look good; just doing some final validations before voting.

> On Dec 18, 2023, at 6:38 AM, Gary Gregory  wrote:
> 
> Since Piotr asked for a fix and Matt can reproduce the issue, I'm not going
> to take the time.to review this RC.
> 
> Gary
> 
> On Mon, Dec 18, 2023, 3:21 AM Volkan Yazıcı  wrote:
> 
>> Even though the vote was intended for 72 hours, it has been 5 days and
>> there hasn't been any official votes on the release. I will wait for
>> another 24 hours and cancel the release. I will appreciate PMC members'
>> participation in the voting.
>> 
>> On Wed, Dec 13, 2023 at 4:26 PM Volkan Yazıcı  wrote:
>> 
>>> This is a vote to release the Apache Log4j 3.0.0-beta1.
>>> 
>>> Website: https://logging.staged.apache.org/log4j
>>> GitHub: https://github.com/apache/logging-log4j2
>>> Commit: c5dbdcfeb0216e1e3e333436e9b4d04cc3b8e6fd
>>> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j
>>> Nexus:
>>> https://repository.apache.org/content/repositories/orgapachelogging-1246
>>> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
>>> 
>>> Please download, test, and cast your votes on this mailing list.
>>> 
>>> [ ] +1, release the artifacts
>>> [ ] -1, don't release, because...
>>> 
>>> This vote is open for 72 hours and will pass unless getting a
>>> net negative vote count. All votes are welcome and we encourage
>>> everyone to test the release, but only the Logging Services PMC
>>> votes are officially counted.
>>> 
>>> == Review Kit
>>> 
>>> The minimum set of steps needed to review the uploaded distribution
>>> files in the Subversion repository can be summarized as follows:
>>> 
>>># Check out the distribution
>>>svn co https://dist.apache.org/repos/... && cd $_
>>> 
>>># Verify checksums
>>>shasum --check *.sha512
>>> 
>>># Verify signatures
>>>wget -O - https://downloads.apache.org/logging/KEYS | gpg --import
>>>for sigFile in *.asc; do gpg --verify $sigFile; done
>>> 
>>># Verify reproduciblity
>>>umask 0022
>>>unzip *-src.zip -d src
>>>cd src
>>>export NEXUS_REPO=https://repository.apache.org/content/...
>>>sh mvnw -Prelease \
>>>verify artifact:compare \
>>>-Dreference.repo=$NEXUS_REPO \
>>>-Dcyclonedx.skip
>>> 
>>> Some SBOM discrepancy is causing reproducibility mismatch, hence the
>>> `-Dcyclonedx.skip`. Since `2.x` and `main` are greatly diverged, I
>> couldn't
>>> figure out the missing piece yet.
>>> 
>>> == Release Notes
>>> 
>>> This is the first beta release of the upcoming major release, i.e.,
>>> `3.0.0`.
>>> 
>>> === Added
>>> 
>>> * Add annotations for nullability. (LOG4J2-1477)
>>> * Remove deprecated code. (LOG4J2-2493)
>>> * Add a more generalized dependency injection system to plugins inspired
>>> by JSR 330. (LOG4J2-2803)
>>> * Add and enhance structured properties for per-context settings outside
>>> configuration files. (1473)
>>> * Automate artifact publishing and release preparation. (LOG4J2-3466)
>>> * Add support for dependency injection of plugins into container types
>>> such as `Optional`, `Collection`, `Set`, `Stream`, `List`,
>>> and `Map`. (LOG4J2-3496)
>>> * Add support for `ConstraintValidator` in plugin classes. (LOG4J2-3497)
>>> 
>>> === Changed
>>> 
>>> * Remove liquibase-log4j2 maven module (#1193)
>>> * Make the output of annotation processing reproducible. (#1520)
>>> * Replace `synchronized` blocks with locks for improved performance with
>>> virtual threads. (#1532)
>>> * Removes additional `isFiltered` checks in `AsyncLoggerConfig`. (#1550)
>>> * Ignore exceptions thrown by PropertySources. Eliminate
>>> ClassCastException when SimpleLoggerContext is used.
>>> (spring-projects/spring-boot#33450, #1799)
>>> * Update `com.lmax:disruptor` to version `4.0.0` (#1829)
>>> * Migrate most tests to JUnit 5. This includes a more powerful set of
>> test
>>> extensions. (LOG4J2-2653)
>>> * Make Log4j use its own BOM. (LOG4J2-3511)
>>> * Change encoding of HTTP Basic Authentication to UTF-8. (#1970)
>>> * Upgraded the required compiler version to Java 17
>>> * Upgraded the required runtime version to Java 17
>>> * Update `actions/checkout` to version `4.1.1` (#1869)
>>> * Update `actions/setup-java` to version `3.13.0` (#1809)
>>> * Update `actions/setup-python` to version `4.7.1` (#1831)
>>> * Update `ch.qos.logback:logback-classic` to version `1.4.14` (#2028)
>>> * Update `com.datastax.cassandra:cassandra-driver-core` to version
>>> `3.11.5` (#1889)
>>> * Update `com.fasterxml.jackson:jackson-bom` to version `2.16.0` (#1974)
>>> * Update `com.github.luben:zstd-jni` to version `1.5.5-11` (#2032)
>>> * Update `com.github.spotbugs:spotbugs-maven-plugin` to version `4.7.3.6`
>>> (#1879)
>>> * Update `com.github.tomakehurst:wiremock-jre8` to version `2.35.1`
>> (#1765)
>>> * Update `com.google.errorprone:error_prone_core` to version `2.23.0`
>>> (#1871)
>>> * Update 

Re: [log4j] Revamping Log4j website & manual

[Matt Sicker]

I’d be interesting in helping with this, particularly around plugin development 
docs which are somewhat anemic. I’ve got several things to document in a 
non-API-doc fashion from the new DI system and such, too.

> On Dec 18, 2023, at 1:13 PM, Volkan Yazıcı  wrote:
> 
> *TLDR:* Log4j websites & manual will be rewritten. Feedback and support are
> more than welcome.
> 
> Starting from February, Christian, Piotr, and I will be working on
> reorganizing the Log4j websites (Log4j, Log4j Scala, Log4j Tools, etc.) and
> rewriting the Log4j manual. Many implementation details are still yet to be
> decided and we look forward to pursuing this big undertaking with the help
> of other PMC members and our community.
> 
> I want to kick-off this project using the following agenda:
> 
>   1. An email thread on *website & manual reorganization* – What are the
>   current shortcomings? Shall we have global support, security, etc. pages?
>   Do we need separate pages for distribution details of each project? Where
>   shall we place `logging-parent`? What about release instructions?
>   2. An email thread on *tooling* – What are the requirements? What are
>   the available solutions in the market?
>   3. A *video call* for resolution – Q Fleshing out a plan forward.
>   4. An email sharing the *conclusive plan* in detail.
> 
> If you have general feedback that doesn't fit into aforementioned
> reorganization- and tooling-related discussions, please go ahead and share
> them in response to this post.

Re: All-in-one site documentation

[Matt Sicker]

This is pretty cool!

> On Dec 18, 2023, at 11:37 AM, Gary D. Gregory  wrote:
> 
> Hi All,
> 
> A while back, I mentioned that we could (not should) have our documentation 
> generated within Javadoc. Here is an example from another Apache project I 
> participate in: 
> 
> https://juneau.apache.org/site/apidocs-9.0.1/index.html
> 
> I'm late to Juneau and I don't know the ins and out of how it's made, but it 
> looks good nonetheless IMO.
> 
> Gary

Re: [VOTE] Release Apache Log4j 3.0.0-beta1

[Matt Sicker]

> On Dec 14, 2023, at 3:08 PM, Volkan Yazıcı  wrote:
> 
> Given we will have several other betas before a GA release, do these issues
> really constitute a serious blocker? Or they can be addressed in the next
> beta?
> 

I can’t run the release verification commands you provided without this fix as 
the build fails after log4j-core-tests.

Re: [log4j-kotlin] Next release (`1.3.1`-vs-`1.4.0`)

[Matt Sicker]

Thanks for the info! I don’t think you’re being pedantic here; the uniform 
release process is very important to keeping all our components healthy.

> On Dec 7, 2023, at 4:54 AM, Volkan Yazıcı  wrote:
> 
> Raman, Matt, great that you want to release the next version of the Log4j
> Kotlin! Thanks for your time and effort.
> 
> I see that you forked the `release/1.3.1` branch from the `rel/1.3.0` tag,
> though this misses some minor, but important  changes (e.g., SBOM & VDR!)
> from `main`. I suggest we fork from `main` and release `1.4.0` instead.
> 
> There are some chores that have been waiting for the next release to be
> performed. I will share them below and appreciate it if you can carry them
> out.
> 
> *Add `project.build.outputTimestamp` Maven property*
> 
> Please copy this property (incl. the large comment block preceding it!)
> from the `2.x` branch of the `logging-log4j2` repository and incorporate it
> into `/pom.xml` of `log4j-kotlin`. Try to place it exactly at the same
> location: right after `revision`.
> 
> I might sound pedantic about this and I am. I try really hard to make all
> `pom.xml`s look alike to avoid the surprise factor maintainers suffer from
> while jumping from one project to another. Not to mention this also helps
> us to carry out automation, e.g., `project.build.outputTimestamp` is
> replaced by `logging-parent` CI magic during release.
> 
> *Remove `cyclonedx-maven-plugin` override*
> 
> Remove the `cyclonedx-maven-plugin` override (and the comment block
> preceding it) from `/pom.xml`. This is not necessary anymore with the
> `logging-parent` version `0.4.0` release.
> 
> *Changelog cleanup*
> 
> Make sure `src/changelog/.1.x.x` contents look good. You can verify this by
> skimming through `src/site` after issuing a `./mvnw process-sources`.
> 
> Could you update all changelog XSDs to version `0.1.2` too, please?
> 
> I would also appreciate it if you can remove all `author` elements from
> `src/changelog/*/*.xml` files. They are neither rendered by the template,
> nor provide information that one cannot obtain from the associated
> commit/ticket.
> 
> *Generated emails*
> 
> Please don't send generated emails verbatim! Nexus URLs need to be manually
> filled in and there are always some minor formatting details to fix in the
> text.

Re: [log4j] Upgrade `main` compiler and runtime to Java 17

[Matt Sicker]

I think it makes sense to default to a supported version, and we can take a 
manual look at what’s required in updates as new Java versions come out rather 
than letting random users find out by accident.

> On Dec 5, 2023, at 6:49 AM, Volkan Yazıcı  wrote:
> 
> I disagree with having an open-ended compiler baseline requirement due to
> 
>   1. It is not a minor task to bump the compiler version. The Java 17
>   upgrade task was on hold already for several months due to its intricacies.
>   As a matter of fact, you were the one who dropped the ball after the Java
>   11 upgrade. (It was a rightful decision and I am glad you did the hard work
>   of Java 11 upgrade. )
>   2. Fixing the baseline will keep everybody (maintainers, individual
>   contributors, etc.) on the same page due to no moving parts. Less "Are you
>   using Java 17?"-kind questions while troubleshooting.
>   3. Those who want to test Log4j against new JDKs still can easily:
>   `-Denforcer.skip`
>   4. `.java-version` doesn't support ranges (AFAIK)
>   5. I don't find it annoying. On the contrary, several other major
>   projects adopt this pattern: Spring Boot
>   
> ,
>   JUnit
>   
> ,
>   JCTools ,
>   Guava , etc.
>   6. I don't see a compelling use case.
> 
> All that said, I am not strongly opinionated about this and I wouldn't stop
> you from replacing `[17,18)` with `[17,)` in `/pom.xml`.
> 
> On Tue, Dec 5, 2023 at 1:19 PM Piotr P. Karwasz 
> wrote:
> 
>> Hi Volkan,
>> 
>> On Tue, 5 Dec 2023 at 11:50, Volkan Yazıcı  wrote:
>>> 
>>> I have just updated `main` to require Java 17 for compiling and to target
>>> Java 17. Previously both were Java 11. If you have objections, please
>>> discuss in this thread.
>> 
>> I am Ok with bumping the bytecode to Java 17.
>> 
>> Regarding the build requirements I would give by **default** a larger
>> spectrum of possible JDKs.
>> Right now we only accept JDK 17 in the default Maven profile. A Log4j
>> contributor is forced to install JDK 17 for a build to succeed, which
>> might be annoying. Therefore it might be better to:
>> 
>> * set the Enforcer plugin to accept
>> [minimum_JDK_required_by_Maven_plugins,)
>> * set [17,18) **only** in the `release` plugin.
>> 
>> BTW: when we bump BND to 7.x, minimum_JDK_required_by_Maven_plugins will
>> be 17.
>> 
>> Piotr
>> 

Re: Logstash and Filebeat guaranteed delivery

[Matt Sicker]

Right, I think I misread that. However, that’s what the curation idea is for :)
—
Matt Sicker

> On Dec 1, 2023, at 18:47, Gary Gregory  wrote:
> 
> Of course RE Volkan, my first sentence must have been poorly written. I
> meant to say that I know who Volkan is and that he is great, but, that's
> not the point. I was trying to say that a peronal GitHub account, in
> general, can't be compared to an official Apache account.
> 
> Gary
> 
> On Fri, Dec 1, 2023, 12:25 PM Matt Sicker  wrote:
> 
>> I may be biased, but I wouldn’t consider Volkan a random GitHub user!
>> However, that does raise an interesting point: we could link to third party
>> plugins and such to help curate it.
>> 
>>> On Dec 1, 2023, at 5:00 AM, Gary Gregory  wrote:
>>> 
>>> Hi all,
>>> 
>>> (Don't take his the wrong way Volkan ;-)
>>> Assuming that I don't know who you are, why would I pick a random github
>>> user's custom appender instead of an official Log4j appender? If your
>>> appender is "battle-tested", why not move it to Log4j (or Redis?)
>>> 
>>> Gary
>>> 
>>> 
>>> On Fri, Dec 1, 2023, 4:08 AM Volkan Yazıcı  wrote:
>>> 
>>>> I appreciate your thoughts on this subject. We can eventually convert
>> this
>>>> into a chapter in the Log4j manual. My goal is to be able to make a
>>>> statement as follows:
>>>> 
>>>> *When Log4j is configured with Y, Y, Z settings, it can provide
>> guaranteed
>>>> delivery against certain types of log sinks such as A, B, C.*
>>>> 
>>>> *A – You need to make sure A has ... feature enabled. Further, it has
>> ...
>>>> caveat.*
>>>> *B – You need to make sure B has ... feature enabled and ...*
>>>> *C – ...*
>>>> 
>>>> 
>>>> That is, a cookbook for users with recipes for guaranteed delivery.
>>>> 
>>>> [I respond to your message below inline.]
>>>> 
>>>> On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers >> 
>>>> wrote:
>>>> 
>>>>> Notice that neither of the links you have provided use the term
>>>>> “guaranteed delivery”. That is because that is not really what they are
>>>>> providing. In addition, notice that Logstash says "Input plugins that
>> do
>>>>> not use a request-response protocol cannot be protected from data
>> loss”,
>>>> 
>>>> 
>>>> But see the rest of that statement
>>>> <
>>>> 
>> https://www.elastic.co/guide/en/logstash/current/persistent-queues.html#persistent-queues-limitations
>>>>> 
>>>> : *"Plugins such as beats and http, which do have an acknowledgement
>>>> capability, are well protected by this [Logstash persistent] queue."*
>>>> 
>>>> 
>>>>> and "Data may be lost if an abnormal shutdown occurs before the
>>>> checkpoint
>>>>> file has been committed”.
>>>> 
>>>> 
>>>> See the following statement further down in that page: *"To avoid losing
>>>> data in the persistent queue, you can set `queue.checkpoint.writes: 1`
>> to
>>>> force a checkpoint after each event is written."*
>>>> 
>>>> These two make me conclude that, if configured correctly (e.g., using
>>>> `http` plugin in combination with `queue.checkpoint.writes: 1`),
>> Logstash
>>>> can deliver guaranteed delivery. Am I mistaken?
>>>> 
>>>> 
>>>>> As for using Google Cloud that would default the whole point. If your
>>>> data
>>>>> center has lost contact with the outside world it won’t be able to get
>> to
>>>>> Google Cloud.
>>>>> 
>>>> 
>>>> But that cannot be an argument against using Google Cloud as a log sink
>>>> with guaranteed delivery. An in-house Flume server can go down too. Let
>> me
>>>> know if I miss your point here.
>>>> 
>>>> 
>>>>> While Redis would work it would require a) an application component
>> that
>>>>> interacts with Redis such as a Redis Appender (which we don’t have) b)
>> a
>>>>> Redis deployment c) a Logstash (or some other Redis consumer) to
>> forward
>>>>> the event. It is a lot simpler to configure Flume than to do all of
>> that.
>>>>> 
>>>> 
>>>> 

Re: [Flume] Integration with OpenTelemetry

[Matt Sicker]

> On Nov 30, 2023, at 12:13 PM, Christian Grobmeier  
> wrote:
> 
> Please do. I would also like to learn more about this stuff and maybe we can 
> make Chainsaw to receive Flume messages or OTel things too!

I think that would make sense. When I do a deeper review of OpenTelemetry, I’ll 
suggest what components here can integrate with what.

Re: Logstash and Filebeat guaranteed delivery

[Matt Sicker]

I may be biased, but I wouldn’t consider Volkan a random GitHub user! However, 
that does raise an interesting point: we could link to third party plugins and 
such to help curate it.

> On Dec 1, 2023, at 5:00 AM, Gary Gregory  wrote:
> 
> Hi all,
> 
> (Don't take his the wrong way Volkan ;-)
> Assuming that I don't know who you are, why would I pick a random github
> user's custom appender instead of an official Log4j appender? If your
> appender is "battle-tested", why not move it to Log4j (or Redis?)
> 
> Gary
> 
> 
> On Fri, Dec 1, 2023, 4:08 AM Volkan Yazıcı  wrote:
> 
>> I appreciate your thoughts on this subject. We can eventually convert this
>> into a chapter in the Log4j manual. My goal is to be able to make a
>> statement as follows:
>> 
>> *When Log4j is configured with Y, Y, Z settings, it can provide guaranteed
>> delivery against certain types of log sinks such as A, B, C.*
>> 
>> *A – You need to make sure A has ... feature enabled. Further, it has ...
>> caveat.*
>> *B – You need to make sure B has ... feature enabled and ...*
>> *C – ...*
>> 
>> 
>> That is, a cookbook for users with recipes for guaranteed delivery.
>> 
>> [I respond to your message below inline.]
>> 
>> On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers 
>> wrote:
>> 
>>> Notice that neither of the links you have provided use the term
>>> “guaranteed delivery”. That is because that is not really what they are
>>> providing. In addition, notice that Logstash says "Input plugins that do
>>> not use a request-response protocol cannot be protected from data loss”,
>> 
>> 
>> But see the rest of that statement
>> <
>> https://www.elastic.co/guide/en/logstash/current/persistent-queues.html#persistent-queues-limitations
>>> 
>> : *"Plugins such as beats and http, which do have an acknowledgement
>> capability, are well protected by this [Logstash persistent] queue."*
>> 
>> 
>>> and "Data may be lost if an abnormal shutdown occurs before the
>> checkpoint
>>> file has been committed”.
>> 
>> 
>> See the following statement further down in that page: *"To avoid losing
>> data in the persistent queue, you can set `queue.checkpoint.writes: 1` to
>> force a checkpoint after each event is written."*
>> 
>> These two make me conclude that, if configured correctly (e.g., using
>> `http` plugin in combination with `queue.checkpoint.writes: 1`), Logstash
>> can deliver guaranteed delivery. Am I mistaken?
>> 
>> 
>>> As for using Google Cloud that would default the whole point. If your
>> data
>>> center has lost contact with the outside world it won’t be able to get to
>>> Google Cloud.
>>> 
>> 
>> But that cannot be an argument against using Google Cloud as a log sink
>> with guaranteed delivery. An in-house Flume server can go down too. Let me
>> know if I miss your point here.
>> 
>> 
>>> While Redis would work it would require a) an application component that
>>> interacts with Redis such as a Redis Appender (which we don’t have) b) a
>>> Redis deployment c) a Logstash (or some other Redis consumer) to forward
>>> the event. It is a lot simpler to configure Flume than to do all of that.
>>> 
>> 
>> For one, there is a battle-tested Log4j Redis Appender
>> , which enabled us to remove
>> `log4j-redis` in `main`.
>> 
>> Indeed, Flume can deliver what Redis+Logstash do. Though my point is, not
>> that Redis has a magical feature set, but there *are* several log sink
>> stacks one can build using modern stock components and provide guaranteed
>> delivery. I would like to document some of those, if not best-practices,
>> known-to-work solutions. This way we can enable our users to make a
>> well-informed decision and pick the best approach that fits into their
>> existing stack.
>> 
>> On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers 
>> wrote:
>> 
>>> Volkan,
>>> 
>>> Notice that neither of the links you have provided use the term
>>> “guaranteed delivery”. That is because that is not really what they are
>>> providing. In addition, notice that Logstash says "Input plugins that do
>>> not use a request-response protocol cannot be protected from data loss”,
>>> and "Data may be lost if an abnormal shutdown occurs before the
>> checkpoint
>>> file has been committed”. Note that Flume’s FileChannel does not face the
>>> second issue while the first would also be a problem if it is using a
>>> source that doesn’t support acknowledgements.However, Log4j’s
>> FlumeAppender
>>> always gets acks.
>>> 
>>> To make this clearer let me review the architecture for my implementation
>>> again.
>>> 
>>> First the phone system maintains a list of ip addresses that can handle
>>> Radius accounting records. We host 2 Flume servers in the same data
>> center
>>> as the phone system and configure the phone system with their ip
>> addresses.
>>> The Radius records will be sent to those Flume servers which will accept
>>> them with our custom Radius Source. That converts them to JSON and passes
>>> the JSON to the File 

Re: [log4j] Upgrade `2.x` compiler baseline to Java 17

[Matt Sicker]

No, only that it requires Java 17 to build. It still targets the Java 8 release 
profile.

> On Nov 30, 2023, at 11:15 AM, Gary Gregory  wrote:
> 
> Nice! That means that 2.23.0 will require Java 17 at runtime right?
> 
> Gary
> 
> 
> On Thu, Nov 30, 2023, 11:05 AM Volkan Yazıcı  wrote:
> 
>> Heads up! #2021  bumps
>> the `2.x` baseline to Java 17. Everything works locally. If CI agrees too,
>> I will merge it tomorrow and start porting to `main`.
>> 

Re: [log4j] Upgrade `2.x` compiler baseline to Java 17

[Matt Sicker]

Sounds great! We even finally updated Spinnaker this week to build on Java 17, 
so great timing.
—
Matt Sicker

> On Nov 30, 2023, at 10:03, Volkan Yazıcı  wrote:
> 
> Heads up! #2021 <https://github.com/apache/logging-log4j2/pull/2021> bumps
> the `2.x` baseline to Java 17. Everything works locally. If CI agrees too,
> I will merge it tomorrow and start porting to `main`.

Re: [Flume] Integration with OpenTelemetry

[Matt Sicker]

Oh yes, I’d still love to see how we can adapt the Log4j plugin system here! 
And yeah, this would likely make sense as its own repository. I’ll make a 
proposal about the OTel stuff before working on any code.
—
Matt Sicker

> On Nov 29, 2023, at 22:54, Ralph Goers  wrote:
> 
> This is a great post Matt!
> 
> Fluentbit, Fluentd, Logstash, and Filebeat are the main tools used for log 
> forwarding. While they all have some amount of plugability none of the are as 
> flexible as Flume. In addition, as I have mentioned before, none of them 
> provide guaranteed delivery so I would never recommend them for forwarding 
> audit logs. 
> 
> I have also previously explained my use case for using Flume, which is for 
> forwarding Call Detail Records that start off as records in the Radius 
> protocol [1] across data centers, which also requires guaranteed delivery. I 
> wouldn’t be able to use any of those other tools to do that without 
> significant modification.
> 
> I am all for supporting standards. If you can outline what you are proposing 
> on a Confluence page I would wholeheartedly support it.
> 
> As you probably know, I started work on separating out things that I don’t 
> consider to be “core” to Flume into separate repos. That work is only half 
> completed. I would suggest that you consider whether what you are proposing 
> also be in its own repo. As it is, the CI for Flume fails because the 
> generated logs are exceeding the available disk space. In addition, the build 
> takes a long time. 
> 
> Also, I have never really been a big fan of the configuration mechanism Flume 
> uses. I was able to somewhat bypass it by implementing support for Spring 
> Boot, but it would be great if the Log4j Plugin system could somehow be used 
> to simplify configuring Flume for those who don’t want to use Spring boot. I 
> know that is right up your alley.
> 
> Ralph
> 
> 
> 1. https://networkradius.com/doc/current/introduction/RADIUS.html
> 
>> On Nov 29, 2023, at 5:32 PM, Matt Sicker  wrote:
>> 
>> One of the main reasons why I supported Flume joining this PMC was that I 
>> noticed it has significant overlap with projects in the observability space 
>> despite not being advertised as such. For example, the project FluentBit is 
>> extremely similar to Flume, but its main purpose is for collecting, 
>> processing, forwarding, etc., logs, metrics, and traces (i.e., observability 
>> data). FluentBit is not the only thing in this space, though it seems to be 
>> fairly popular. These sorts of tools are used for ultimately publishing 
>> observability data to one or more observability tools like Prometheus, 
>> Splunk, Jaeger, Grafana, etc., and with a unified collector and processor, 
>> it becomes possible to publish all your observability data into one tool 
>> rather than three or more disparate tools (and the added operational costs 
>> of storing tons of duplicated log data from three or more methods of 
>> generating log data).
>> 
>> A project at the CNCF, OpenTelemetry, has become the sort of de facto 
>> standard for interoperability in this space. In particular, they’ve 
>> published the OTLP specification <https://opentelemetry.io/docs/specs/otlp/> 
>> for general telemetry data delivery and the OpenTelemetry specification 
>> <https://opentelemetry.io/docs/specs/otel/> for various common APIs. While 
>> I’m still researching in this space, I think it would be useful for Flume to 
>> integrate with some of these APIs and SDKs (while other parts might be more 
>> relevant in our logging libraries instead). There is also the Open Agent 
>> Management Protocol <https://opentelemetry.io/docs/specs/opamp/> which is 
>> still in beta status that might also be relevant here (and potentially 
>> relevant in the logging libraries).
>> 
>> Supporting common standards for our projects seems like a useful thing to 
>> do, and despite the popularity of some existing solutions there, I believe 
>> there is plenty of space for us to contribute. I also think that this can 
>> provide opportunity for the various components in this PMC to interoperate 
>> as these specs are fairly language neutral with some sample versions in many 
>> different languages.
> 
> 

[Flume] Integration with OpenTelemetry

[Matt Sicker]

One of the main reasons why I supported Flume joining this PMC was that I 
noticed it has significant overlap with projects in the observability space 
despite not being advertised as such. For example, the project FluentBit is 
extremely similar to Flume, but its main purpose is for collecting, processing, 
forwarding, etc., logs, metrics, and traces (i.e., observability data). 
FluentBit is not the only thing in this space, though it seems to be fairly 
popular. These sorts of tools are used for ultimately publishing observability 
data to one or more observability tools like Prometheus, Splunk, Jaeger, 
Grafana, etc., and with a unified collector and processor, it becomes possible 
to publish all your observability data into one tool rather than three or more 
disparate tools (and the added operational costs of storing tons of duplicated 
log data from three or more methods of generating log data).

A project at the CNCF, OpenTelemetry, has become the sort of de facto standard 
for interoperability in this space. In particular, they’ve published the OTLP 
specification  for general telemetry 
data delivery and the OpenTelemetry specification 
 for various common APIs. While I’m 
still researching in this space, I think it would be useful for Flume to 
integrate with some of these APIs and SDKs (while other parts might be more 
relevant in our logging libraries instead). There is also the Open Agent 
Management Protocol  which is still 
in beta status that might also be relevant here (and potentially relevant in 
the logging libraries).

Supporting common standards for our projects seems like a useful thing to do, 
and despite the popularity of some existing solutions there, I believe there is 
plenty of space for us to contribute. I also think that this can provide 
opportunity for the various components in this PMC to interoperate as these 
specs are fairly language neutral with some sample versions in many different 
languages.

Re: [log4j] Unstable tests on Windows

[Matt Sicker]

There’s a JUnit annotation `@Issue` or something like that (Spock has a similar 
annotation) which you can use for linking to an issue. Otherwise, adding `@Tag` 
annotations allows for arbitrary tags (of which we use several already such as 
“functional” and “sleepy”).

> On Nov 28, 2023, at 5:55 AM, Gary Gregory  wrote:
> 
> I think JUnit can group tests in categories with annotations (AFK).
> 
> Gary
> 
> On Tue, Nov 28, 2023, 12:01 AM Ralph Goers 
> wrote:
> 
>> I would be -1 if the issues are going to be ignored or not tracked in any
>> way. I don’t know if GitHub has something like a Jira Epic or if they can
>> be tagged in some way so that they can be easily located but something like
>> that would be fine. Even tracking them in Confluence would be fine.
>> 
>> It would also be great if only the failing tests could be run under a
>> profile making it easy to fix them.
>> 
>> Hopefully you get what I mean. I am not looking for something complicated,
>> just a way to make it easy to find them when someone has the urge to fix
>> them.
>> 
>> Ralph
>> 
>>> On Nov 27, 2023, at 3:28 AM, Volkan Yazıcı  wrote:
>>> 
>>> Ralph did not agree, but did not strongly object either. Ralph, are you
>> -1
>>> on disabling tests only Windows that are failing frequently on Windows
>> and
>>> capturing them in tickets to be addressed?
>>> 
>>> On Thu, Nov 23, 2023 at 12:23 AM Christian Grobmeier <
>> grobme...@apache.org>
>>> wrote:
>>> 
 Ralph said, nobody would ever fix these tests if you do it like this. I
 think you should create the ticket but keep the tests until we find the
 issue. Otherwise there issues will rot
 
 On Wed, Nov 22, 2023, at 09:13, Volkan Yazıcı wrote:
> AFAIC, nobody[1] shows a strong opposition against the idea of
>> disabling
> frequently failing Windows tests only on Windows and creating a ticket
 for
> each one. I will proceed with that.
> 
> [1] Except Piotr, whom I discussed the issue with in Slack and he
>> agreed
> with the above shared approach.
> 
> On Mon, Nov 20, 2023 at 12:57 PM Volkan Yazıcı  wrote:
> 
>> I am not asking to disable Windows tests. I am asking to disable tests
>> and only those tests that have a failure rate on Windows higher than,
>> say, 30%. To be precise, I think there are 2-3 of them dealing with
>> network sockets and rolling file appenders. I am not talking about
>> dozens or such.
>> 
>> After disabling them, we can create a ticket referencing them. So that
>> interested parties can fix them.
>> 
>> On Mon, Nov 20, 2023 at 12:25 PM Piotr P. Karwasz
>>  wrote:
>>> 
>>> Hi Volkan,
>>> 
>>> On Mon, 20 Nov 2023 at 09:36, Volkan Yazıcı  wrote:
 
 As Gary (the only Windows user among the active Log4j maintainers,
 AFAIK) has noticed several times, Log4j tests on Windows are pretty
 unstable. It not only fails on Gary's laptop, but Piotr and I need
 to
 give Windows tests in CI a kick on a regular basis. Approximately
 one
 out of three CI runs fails on Windows. Piotr already improved the
 situation extensively, though there are still several leftovers that
 need attention.
 
 Unless somebody steps up to improve the unstable Windows tests, I
 would like to disable those only for the WIndows platform.
>>> 
>>> Please don't. Windows has an annoying file locking policy that
>>> prevents users from deleting files with open file descriptors, but
>>> that is one of the few ways to detect resource leakage we have.
>>> 
>>> Tests running on *NIXes will ignore problems with open file
>>> descriptors and delete the log files, but on a production system
>> those
>>> leaks will accumulate and cause application crashes. We had such a
>>> leak, when we used `URLConnection#getLastModified` on a `jar:...`
>> URL.
>>> This call caused file descriptor exhaustion on both Windows and
>>> *NIXes, but only the Windows test was able to detect it.
>>> 
>>> Piotr,
>>> who never thought would ever defend Microsoft Windows.
>>> 
>>> PS: Gary reports the failures, but always runs the build again until
>>> it succeeds, even on Friday 13th, when he had to wait until Saturday
>>> 14th for the test run to succeed.
>> 
 
>> 
>> 

Re: [log4j] Releasing Log4j `3.0.0`

[Matt Sicker]

I’m good with a beta release to come soon. I’m still working on a branch to fix 
up the remaining SPI updates to avoid guessing at ClassLoaders, but that isn’t 
required for a beta release. As for the branches, if 2.x was made into 3.0, 
then main would be 4.0, and where would that leave us? Back in the same debate. 
I suggest you look at the main branch more closely.

> On Nov 28, 2023, at 4:19 AM, Volkan Yazıcı  wrote:
> 
> I plan to work on `main` until February, finalize recycler implementation,
> carry out whatever improvement I can, and release `3.0.0`.
> 
> *If you have any objections with this plan, or if you have things to do on
> `main` and you cannot comply with this schedule, etc., let's discuss.* I
> want to agree on a plan and timeline that works for you.
> 
> *Personal remark:* I am against releasing `3.0.0` from `main`. `2.x`
> changes that didn't go into `main` are titanic. `main` also contains
> several incomplete code, doc, or both. I support the idea of forking `3.x`
> from `2.x`, backporting crucial features from `main` to `3.x`, and then
> releasing `3.0.0`. I had several email, Slack, and video conversations with
> Ralph, Matt, and Piotr. They don't agree with me. Ralph even threatened to
> veto all non-bugfix changes on `2.x`
> . I am
> outnumbered and I accept the defeat. Let's release `3.0.0` from `main` and
> move on. I don't want to spend time discussing this subject further.

Re: Optional dependencies and JPMS

[Matt Sicker]

Small changes to that:

* Should move the XML configuration format to its own module, too, for the 
java.xml dependency
* Should remove the Jackson JSON configuration format as we have a 
dependency-free JSON format in core (the YAML format support is based on the 
Jackson JSON support, so this code can be simplified to only supporting YAML)

> On Nov 29, 2023, at 5:49 AM, Piotr P. Karwasz  wrote:
> 
> Hi all,
> 
> I have been using a modularized sandbox project to test Log4j and
> optional dependencies are a nightmare.
> 
> Small example: if I use Log4j API in my project and I add:
> 
> requires org.apache.logging.log4j;
> 
> then when I run the app, the JVM will automatically add `log4j-api` to
> the boot layer.
> 
> It will also add `log4j-core`, since it provides a service that
> `log4j-api` uses.
> 
> However it will **not** load `jackson-datatype-yaml`, because it is an
> **optional** dependency of `log4j-core` and provides no services
> `log4j-core` is interested in. The user must go to the additional pain
> of specifying `--add-modules` or the application developer must add a
> bogus `requires` statement.
> 
> Therefore I would propose to:
> 
> * move the YAML configuration factory to a separate module and delete
> one of the JSON configuration factories (the one that uses Jackson).
> 
> What do you think?

Re: Change semantics of `Throwable` parameters

[Matt Sicker]

I’ll note that MessageSupplier is deprecated, and we eventually added support 
for Supplier, so that’s already there. Since it sounds like we have 
some good ideas on what a minimal v3.Logger API would look like, it would be 
cool if anyone would like to propose a full API. Here’s another Logger API to 
consider (from OpenTelemetry, something I’ve been considering for future 
integration purposes, though this might be mostly on the Flume side of things): 
https://javadoc.io/doc/io.opentelemetry/opentelemetry-api-logs/latest/io/opentelemetry/api/logs/LogRecordBuilder.html
 (their Logger returns this builder class).
—
Matt Sicker

> On Nov 23, 2023, at 03:31, Piotr P. Karwasz  wrote:
> 
> Hi Matt,
> 
> On Tue, 21 Nov 2023 at 23:22, Matt Sicker  wrote:
>> 
>> This sounds like it might be a good basis for figuring out a parallel v3 API 
>> for a “hard to mis-use” style API. However, once you go that route, you 
>> start to wonder how useful templated log messages are when you can capture a 
>> lambda instead. Parameterized log messages might work better as structured 
>> log messages, something that is awkward to use in the API at the moment.
> 
> If we'll create a separate `v3.Logger` interface I would clean it up
> from many methods, e.g.:
> 
> * getLevel() and getName(): how are these useful for the user? An `if
> (logger.getLevel() == Level.INFO)` should be replaced by `isEnabled`,
> * getMessageFactory() and getFlowMessageFactory() (the latter is my
> fault): again these are not useful to the user. If I need a message
> factory, it will be a different message factory,
> * printf(): a better approach is to use StringFormatterMessageFactory
> for the logger,
> * catching(Throwable): can be replaced with `error(Object)` and the
> semantics described in this thread,
> * throwing: does anybody use it? Maybe it could stay,
> * entry/exit, traceEntry/traceExit: I can not imagine using these on
> each method (or important method). I'd rather use AspectJ pointcuts
> instead (or a @LogTrace annotation),
> * methods that use `MessageSupplier` like `info(MessageSupplier)`:
> couldn't these be integrated into the logic of `info(Supplier)`?
> * the `is*Enabled` methods are prone to misuse: a snippet like:
> 
> if (logger.isDebugEnabled()) {
>  logger.debug(MARKER, "Hello world!");
> }
> 
> will not print any message if the level of the logger is less specific
> than DEBUG, even if the user asks for **all** MARKER messages to be
> printed.
> 
> IMHO opinion `v3.AbstractLogger` should only have 2 abstract methods:
> * logMessage(Level level, Marker marker, String fqcn,
> StackTraceElement location, Message message, Throwable throwable)
> * isEnabled(Level level, Marker marker, String fqcn,
> StackTraceElement location, Message message, Throwable throwable)
> 
> Piotr

Re: Disable new issues in JIRA

[Matt Sicker]

I’d like it if we could copy over open issues to GH.
—
Matt Sicker

> On Nov 24, 2023, at 07:12, Piotr P. Karwasz  wrote:
> 
> Hi all,
> 
> Since we switched to Github Issues, I rarely look at JIRA any more and
> certainly I do not expect to see new issues being posted there.
> 
> Should we ask INFRA to disable issue creation on JIRA for LOG4J2? Do
> we need a vote for that?
> 
> Piotr

Re: [log4j] Unstable tests on Windows

[Matt Sicker]

The GH CI build also fails randomly compared to the green builds that I push in 
the first place. Occasionally, this might be a platform-specific issue (which 
is what I’d ideally be discovering from CI), but the failing tests are 
generally fragile ones in the first place. While I have ideas on how to make 
those areas of the codebase more testable, some of them can’t be more readily 
tested without drastic measures. Specifically, our tests around rolling file 
appender things and anything else relying on scheduled actions. 
Besides using an actual file system in the tests (these could run faster if 
using a virtual file system instead like [jimfs][0]), the largest limitation 
here is that we rely on a ScheduledThreadPoolExecutor for scheduling actions 
like rollovers, and this class relies directly on System::nanoTime for 
scheduling decisions, so the only way to mock the clock here is to either run 
the test in a modified runtime where the system calls behind nanoTime are 
mocked or to find a similar OSS implementation (or adapt from the JDK here if 
the classes are public domain like the Doug Lea ones usually are) to use 
instead which allows for controlling the clock. The latter option is something 
I considered before, but it’s not something to be taken lightly since we 
already expect the platform version of ScheduledThreadPoolExecutor to be stable 
and correct (or hopefully correct; this implementation is more likely to be 
thoroughly reviewed compared to something custom). In fact, this alternative 
path is basically porting or reimplementing the core of Quartz, so there is 
some reference code to adapt there come to think of it.

On the file system side, this would require either more direct adoption of the 
NIO2 API rather than the original IO API, or we need a similar abstraction to 
make mocking a file system easier to accomplish. Then we can have separate 
integration tests that use a real filesystem that could be more predictable if 
necessary.

Suffice to say, there’s usually a reason why I get into refactoring in this 
project, and most of the time, it involves making it easier to test and extend 
things both in the project and as a downstream user. The rolling appender area 
is something I’d like to get to sometime, but we’ve got about ten years worth 
of development to untangle (which we’ve largely accomplished in the main branch 
already with the modules there, but we can still break out more modules that 
involve more complicated refactoring).

[0]: https://github.com/google/jimfs
—
Matt Sicker

> On Nov 20, 2023, at 08:40, Gary Gregory  wrote:
> 
> The GH CI builds on every push (as opposed to commi) IIRC.
> 
> Gary
> 
> On Mon, Nov 20, 2023, 9:34 AM Ralph Goers 
> wrote:
> 
>> Gary uses Windows as his development OS. He is probably the only one of us
>> who does. So he inevitably finds these issues before the rest of us.
>> 
>> I don’t know if the CI has a build that runs on Windows for every commit.
>> 
>> Ralph
>> 
>>> On Nov 20, 2023, at 6:12 AM, Robert Middleton 
>> wrote:
>>> 
>>> Are the tests run on Windows through Github workflows?  It doesn't
>>> look like it to me.
>>> 
>>> If you need access to a Windows machine, you can download a
>>> development VM straight from Microsoft:
>>> 
>> https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/
>>> 
>>> -Robert Middleton
>>> 
>>> On Mon, Nov 20, 2023 at 7:40 AM Apache 
>> wrote:
>>>> 
>>>> In my experience they never get fixed. To be honest, when I was doing
>> the releases I would have these failures investigated to determine if it
>> was a trait problem vs a problem in the code being released. If it was the
>> latter I would cancel the vote. The only time tests should be disabled is
>> if we know it is a problem in the test but can’t figure out how to fix it.
>>>> 
>>>> I also don’t ever recall Gary ever having more than one or two tests
>> fail in a run.
>>>> 
>>>> Ralph
>>>> 
>>>>> On Nov 20, 2023, at 5:00 AM, Volkan Yazıcı  wrote:
>>>>> 
>>>>> I am not asking to disable Windows tests. I am asking to disable tests
>>>>> and only those tests that have a failure rate on Windows higher than,
>>>>> say, 30%. To be precise, I think there are 2-3 of them dealing with
>>>>> network sockets and rolling file appenders. I am not talking about
>>>>> dozens or such.
>>>>> 
>>>>> After disabling them, we can create a ticket referencing them. So that
>>>>> interested parties can fix them.
>>>>> 
>>>>>> On Mon, Nov 20, 2023 at 12:25 PM Pi

Re: Change semantics of `Throwable` parameters

[Matt Sicker]

This sounds like it might be a good basis for figuring out a parallel v3 API 
for a “hard to mis-use” style API. However, once you go that route, you start 
to wonder how useful templated log messages are when you can capture a lambda 
instead. Parameterized log messages might work better as structured log 
messages, something that is awkward to use in the API at the moment.
—
Matt Sicker

> On Nov 21, 2023, at 08:16, Piotr P. Karwasz  wrote:
> 
> Hi,
> 
> Profiting from release 3.x we could slightly change the interpretation
> of throwable parameters used in logging methods.
> 
> IIRC in one of our online meetings, we raised the issue of logging
> statements like this:
> 
> logger.info("Error nr {}: {}", nr, throwable);
> 
> which is not considered best practice and is probably not what the
> user intended:
> 
> logger.info("Error nr {}: {}", nr, throwable.getMessage(), throwable);
> 
> A similar problem was reported in PR #1050[2].
> 
> Regarding this kind of statements, the OSGi Log[1] Service provides a
> very nice alternative interpretation on how to deal with throwable
> parameters:
> 
> "If the last argument is a Throwable or a ServiceReference, it is
> added to the generated LogEntry and then, if the next to last argument
> is a ServiceReference or Throwable and not the same type as the last
> argument, it is also added to the generated LogEntry. These arguments
> will not be used as message arguments."
> 
> Should we adopt a similar interpretation in 3.x? The advantages I see are:
> 
> 1. We fix some inconsistencies in the API:
> * we have a pair `info(String, Throwable)/info(String, Object)`
> methods that prevents users from mistakenly pass a `Throwable` as a
> log message argument,
> * we only have `info(Object)`,
> * we only have `info(String, Object, Object)`.
> 2. We simplify our logic: right now we need logic to deal with
> throwables in `Message`. On the other hand throwables are not
> considered part of a message by any layout and are also included
> separately in LogEvent. If we don't need to chase throwables in
> parameterized messages, we don't need to parse the format string until
> it is required.
> 3. There might be some cases when including throwables in a log
> message leads to information disclosure (e.g. when a user sees a log
> message, but not the attached throwable). This way we'll have an API
> which is easy to use and hard to misuse.
> 
> Piotr
> 
> [1] 
> https://docs.osgi.org/javadoc/osgi.core/8.0.0/org/osgi/service/log/Logger.html
> [2] https://github.com/apache/logging-log4j2/pull/1050

Re: [PR] Bump org.junit:junit-bom from 5.10.0 to 5.10.1 [logging-log4j-jakarta]

[Matt Sicker]

Oh I see. The default options were updated for everyone for the email subjects, 
but by default, these bot emails are still going to dev@ instead of the robots 
list.

> On Nov 15, 2023, at 12:14 PM, Robert Middleton  wrote:
> 
> I suspect the settings are on a per-repo basis.  Since this is a new
> repo, it doesn't have the proper settings.
> 
> -Robert Middleton
> 
> On Wed, Nov 15, 2023 at 12:56 PM Matt Sicker  wrote:
>> 
>> Why do our email settings keep getting reset? Do we have some default 
>> settings somewhere?
>> 
>>> On Nov 15, 2023, at 6:19 AM, dependabot[bot] (via GitHub)  
>>> wrote:
>>> 
>>> 
>>> dependabot[bot] opened a new pull request, #1:
>>> URL: https://github.com/apache/logging-log4j-jakarta/pull/1
>>> 
>>>  Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 
>>> 5.10.0 to 5.10.1.
>>>  
>>>  Release notes
>>>  Sourced from >> href="https://github.com/junit-team/junit5/releases;>org.junit:junit-bom's 
>>> releases.
>>>  
>>>  JUnit 5.10.1 = Platform 1.10.1 + Jupiter 5.10.1 + Vintage 5.10.1
>>>  See >> href="http://junit.org/junit5/docs/5.10.1/release-notes/;>Release 
>>> Notes.
>>>  Full Changelog: >> href="https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1;>https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1
>>>  
>>>  
>>>  
>>>  Commits
>>>  
>>>  >> href="https://github.com/junit-team/junit5/commit/e5f50d8720741623915979ac154b65bcbcd6a577;>e5f50d8
>>>  Release 5.10.1
>>>  >> href="https://github.com/junit-team/junit5/commit/ac86d18e9b15dbebe046e82743ac7f9534a17582;>ac86d18
>>>  Fix typo in AfterAll documentation
>>>  >> href="https://github.com/junit-team/junit5/commit/388c5beaf42944961ab5b455c900d958a6e15588;>388c5be
>>>  Harmonize application of method and field filters in search algorithms
>>>  >> href="https://github.com/junit-team/junit5/commit/f82dd1e716f8717e012152b1d1d5cc0da10d33cd;>f82dd1e
>>>  Apply field predicate before searching type hierarchy
>>>  >> href="https://github.com/junit-team/junit5/commit/1d1eb8571552bbf28e578241965010de6c8ee779;>1d1eb85
>>>  Polishing
>>>  >> href="https://github.com/junit-team/junit5/commit/5ce280eff69b43759a3cb0c176207abe0a41b579;>5ce280e
>>>  Update picocli to 4.7.5 and enable help width computation
>>>  >> href="https://github.com/junit-team/junit5/commit/fea05c3aa80de76686f326b5ce26ddf7f153ff5a;>fea05c3
>>>  Fix ConsoleLauncherTests and StandaloneTests
>>>  >> href="https://github.com/junit-team/junit5/commit/c5567354c2556e772f8a0035ef7647161011d1c0;>c556735
>>>  Use same expected files for all JDK versions
>>>  >> href="https://github.com/junit-team/junit5/commit/808493ab09b30970b506a48fda3d616ac1ba4fff;>808493a
>>>  Run StandaloneTests for Java 8 under Java 8
>>>  >> href="https://github.com/junit-team/junit5/commit/9ec57661c78c3889db004ab6a89416e56a2fb2e0;>9ec5766
>>>  Unify messages about exit codes in StandaloneTests
>>>  Additional commits viewable in >> href="https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1;>compare
>>>  view
>>>  
>>>  
>>>  
>>> 
>>> 
>>>  [![Dependabot compatibility 
>>> score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.junit:junit-bom=maven=5.10.0=5.10.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
>>> 
>>>  Dependabot will resolve any conflicts with this PR as long as you don't 
>>> alter it yourself. You can also trigger a rebase manually by commenting 
>>> `@dependabot rebase`.
>>> 
>>>  [//]: # (dependabot-automerge-start)
>>>  [//]: # (dependabot-automerge-end)
>>> 
>>>  ---
>>> 
>>>  
>>>  Dependabot commands and options
>>>  
>>> 
>>>  You can trigger Dependabot actions by commenting on this PR:
>>>  - `@dependabot rebase` will rebase this PR
>>>  - `@dependabot recreate` will recreate this PR, overwriting any edits that 
>>> have been made to it
>>>  - `@dependabot merge` will merge this PR after your CI passes on it
>>>  - `@dependabot squash and merge` will squash and merge this PR after your 
>>> CI passes on it
>>

Re: [PR] Bump org.junit:junit-bom from 5.10.0 to 5.10.1 [logging-log4j-jakarta]

[Matt Sicker]

Why do our email settings keep getting reset? Do we have some default settings 
somewhere?

> On Nov 15, 2023, at 6:19 AM, dependabot[bot] (via GitHub)  
> wrote:
> 
> 
> dependabot[bot] opened a new pull request, #1:
> URL: https://github.com/apache/logging-log4j-jakarta/pull/1
> 
>   Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 
> 5.10.0 to 5.10.1.
>   
>   Release notes
>   Sourced from  href="https://github.com/junit-team/junit5/releases;>org.junit:junit-bom's 
> releases.
>   
>   JUnit 5.10.1 = Platform 1.10.1 + Jupiter 5.10.1 + Vintage 5.10.1
>   See http://junit.org/junit5/docs/5.10.1/release-notes/;>Release 
> Notes.
>   Full Changelog:  href="https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1;>https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1
>   
>   
>   
>   Commits
>   
>href="https://github.com/junit-team/junit5/commit/e5f50d8720741623915979ac154b65bcbcd6a577;>e5f50d8
>  Release 5.10.1
>href="https://github.com/junit-team/junit5/commit/ac86d18e9b15dbebe046e82743ac7f9534a17582;>ac86d18
>  Fix typo in AfterAll documentation
>href="https://github.com/junit-team/junit5/commit/388c5beaf42944961ab5b455c900d958a6e15588;>388c5be
>  Harmonize application of method and field filters in search algorithms
>href="https://github.com/junit-team/junit5/commit/f82dd1e716f8717e012152b1d1d5cc0da10d33cd;>f82dd1e
>  Apply field predicate before searching type hierarchy
>href="https://github.com/junit-team/junit5/commit/1d1eb8571552bbf28e578241965010de6c8ee779;>1d1eb85
>  Polishing
>href="https://github.com/junit-team/junit5/commit/5ce280eff69b43759a3cb0c176207abe0a41b579;>5ce280e
>  Update picocli to 4.7.5 and enable help width computation
>href="https://github.com/junit-team/junit5/commit/fea05c3aa80de76686f326b5ce26ddf7f153ff5a;>fea05c3
>  Fix ConsoleLauncherTests and StandaloneTests
>href="https://github.com/junit-team/junit5/commit/c5567354c2556e772f8a0035ef7647161011d1c0;>c556735
>  Use same expected files for all JDK versions
>href="https://github.com/junit-team/junit5/commit/808493ab09b30970b506a48fda3d616ac1ba4fff;>808493a
>  Run StandaloneTests for Java 8 under Java 8
>href="https://github.com/junit-team/junit5/commit/9ec57661c78c3889db004ab6a89416e56a2fb2e0;>9ec5766
>  Unify messages about exit codes in StandaloneTests
>   Additional commits viewable in  href="https://github.com/junit-team/junit5/compare/r5.10.0...r5.10.1;>compare 
> view
>   
>   
>   
> 
> 
>   [![Dependabot compatibility 
> score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.junit:junit-bom=maven=5.10.0=5.10.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
> 
>   Dependabot will resolve any conflicts with this PR as long as you don't 
> alter it yourself. You can also trigger a rebase manually by commenting 
> `@dependabot rebase`.
> 
>   [//]: # (dependabot-automerge-start)
>   [//]: # (dependabot-automerge-end)
> 
>   ---
> 
>   
>   Dependabot commands and options
>   
> 
>   You can trigger Dependabot actions by commenting on this PR:
>   - `@dependabot rebase` will rebase this PR
>   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
> have been made to it
>   - `@dependabot merge` will merge this PR after your CI passes on it
>   - `@dependabot squash and merge` will squash and merge this PR after your 
> CI passes on it
>   - `@dependabot cancel merge` will cancel a previously requested merge and 
> block automerging
>   - `@dependabot reopen` will reopen this PR if it is closed
>   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
> You can achieve the same result by closing it manually
>   - `@dependabot show  ignore conditions` will show all of 
> the ignore conditions of the specified dependency
>   - `@dependabot ignore this major version` will close this PR and stop 
> Dependabot creating any more for this major version (unless you reopen the PR 
> or upgrade to it yourself)
>   - `@dependabot ignore this minor version` will close this PR and stop 
> Dependabot creating any more for this minor version (unless you reopen the PR 
> or upgrade to it yourself)
>   - `@dependabot ignore this dependency` will close this PR and stop 
> Dependabot creating any more for this dependency (unless you reopen the PR or 
> upgrade to it yourself)
> 
> 
>   
> 
> 
> -- 
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
> 
> To unsubscribe, e-mail: dev-unsubscr...@logging.apache.org
> 
> For queries about this service, please contact Infrastructure at:
> us...@infra.apache.org
> 

Re: Code reformatting

[Matt Sicker]

We have our great deletion branch I started to collaborate with you, though I 
can merge that beforehand.

> On Nov 15, 2023, at 1:53 AM, Piotr P. Karwasz  wrote:
> 
> Hi all,
> 
> On Wed, 8 Nov 2023 at 09:05, Volkan Yazıcı  wrote:
>> Piotr, it has been two months or so since we are discussing this. No
>> objections so far. Please go ahead and implement this. It will help a lot
>> for sync'ing `2.x` and `3.x`.
> 
> Since Logging Parent 10.4.0 has been published, I plan to reformat
> `main` this weekend (branch `2.x` will be reformatted before the
> 2.22.0 release).
> 
> Do you have any long outstanding branches, which were not already
> merged into the code? Reformatting will render merging or rebasing
> very frustrating. Can you rebase them now on the current `main`?
> 
> Piotr

Re: OSGi package versioning

[Matt Sicker]

It’s similar to the compatible API versions, but the BND equivalent here would 
be the highest package version of all the packages in log4j-api (since the 
packages don’t get split out from the modules in practice).
—
Matt Sicker

> On Nov 13, 2023, at 16:02, Piotr P. Karwasz  wrote:
> 
> Hi Ralph,
> 
> On Mon, 13 Nov 2023 at 22:18, Ralph Goers  wrote:
>> 
>> How does this relate, if at all, to the COMPATIBLE_API_VERSIONS constant 
>> defined in ProviderUtil.java? We use that to ensure the API only binds with 
>> a compatible implementation of the API.
> 
> It is certainly related, although in practice I don't know how.
> In theory every minor version bump of `org.apache.logging.log4j` or
> `o.a.l.l.spi` can cause an incompatibility between API and
> implementation.
> In practice the implementations also rely on the API for
> `o.a.l.l.util` classes, so they will not work with a version that is
> too old.
> 
> I would certainly bring some order here, if you have any ideas how...
> 
> Piotr

Re: [log4j] `2.22.0` release

[Matt Sicker]

Go for it! I love how streamlined our release process is now. :)
—
Matt Sicker

> On Nov 13, 2023, at 08:22, Volkan Yazıcı  wrote:
> 
> I would like to do the Log4j `2.22.0` release in a couple of days;
> there are some bug fixes and SBOM is ready to be shipped. Let me know
> if you have objections.

Upcoming changes to main for Log4j 3.x

[Matt Sicker]

After some discussions yesterday about what’s up for 3.0 (notes are somewhere), 
I wanted to share the list of action items I'm planning to implement in main:

* Removal of legacy OSGi support. We will rely on ServiceLoader and things like 
SPI Fly or other common OSGi mechanisms for supporting ServiceLoader while 
deferring lower level OSGi integration to libraries like Pax-Logging.
* Removal of support for SecurityManager. Since Java 21 doesn’t allow for 
specifying a SecurityManager, and considering our anemic support for running 
with one, we thought it would be simpler to deprecate support for this and log 
a warning at startup if a custom SecurityManager is installed.
* Further refactoring in log4j-api/core to better support more context-scoped 
properties (we still have various things relying on static fields for this)

We also talked about upgrading the JavaEE API integrations to all use 
JakartaEE, switching the baseline Java version for 3.0 to be 17 (to match 
Spring Boot 3 et al.), and some other things I can’t recall at the moment. 
These encompassed most of what we wanted to definitely have before 3.0 GA, 
though we also plan to make some regular beta releases. There was also a desire 
to split out all the JakartaEE stuff into its own git repository (SMTP, JMS, 
JDBC, JNDI, JPA, and anything else using JavaEE/JakartaEE libs).

I know that Volkan plans on working on the Recycler API which touches things 
largely in the Layout implementations along with reusable Message and LogEvent 
objects, though this is largely an implementation detail until we expose the 
Recycler API for more general reuse. What else have I missed in this email?

—
Matt Sicker

Re: Vulnerability Disclosure Report (VDR)

[Matt Sicker]

Very neat! Thanks for getting this started.

> On Nov 8, 2023, at 9:49 AM, Volkan Yazıcı  wrote:
> 
> Today I have published the CycloneDX Vulnerability Disclosure Report (VDR)
>  Piotr and I have been working on.
> This VDR is expected to contain all CVEs filed by Logging Services. All our
> SBOMs will point to this one-and-only VDR file with the most recent
> `logging-parent` release.
> 
> *Public URL:* https://logging.apache.org/cyclonedx/vdr.xml
> *Canonical URL:*
> https://logging.apache.org/cyclonedx/urn:uuid:dfa35519-9734-4259-bba1-3e825cf4be06
> *Source:*
> https://github.com/apache/logging-site/blob/cyclonedx/urn%3Acdx%3Adfa35519-9734-4259-bba1-3e825cf4be06/1.xml
> 
> I already filled in all CVEs published against Log4j. I will appreciate it
> if the rest of you can help us to do the same for Chainsaw, Log4cxx, etc.
> 
> Please do create a PR for your changes (I will be more than happy to review
> them!) and follow a chronological order (newer CVE comes first) in
> `vulnerability` entries.

Re: [log4j] Security page refactoring

[Matt Sicker]

Actually, I was able to review this just now during the lunch break.

> On Nov 7, 2023, at 1:24 PM, Matt Sicker  wrote:
> 
> Also note that I’m at KubeCon right now, so I won’t have a chance to review 
> this PR until, well, sometime during the conference. Possibly today, but 
> likely within the next day or two.
> 
>> On Nov 6, 2023, at 2:17 PM, Volkan Yazıcı  wrote:
>> 
>> I have created #1948 <https://github.com/apache/logging-log4j2/pull/1948>
>> refactoring the Log4j security page. See the PR description for details.
>> Reviews are welcome.
> 

Re: [log4j] Security page refactoring

[Matt Sicker]

Also note that I’m at KubeCon right now, so I won’t have a chance to review 
this PR until, well, sometime during the conference. Possibly today, but likely 
within the next day or two.

> On Nov 6, 2023, at 2:17 PM, Volkan Yazıcı  wrote:
> 
> I have created #1948 
> refactoring the Log4j security page. See the PR description for details.
> Reviews are welcome.

Re: [log4j] Security page refactoring

[Matt Sicker]

Not that it’s relevant to adopt right away, but do note that there is a CVSS 
4.0 now (that was finished quite recently) which is supposed to produce more 
useful severity scores.

> On Nov 6, 2023, at 2:17 PM, Volkan Yazıcı  wrote:
> 
> I have created #1948 
> refactoring the Log4j security page. See the PR description for details.
> Reviews are welcome.

Re: Deterministic formatter

[Matt Sicker]

In the worst case scenario, we can still format from maven before committing 
(which is what I used to do before finding that there were IntelliJ plugins for 
this). In fact, I have to do that all the time lately anyways by running `mvn 
spotless:apply`.

> On Nov 6, 2023, at 9:00 AM, Carter Kozak  wrote:
> 
> I'd be happy to review+release changes to get the eclipse plugin in that repo 
> into a good place as long as it doesn't make the build process a great deal 
> more complicated. We don't have many folks internally using eclipse so 
> support hasn't been a priority, but the easier it is to use across common 
> toolchains, the better!
> 
> -ck
> 
> On Mon, Nov 6, 2023, at 06:55, Piotr P. Karwasz wrote:
>> Hi Gary,
>> 
>> On Mon, 6 Nov 2023 at 11:45, Gary Gregory  wrote:
>>> 
>>> Well, I use Eclipse, so... I won't be using whatever this does or when it
>>> does it.
>> 
>> What version of Eclipse do you use? The Eclipse plugin is one class, I
>> can probably fix it, compile it and release it.
>> 
>> Piotr
>> 

Re: [log4j] What is JPMS support and its state

[Matt Sicker]

I’m not going to backport the DI system. It relies on Java 11 in all sorts of 
random places, first of all. I had to update numerous plugins and tests along 
the way, especially things that set up or manipulate static state, much of 
which has been cleaned up in main. It’s hard to describe all the little things 
that went into that.
—
Matt Sicker

> On Nov 7, 2023, at 05:31, Piotr P. Karwasz  wrote:
> 
> Hi Ralph,
> 
>> On Tue, 7 Nov 2023 at 05:05, Ralph Goers  wrote:
>> Next, there is a good chance that users really using JPMS won’t be able to 
>> access any plugins outside of log4j-core. See 
>> https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ClassLoader.html#getResource(java.lang.String).
>>  This is precisely why 3.x uses ServiceLoader to locate all plugins.
> 
> The `META-INF` directory is not encapsulated, so `getResource` should
> work fine. Nevertheless using `ServiceLoader` is a much cleaner
> solution and would allow us to greatly simplify our OSGi-specific
> code.
> 
> Piotr

Re: Features for 3.x was: Why is JNDI still necessary?

[Matt Sicker]

I added @InternalApi to all the classes that were marked “consider this class 
private”. As for the plugins, yes, now that we’ve got that stabilized, I think 
that makes sense to keep stable, too.
—
Matt Sicker

> On Nov 5, 2023, at 16:22, Ralph Goers  wrote:
> 
> 
> 
>> On Nov 5, 2023, at 2:58 PM, Matt Sicker  wrote:
> 
>> I’ve suggested that we annotate code around API compatibility guarantees, 
>> and we are using @InternalApi in main to mark things that shouldn’t be used 
>> as stable code (even if it’s unlikely to change over time).
>> 
> 
> Please be careful of your usage of the term “API”. Log4j-API is one thing. 
> Very little in it should be private but there is some stuff. We have done a 
> decent job of adding comments of “consider this priviate” but using 
> annotations and any tooling we can to enforce that would greatly help.
> 
> As for the other modules, Log4j-Plugins obviously has a publicly exposed API. 
> That was intentional as my hope is that eventually Log4j-Plugins could become 
> a general use framework, not specific to Log4j. Log4j-Core is where the real 
> meat of this is. If we wanted to be really clear we could have split 
> log4j-core into log4j-core-api and log4j-core-impl but using JPMS and 
> annotations should be enough.
> 
> Ralph
> 

Re: Features for 3.x was: Why is JNDI still necessary?

[Matt Sicker]

> On Nov 4, 2023, at 00:08, Christian Grobmeier  wrote:
> […]
> 
> Agreed, the earlier, so better, but making Log4j 3.x GA "first thing next 
> year" seems to be quite of a rush. 

After another beta release, I don’t think that’s a rush. It’s not like the 
entire ecosystem will upgrade all at once!

> As far as I know, we still have changes in 2.x that are not in 3.x.

Any missing changes are likely minor and can be included in 3.0.1 or 3.1.0. We 
can do a diff between branches more easily now that they’ve been somewhat 
reformatted when combined with `git diff -w` to ignore whitespace changes, and 
this can reveal relevant changes. I’ve looked at some of that recently, and it 
looks like there are pom.xml differences, but I wasn’t finding much relevant 
changes from the bits I sampled.

> I heard we lack integration tests and parallel testing in 3.x is disabled. 
> This does not make me feel like a GA, but starting a longer beta phase.

I was wondering why the 2.x branch builds faster than main, but that’s not a 
blocker for releases.

>>>>> With flume, audit and the other stuff eating time, it will be hard to get 
>>>>> this done.
>>>> 
>>>> Get what done? As far as I am concerned Log4j 3.x can go to beta now 
>>>> and GA the first of the year. There are no required features left, just 
>>>> stuff that would be nice to get done but can be done after the initial 
>>>> GA release.
>>> 
>>> I'm talking about adding Flume to our project, adding the community, and 
>>> maintaining the new component.
>> 
>> Up til now that is all pretty much me. I expect it will stay that way 
>> for a time, although Matt has indicated an interest in contributing.
> 
> Yes, that time you don't have for 3.x

Flume doesn’t require a massive rewrite or anything like that. In fact, it 
could probably use trimming down of some now irrelevant modules.

>>> We have open security issues with log4j-audit, and log4j-server still needs 
>>> to be cleaned up.
>> 
>> Log4j-Audit has no open security issues. It simply needs dependency 
>> updates. Please don’t conflate one with the other.
> 
> Ok, great, yes, it is using dependencies with major security holes and does 
> not look maintained, yet it is promoted as a main product on our website. Did 
> we check if it is OK to deliver those dependencies? Upgrading does not seem 
> trivial, we haven't also even started.

The audit web app thing is a single user application that you can realistically 
only run locally. It’s all offline software besides the API, and the API 
depends on Log4j API which can be overridden by the user.

>> Others have expressed things they would like to do because it is a 
>> major release and so some end-user impact is allowable. But most, if 
>> not all, of the things I have seen proposed could be done post 3.0 GA. 
>> So no, it is not a requirement that they be addressed. For example, 
>> Matt did a lot of work on the DI system. It took him forever. None of 
>> it was required. It was/is very beneficial to do but it wasn’t required.
> 
> Why do you bring up Matts work here? It seems unrelated.

The biggest change we wanted for 3.x was related to JPMS support and beginning 
to break out a large portion of log4j-core into separate modules. That was 
completed a while back. I did work on a few iterations of the plugin system to 
transform it into a more generic DI system, but that’s largely an 
implementation detail that could have been done post-3.0 as it was designed to 
be backward-compatible (which itself took some finagling to figure out how to 
unify, but it’s fine now).

There have been some things I’ve changed that should be done before 3.0, but 
those largely involved removal of deprecated code and some small changes to 
APIs (such as flattening of extension interfaces, removal of Serializable, 
removal of the generic parameter from Layout). We can do some additional passes 
for anything else that should be cleaned up before 3.0, but it’s not much. I’ve 
suggested that we annotate code around API compatibility guarantees, and we are 
using @InternalApi in main to mark things that shouldn’t be used as stable code 
(even if it’s unlikely to change over time).

—
Matt Sicker

Re: (logging-log4j2) branch 2.x updated: Backport Lazy util

[Matt Sicker]

I split this one up by accident. There was a follow up commit that updated the 
places where they were updated in main. I don’t recall any particular issue 
being resolved, though I was comparing the 2.x and main branches to look for 
differences. Guess I did it in reverse here.
—
Matt Sicker

> On Nov 5, 2023, at 00:08, Ralph Goers  wrote:
> 
> Matt,
> 
> I received 3 emails regarding this lazy stuff being back ported. Is there a 
> related issue for this? With the volume of emails I am getting I may have 
> missed it but I would like to understand, Furthermore, in none of these 
> commit emails did I see a changes file listed? How can I know why you did 
> this without that and user’s won’t either since it won’t appear in the 
> release notes.
> 
> Note that while I would like to avoid back porting from 3.x if it is being 
> done to resolve a bug then I am fine with that. But if this is just a random 
> “improvement” please don’t.
> 
> Ralph
> 
>> On Nov 3, 2023, at 4:22 PM, mattsic...@apache.org wrote:
>> 
>> This is an automated email from the ASF dual-hosted git repository.
>> 
>> mattsicker pushed a commit to branch 2.x
>> in repository https://gitbox.apache.org/repos/asf/logging-log4j2.git
>> 
>> 
>> The following commit(s) were added to refs/heads/2.x by this push:
>>new 4243f7cfce Backport Lazy util
>> 4243f7cfce is described below
>> 
>> commit 4243f7cfceae535c898b863ada75417c0e4217fc
>> Author: Matt Sicker 
>> AuthorDate: Fri Nov 3 18:18:10 2023 -0500
>> 
>>   Backport Lazy util
>> ---
>> .../java/org/apache/logging/log4j/util/Lazy.java   |  93 +++
>> .../org/apache/logging/log4j/util/LazyBoolean.java |  66 
>> .../org/apache/logging/log4j/util/LazyUtil.java| 171 
>> +
>> 3 files changed, 330 insertions(+)
>> 
>> diff --git a/log4j-api/src/main/java/org/apache/logging/log4j/util/Lazy.java 
>> b/log4j-api/src/main/java/org/apache/logging/log4j/util/Lazy.java
>> new file mode 100644
>> index 00..2e0d84b3ce
>> --- /dev/null
>> +++ b/log4j-api/src/main/java/org/apache/logging/log4j/util/Lazy.java
>> @@ -0,0 +1,93 @@
>> +/*
>> + * Licensed to the Apache Software Foundation (ASF) under one or more
>> + * contributor license agreements.  See the NOTICE file distributed with
>> + * this work for additional information regarding copyright ownership.
>> + * The ASF licenses this file to you under the Apache License, Version 2.0
>> + * (the "License"); you may not use this file except in compliance with
>> + * the License.  You may obtain a copy of the License at
>> + *
>> + *  http://www.apache.org/licenses/LICENSE-2.0
>> + *
>> + * Unless required by applicable law or agreed to in writing, software
>> + * distributed under the License is distributed on an "AS IS" BASIS,
>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> + * See the License for the specific language governing permissions and
>> + * limitations under the License.
>> + */
>> +package org.apache.logging.log4j.util;
>> +
>> +import java.util.Objects;
>> +import java.util.function.Function;
>> +import java.util.function.Supplier;
>> +
>> +/**
>> + * Provides a lazily-initialized value from a {@code Supplier}.
>> + *
>> + * @param  type of value
>> + */
>> +public interface Lazy extends Supplier {
>> +/**
>> + * Returns the value held by this lazy. This may cause the value to 
>> initialize if it hasn't been already.
>> + */
>> +T value();
>> +
>> +/**
>> + * Returns the value held by this lazy. This may cause the value to 
>> initialize if it hasn't been already.
>> + */
>> +@Override
>> +default T get() {
>> +return value();
>> +}
>> +
>> +/**
>> + * Creates a new lazy value derived from this lazy value using the 
>> provided value mapping function.
>> + */
>> +default  Lazy map(final Function 
>> function) {
>> +return lazy(() -> function.apply(value()));
>> +}
>> +
>> +/**
>> + * Indicates whether this lazy value has been initialized.
>> + */
>> +boolean isInitialized();
>> +
>> +/**
>> + * Sets this lazy value to the provided value.
>> + *
>> + * @throws UnsupportedOperationException if this type of lazy value 
>> cannot be updated
>> + */
>> +void set(final T newValue);
>> +
>> +/**
>&g

Re: Deterministic formatter

[Matt Sicker]

Alright, after looking at the differences, I’m strongly in favor of the 
Palantir version. The differences in how it handles lambdas solves one of the 
main complaints I ever had about the Google version.

> On Oct 25, 2023, at 11:47 PM, Piotr P. Karwasz  
> wrote:
> 
> Hi Matt,
> 
> On Thu, 26 Oct 2023 at 01:03, Matt Sicker  wrote:
>> 
>> Seems reasonable to me. I can’t really tell what the difference is between 
>> the two resulting outputs, though.
> 
> There was no difference (I must have used the same Spotless config
> twice), so I corrected it.
> 
> Anyway the differences are very subtle and are mainly in the wrapping 
> algorithm:
> 
> https://github.com/palantir/palantir-java-format
> 
> Piotr

Re: [Discuss]: [VOTE] Deprecation of 2.x modules/features and removal in 3.x

[Matt Sicker]

Gary, have you ever heard of JNoSQL before? <http://www.jnosql.org/> It’s 
basically a JDBC-like effort to create a generic NoSQL API. Been around for a 
few years at least, and I had considered adding an appender for this back when 
I learned about it, but I think this was by the point where I stopped making 
proof of concept appenders.
—
Matt Sicker

> On Nov 3, 2023, at 09:08, Gary Gregory  wrote:
> 
> What I mean is this: When I want to work with a DB-like thing from Java, I
> look 1st for a JDBC driver, when there is no FOSS option, I have to use a
> DB-specifc API, for example MongoDB. Granted Cassandra is NoSQL doc store
> but there are commercial JDBC drivers IIRC.
> 
> All of this to say that I feel it is nice for us to support non-JDBC stores
> when there are no FOSS JDBC drivers available for that datastore type.
> 
> 2. I got it backward then, thank you for explaining it again. +1 to remove
> in 3.0 but keep in 2.x.
> 
> Gary
> 
> On Thu, Nov 2, 2023, 11:04 AM Ralph Goers 
> wrote:
> 
>> Gary, a couple of comments:
>> 
>> 1. I don’t understand what “there is no official JDBC access” has to do
>> with Cassandra or CouchDB. Could you please elaborate?
>> 2. The functionality in log4j-spring-boot is directly incorporated into
>> Spring 3 due to a PR I submitted. So with Spring Boot 3 log4j-spring-boot
>> should NOT be included. Since Log4j 3.x (at least in my mind) goes hand in
>> hand with Spring Boot 3 then removing log4j-spring-boot in 3.x makes sense.
>> 
>> Ralph
>> 
>>> On Nov 2, 2023, at 7:42 AM, Gary Gregory  wrote:
>>> 
>>> My votes:
>>> 
>>> On Mon, Oct 30, 2023 at 4:45 AM Piotr P. Karwasz
>>>  wrote:
>>>> 
>>>> This is a vote to deprecate the following `2.x` modules and features
>>>> and remove them from the `3.x` release:
>>>> 
>>>> * `log4j-cassandra`:
>>> -0: Prefer keeping since there is no official JDBC access that I know
>>> of but I have not looked for a while.
>>> 
>>>> * CouchDB appended:
>>> -0: Prefer keeping since there is no official JDBC access that I know
>>> of but I have not looked for a while.
>>> 
>>>> * `log4j-docker`
>>> -1 Docker seems too important.
>>> 
>>>> * GELF appended:
>>> +0
>>> 
>>>> * Kafka appended:
>>> -0: Prefer keeping since there is no official JMS access that I know
>>> of but I have not looked for a while.
>>> 
>>>> * `log4j-kubernetes`:
>>> -1: This should match what we do with Docker.
>>> 
>>>> * JeroMQ appender:
>>> -0: Prefer keeping since there is no official JMS access that I know
>>> of but I have not looked for a while.
>>> 
>>>> * JNDI-related features:
>>> -1: Needed in enterprise environments, OK to split out in a separate
>>> Maven module.
>>> 
>>>> * `log4j-jpa`:
>>> -0: JDBC Appender is good enough for me. Use-case seems narrow except
>>> for a JPA shop.
>>> 
>>>> * Jackson based layouts (JsonLayout, XmlLayout, YamlLayout)
>>> -1: I want the ability to log to XML (narrow use-case, sure, but handy).
>>> 
>>>> * `log4j-mongodb3`:
>>> +1: I use mongodb4
>>> 
>>>> * `log4j-spring-boot`:
>>> -1: Spring seems too important.
>>> 
>>>> * Java EE SMTP appended:
>>> +1: Use the Jakarta version.
>>> 
>>>> * Jakarta EE SMTP appended:
>>> -1: Handy, sometimes.
>>> 
>>>> * `log4j-taglib`:
>>> +1, never used it.
>>> 
>>> Gary
>>> 
>>>> 
>>>> Please cast votes for each module/feature separately on this mailing
>> list:
>>>> 
>>>> [ ] +1,  drop the artifact/module,
>>>> [ ] +/-0
>>>> [ ] -1,  keep the artifact/module, because...
>>>> 
>>>> This vote is open for 168 hours (i.e. one week) and each deprecation
>>>> will pass unless getting a net negative vote count. All votes are
>>>> welcome, but only the Logging Services PMC votes are officially
>>>> counted.
>>>> 
>>>> Piotr
>> 
>> 

Re: Features for 3.x was: Why is JNDI still necessary?

[Matt Sicker]

I believe there is confusion here as to what level of compatibility 
requirements we set at the release of 3.0.0. Some remaining changes proposed 
for 3.x are technically breaking changes depending on the level of API 
stability in question. Personally, I think we should demarcate what APIs are 
truly stable public APIs and which ones are not, but we do a fairly good job of 
that with the API module itself at least (which is not where any of the 
proposed breaking changes happen to be as far as I know).

I do have experience working in codebases where backward compatibility is 
demanded above all else (Jenkins mainly, though our own API here is similar), 
so it’s not like I’m against it. However, I think it’s important to be 
realistic about what that API compatibility guarantee is, especially if you 
don’t already have some sort of stable meta-API like Linux does with system 
calls (which is itself a unique implementation detail).
—
Matt Sicker

> On Nov 2, 2023, at 21:36, Ralph Goers  wrote:
> 
> 
> 
>> On Nov 2, 2023, at 10:30 AM, Christian Grobmeier  
>> wrote:
>> 
>> 
>> 
>> On Thu, Nov 2, 2023, at 17:04, Gary Gregory wrote:
>>> Log4j 3 matching Spring 3 seems obviously a good thing to me. Going to Java
>>> 17 seems to me as well a good thing.
>>> 
>>> Gary
>>> 
>>> On Thu, Nov 2, 2023, 7:04 AM Ralph Goers  wrote:
>>> 
>>>> I should add that I am concerned that we are missing a huge opportunity
>>>> with Spring 3. A lot of folks will start their migration to Spring 3 early
>>>> next year. Tying Log4J 3.x to that is a big opportunity for people to
>>>> upgrade at the same time.
>> 
>> If this is the case, then how would we achieve the goal of releasing it 
>> together?
> 
> Releasing what together? Spring Boot 3 is already GA. It is using Log4j 2.x 
> currently. However, with Log4j 3.x targeting Java 11 and 17 (Spring Boot 3 
> requires Java 17) it will be a better choice for most users. I would also not 
> be surprised to see Spring Boot update their dependencies to use Log4j 3.x at 
> some point since they won’t require any code changes.
> 
>> 
>> With flume, audit and the other stuff eating time, it will be hard to get 
>> this done.
> 
> Get what done? As far as I am concerned Log4j 3.x can go to beta now and GA 
> the first of the year. There are no required features left, just stuff that 
> would be nice to get done but can be done after the initial GA release.
> 
>> 
>> This workload is too much for this handful of people. Removing load was btw 
>> also the idea of retiring components, so we can spend this time on other 
>> things of interest
> 
> Again, I don’t know to what workload you are referring.
> 
>> 
>> I would like to suggest that we have a team call soon where we collect our 
>> ideas of how we can reach that and refresh our shared vision.
> 
> I’m pretty much always up for a team call.
> 
>> 
>> Btw what is meant with spring 3? Spring boot is 3.1.15 already, spring 
>> framework is 6.x.
> 
> Spring Boot 3.x
> 
> Ralph

Re: [Discuss]: [VOTE] Deprecation of 2.x modules/features and removal in 3.x

[Matt Sicker]

What’s great about JTL is that it generates valid YAML, too! YAML is a superset 
of JSON. :)

> On Nov 2, 2023, at 11:15 AM, Ralph Goers  wrote:
> 
> IMO JSONLayout should be deprecated for 2 reasons:
> 1. JTL is so much better
> 2. It doesn’t generate great JSON.
> 
> As for the YAMLLayout, I have no idea who would want that. If they did there 
> has got to be a better solution than using Jackson. To be honest, it would be 
> great if JTL could be generalized to handle all 3, but that may be asking too 
> much.
> 
> Ralph
> 
>> On Nov 2, 2023, at 8:43 AM, Volkan Yazıcı  wrote:
>> 
>> On Thu, Nov 2, 2023 at 4:04 PM Ralph Goers 
>> wrote:
>> 
 On Nov 2, 2023, at 7:42 AM, Gary Gregory  wrote:
 On Mon, Oct 30, 2023 at 4:45 AM Piotr P. Karwasz
> * Jackson based layouts (JsonLayout, XmlLayout, YamlLayout)
 -1: I want the ability to log to XML (narrow use-case, sure, but handy).
>> 
>> 
>> Gary, do you only want to keep `XmlLayout` or all 3?
> 

Re: [VOTE] Deprecation of 2.x modules/features and removal in 3.x

[Matt Sicker]

Votes below

> On Oct 30, 2023, at 3:44 AM, Piotr P. Karwasz  wrote:
> 
> This is a vote to deprecate the following `2.x` modules and features
> and remove them from the `3.x` release:
> 
> * `log4j-cassandra`:
+1 deprecate
> * CouchDB appender:
+1 deprecate
> * `log4j-docker`
-0
> * GELF appender:
+1 deprecate
> * Kafka appender:
+1 deprecate
> * `log4j-kubernetes`:
-0
> * JeroMQ appender:
+1 deprecate
> * JNDI-related features:
-0, would only deprecate if we deprecate the things that require it, too
> * `log4j-jpa`:
+0, sort of seems redundant with the JDBC plugins, but I’m not super familiar
> * Jackson based layouts (JsonLayout, XmlLayout, YamlLayout)
+1 deprecate
> * `log4j-mongodb3`:
+1 deprecate, this version of mongo is ancient
> * `log4j-spring-boot`:
+1 deprecate, Spring Boot 3 has better built-in support for this already
> * Java EE SMTP appender:
-0, all programs eventually learn how to send email, but it’s the old package 
name
> * Jakarta EE SMTP appender:
-1, all programs eventually learn how to send email, current API
> * `log4j-taglib`:
+1 deprecate

Re: Features for 3.x was: Why is JNDI still necessary?

[Matt Sicker]

If we actually annotate the various public APIs like how JUnit 5 does, then we 
could have more flexibility to remove things.

> On Nov 2, 2023, at 4:10 AM, Piotr P. Karwasz  wrote:
> 
> Hi Ralph,
> 
> On Thu, 2 Nov 2023 at 09:42, Apache  wrote:
>> I’m confused. 3.0.0 hasn’t even been released so how can I be preventing 
>> adding anything. Personally I would prefer the monitoring to be in a 
>> separate repo but I am ok with adding it to the main build. IAM all for 
>> moving async out but unless it can be done quickly I’d rather do it in a 
>> future 3.x release. The same is generally true for your other bullets as 
>> well.
> 
> What I meant: we can not **remove** anything after 3.0.0 GA is
> released, this includes module refactorings.
> I know that you are pushing hard to have a beta by the end of this
> year and GA shortly after.
> 
> The problem is: the development of 3.x has been stagnating since June.
> Since then I have only seen Matt, Volkan and myself committing
> something.
> Since most of these changes can not be done in our day jobs, a more
> realistic release date for 3.x is the end of the year 2024.
> Right now the JLink project I added to Samples works with 2.x, but
> **fails** with 3.x (besides, 3.x snapshots are not getting published
> since a month and I don't have the faintest idea why).
> 
> Piotr

Re: [Log4j] Main website page

[Matt Sicker]

Oh, I thought you meant you updated the website itself already.

> On Nov 1, 2023, at 3:03 PM, Volkan Yazıcı  wrote:
> 
> It is the last commit to the `logging-log4j2` repository:
> https://github.com/apache/logging-log4j2/commit/96eb0d00099b709663a251484f2967cbb93b0230
> 
> On Wed, Nov 1, 2023 at 5:23 PM Ralph Goers 
> wrote:
> 
>> The Log4j site or the Logging site?  Where can we review this?
>> 
>> Ralph
>> 
>>> On Nov 1, 2023, at 9:07 AM, Volkan Yazıcı  wrote:
>>> 
>>> Heads up! I have updated the main website page: removed the clutter,
>> added
>>> quick navigation links, and rewrote the pitch for features.
>> 
>> 

Re: Why is JNDI still necessary?

[Matt Sicker]

> On Nov 1, 2023, at 2:22 PM, Christian Grobmeier  wrote:
> 
> We should not underestimate the impact log4shell had. Jndi was at the epi 
> center. Us, providing a giant jar including so much stuff with potential 
> security holes don’t do us a favor.

This is exactly why in 3.x, the main branch, most of the non-core functionality 
was removed from the core jar. JNDI support is offloaded into its own dedicated 
jar, and besides that, you still need to specify a system property to enable 
it. I still think there’s more that can be done in core to remove other 
dependencies, but it’s already at the point where it only (or almost only) 
requires the java.base module (while still bundling some optional things that 
should be split out).

To answer your other question, this makes it its own dependency. I’m not sure 
if any other modules besides the JMX GUI are being split into their own 
dedicated git repositories, but 3.x has significantly split up the bloat of 
optional plugins into their own modules.

Re: Why is JNDI still necessary?

[Matt Sicker]

I don’t see any reason why we shouldn’t publish the JNDI support in its own 
module as we’re planning in main already. Whether we eventually split out 
anything from the main repo is another story, but in 3.x, JNDI, like most of 
the optional features, will require downloading additional dependencies to 
explicitly opt in to using the feature (plus the JNDI stuff has an additional 
property you have to set to enable it, so you can’t end up with JNDI 
integration by accident even if the dependency is available). That seems fairly 
safe to me, and the naysayers at that point tend to be the same people who 
prefer printf-logging or using C because they’re elite programmers who never 
make mistakes and always design things perfectly decades ahead of time.

> On Nov 1, 2023, at 7:18 AM, Apache  wrote:
> 
> 
> If you want separate logging config files in an EJB environment using JNDI 
> is pretty much required. The same would be true for any properties needed in 
> the configuration. 
> 
> In any case, despite Piotr saying this is a majority vote, it is not. I will 
> veto any attempt to remove JNDI components so one really needs to view this 
> vote in the light that any component receiving a -1 vote cannot be removed as 
> it will simply get a -1 when the PR/commit is attempted from whomever gave 
> the -1 vote. In other words, this is NOT a procedural vote but a code 
> modification vote that is taking place before the code is modified.
> 
> Ralph
> 
>> On Nov 1, 2023, at 3:13 AM, Christian Grobmeier  wrote:
>> 
>> On Wed, Nov 1, 2023, at 00:01, Ralph Goers wrote:
>>> There is a difference between a JEE application that only uses servlets 
>>> vs one that uses EJBs. At a former employer we often used JBoss to run 
>>> servlets even though we had no EJBs. In an environment with EJBs I am 
>>> not sure how you can distinguish the various components from each other 
>>> without JNDI.
>> 
>> I understand that in the EJB world you would need JNDI, but I fail to 
>> understand how your EJBs would use Log4j JNDI features? It appears to me 
>> that while EE world needs it, the Logging world does not necessary need it
>> 
>>> 
>>> Ralph
>>> 
 On Oct 31, 2023, at 3:12 PM, Volkan Yazıcı  wrote:
 Piotr, I think it is important to differentiate what is a requirement and
 what is just another way of achieving something. My employer has several
 Tomcat- and JBoss-based JEE applications (using Log4j) and we don't have a
 single JNDI usage I know of.
 I would like to hear "the functional need" that can't be done in a JEE
 application without JNDI. My emphasis is important, since "using JNDI" is
 not a functional need.
 On Tue, Oct 31, 2023 at 10:55 PM Piotr P. Karwasz 
 wrote:
> Hi Christian,
> On Tue, 31 Oct 2023 at 21:57, Christian Grobmeier 
> wrote:
>> I am surprised we still have JNDI in the code at all, but this made me
> curious:
>> why do JEE users need JNDI features for logging? Why can't they just use
> the normal log mechanism?
> JNDI is basically a bean container/factory that allows Java EE
> applications to retrieve database connection pools, message queues or
> mail sessions (and remote code as a bonus).
> For the JMS appender, JNDI is essential.
> For the JDBC appender there is an alternative: DBCP2 can provide a
> database connection pool via a special connection string.
> The SMTP appender does not use JNDI as far as I remember.
> Of course there is an alternative to JNDI also in the Java EE world
> (CDI), but it doesn't work with simple servlet containers like Tomcat.
> Piotr
> 

Re: [LOG4J2-3672] Time zone parsing in `FastDateParser`

[Matt Sicker]

I can’t think of any reason why we’d need fast date parsing. Unless it’s 
relevant in the rolling file appender family?

> On Nov 1, 2023, at 6:07 AM, Volkan Yazıcı  wrote:
> 
> As reported in LOG4J2-3672
> , `FastDateParser`[1]
> contains `DateFormatSymbols#getZoneStrings()`, which causes initialization
> and caching of all time zones, resulting in a ~3MB heap overhead on x86_64.
> The reporter also provided the PR #1848
> , though it needs
> several assumptions to hold to be effective.
> 
> Curious: *In the context of logging, does `FastDateParser` need to be fast
> while parsing?* We certainly need to *"format"* the instant fast. Though do
> we really need to *"parse"* it fast too?
> 
> [1] FDP was borrowed from Apache Commons Lang in 2015. I have checked the
> most recent version
> ,
> they still use `getZoneStrings()`.

Re: Why is JNDI still necessary?

[Matt Sicker]

I’m not sure how much people use this deployment model anymore, but JNDI was 
and still is at the core of JavaEE (now JakartaEE) dependency injection APIs. 
While CDI is the current way of using dependency injection there, CDI is 
compatible with JNDI and the other JavaEE tech that came in between (the 
resource loader API thing). Typically, a JavaEE application server (think like 
JBoss/Wildfly, WebSphere, Weblogic, etc.) would be configured with shared 
resources like database and JMS connection pools, and those would be obtainable 
via JNDI. A single application server could host multiple servlets or more 
advanced packaging formats. All this JNDI usage, however, relies on the Java 
provider, not the LDAP or DNS or similar providers, so it works without 
networking being involved.

Do note, though, that there are ways to use this same tech without JNDI, but 
that typically involves being part of some JavaEE lifecycle, using CDI for 
dependency injection, etc.

> On Oct 31, 2023, at 3:57 PM, Christian Grobmeier  wrote:
> 
> Hi,
> 
> on the recent deprecation vote I saw this vote (from Ralph):
> 
>> * JNDI-related features:
> -1 Unfortunately, JEE users require this.
> 
> I am surprised we still have JNDI in the code at all, but this made me 
> curious:
> why do JEE users need JNDI features for logging? Why can't they just use the 
> normal log mechanism?
> 
> To me, JNDI is a burned word in Logging. No matter what we say about it, 
> nobody will ever believe we will have this under control.
> 
> I am a strong +1 on removing this, even for 2.x. Users can have a separate 
> module they can add to their pom, if they really need it. I am aware of 
> versioning and this could be considered a breaking change, but after all, the 
> APIs are intact, and only the pom changes. 
> 
> That being said, I would like to understand better what "jndi related" 
> features in our code base are actually solving, because from marketing 
> perspective, they are super GAU.
> 
> Kind regards,
> Christian

Re: Serializable gone for 3.0?

[Matt Sicker]

Yeah I nuked it a while back.

> On Oct 27, 2023, at 5:44 AM, Gary Gregory  wrote:
> 
> Should we remove implements Serializable from classes for 3.0?
> 
> Gary

Re: Deterministic formatter

[Matt Sicker]

Seems reasonable to me. I can’t really tell what the difference is between the 
two resulting outputs, though.

> On Oct 25, 2023, at 3:06 PM, Piotr P. Karwasz  wrote:
> 
> Hi all,
> 
> On Thu, 21 Sept 2023 at 07:15, Piotr P. Karwasz  
> wrote:
>> Spotless supports many formatter plugins, the main ones being Eclipse,
>> Google and Palantir formatters. All three are IDE agnostic.
> 
> I created two PRs using the Google (AOSP variant)[1] and Palantir[2] 
> formatter.
> 
> If we were to adopt one of these we are facing around 15% of changes
> in LOCs (mostly wrapping of long lines). This seems acceptable to me,
> what do you think?
> 
> Piotr
> 
> [1] https://github.com/apache/logging-log4j2/pull/1910
> [2] https://github.com/apache/logging-log4j2/pull/1911

Re: [chainsaw] DB package?

[Matt Sicker]

If you use Log4j in Chainsaw, you could set up a JDBC appender or similar to 
forward things along. The Flume appender could be useful there, too, but that 
sort of moves the database configuration into your Flume config.

> On Oct 24, 2023, at 3:14 PM, Christian Grobmeier  wrote:
> 
> Hi,
> 
> in this package:
> org.apache.log4j.db
> 
> all classes are commented. I am tempted to just remove it in full, but I am 
> not sure what i was supposed to do.
> 
> Docs read:
> 
> "Provides means to append logging events into various databases."
> 
> Does that mean, Chainsaw would collect logs and then send it to databases? 
> Isn't it what Flume (et al) is expected to do?
> 
> Kind regards,
> Christian

Re: [log4j] Project URLs per major version

[Matt Sicker]

Considering I’m one of the only people who adds @since tags (or Javadocs in 
general), I think we’ll need some tooling to help with this. In Jenkins, we had 
some sort of script for this so that we could add “@since TODO” to new files, 
and the script would replace them all before tagging a release (which is more 
relevant when you have weekly releases).

> On Oct 23, 2023, at 6:28 AM, Volkan Yazıcı  wrote:
> 
> Agreed, we should use `@since` tags. Though this doesn't warrant a new
> website folder for each release, does it? Given we never disclosed these
> folders, I want to understand the value it delivered in the past that can't
> be achieved if we switch to using a single folder.
> 
> On Mon, Oct 23, 2023 at 12:49 PM Piotr P. Karwasz 
> wrote:
> 
>> Hi Volkan,
>> 
>> On Mon, 23 Oct 2023 at 10:40, Volkan Yazıcı  wrote:
>>> 
 it [doesn't] fully documents what was in prior releases
>>> 
>>> Why is this a problem? We document newly added features with "starting
>> from
>>> version X, Log4j ships Y...". Doesn't this address your concern?
>> 
>> I prefer to document each new public/protected class or method with a
>> `@since` tag. We often don't do that (e.g. even in 2.21.0 there are a
>> couple of new methods).
>> 
>> Maybe there is an automatic scanner that can do it.
>> 
>> Piotr
>> 

Re: [log4j] Annotation processing alternatives

[Matt Sicker]

So really, the question is whether we can make the generated service classes 
easier to create without annotation processing?

> On Oct 23, 2023, at 12:11 PM, Gary Gregory  wrote:
> 
> Staying from the built-in Service Loader is a recipe for even more
> custom code and complications. I say we stick with the built-in
> Service Loader.
> 
> Gary
> 
> On Sun, Oct 22, 2023 at 5:01 PM Piotr P. Karwasz
>  wrote:
>> 
>> Hi Matt,
>> 
>> On Sun, 22 Oct 2023 at 22:49, Matt Sicker  wrote:
>>> So now we come to your question about a factories file. That’s an 
>>> interesting idea, though it’s extremely similar to what we’re doing here 
>>> with ServiceLoader (though the split between META-INF/services/ files and 
>>> module-info.java provides/uses entries is confusing). Moving away from 
>>> ServiceLoader to a custom mechanism might be more flexible, though I’m not 
>>> very knowledgeable about how Java modules work with them.
>> 
>> I think that `ServiceLoader` is the only standard feature that is
>> supported by both OSGi and JPMS. If we move away from it, we'll need
>> some custom code to support it (or `opens`, etc). I wouldn't go that
>> way.
>> 
>> Piotr

Re: JDK 21 Is Now GA, a New VS Code Extension, and an Annotation Processing Heads-up

[Matt Sicker]

Yeah I think it’s mainly a documentation thing for 3.x.
—
Matt Sicker

> On Oct 20, 2023, at 18:06, Ralph Goers  wrote:
> 
> I don’t think this is a problem. Only users of Log4j 3.x should be using Java 
> 17 and up by the time this makes it to an LTS release. Log4j 3.x has put the 
> annotation processor in its own jar. When using JPMS the annotation processor 
> has to be explicitly specified so we really had no choice but to do this. I 
> suspect this isn’t documented as well as it should be though.
> 
> Ralph
> 
>> On Oct 20, 2023, at 10:38 AM, Matt Sicker  wrote:
>> 
>> Snippet from the JDK 21 announcement email that pertains to us. While the 
>> annotation processor was initially developed to try to be implicit and 
>> magic, as we’ve seen both in our build and in IDEs, annotation processing is 
>> typically either disabled by default (in IDEs) or sometimes requires manual 
>> setup anyways (like in our build). It sounds like starting in Java 22, 
>> implicit annotation processing will no longer be available. We’ll need to 
>> ensure our docs about this mention that you need to enable annotation 
>> processing in your build, too.
>> 
>>> Begin forwarded message:
>>> 
>>> From: David Delabassee 
>>> Subject: JDK 21 Is Now GA, a New VS Code Extension, and an Annotation 
>>> Processing Heads-up
>>> Date: October 20, 2023 at 4:39:07 AM CDT
>>> To: "dev@logging.apache.org" 
>>> Reply-To: dev@logging.apache.org
>>> List-Id: 
>>> 
>>> ## Heads-Up - JDK 22: Implicit Annotation Processing Behavior Change
>>> 
>>> As discussed in the July 2023 Quality Outreach update [8], starting in JDK 
>>> 21 javac emits a note if _implicit_ annotation processing is being used, 
>>> that is, if one or more annotation processors are found and run from the 
>>> class path when no explicit annotation processing configuration options are 
>>> used.
>>> 
>>> The note is reported since, quoting from the note text: "A future release 
>>> of javac may disable annotation processing unless at least one processor is 
>>> specified by name (-processor), or a search path is specified 
>>> (--processor-path, --processor-module-path), or annotation processing is 
>>> enabled explicitly (-proc:only, -proc:full)."
>>> 
>>> That future version of javac has arrived in JDK 22 b19+ with JDK-8306819 
>>> ("Consider disabling the compiler's default active annotation processing"). 
>>> In the situation where a note was emitted in JDK 21, in JDK 22 no note is 
>>> emitted, and annotation processors are *not* run. To restore the previous 
>>> behavior with respect to running annotation processors, add the 
>>> '-proc:full' javac option.
>>> 
>>> Feedback on the annotation processing policy change can be sent to 
>>> compiler-dev [9].
>>> 
>>> [8] https://mail.openjdk.org/pipermail/quality-discuss/2023-July/001122.html
>>> [9] https://mail.openjdk.org/mailman/listinfo/compiler-dev
>> 
> 

Re: [log4j] Annotation processing alternatives

[Matt Sicker]

Let me expand on this question with further context as it relates to why the 
annotation processor exists in the first place. Back when the initial plugin 
system was developed by Ralph, we had an option to specify a comma-separated 
list of packages to scan for plugins (note that scanning the entire classpath 
was and still is a poor way to implement this), and we had a default package in 
the code to scan for log4j-core plugins. Everything still leads back to this 
general need for discovering the list of classes and their corresponding plugin 
metadata (primarily the plugin name which is used for configuring things like 
`…` or `…` rather than 
`…` as well as 
the category/namespace) so that various configurable classes are instantiated 
from the parsed configuration data. This can be sort of compared to use of 
things like type discriminators in JSON where types are all resolved against 
plugin names rather than arbitrary class names.

To alleviate this scan, we started with a cache file that dumped out metadata 
about plugins discovered from reflection-based classpath scanning (via a class 
in log4j-core with a main method). Later on, we added an annotation processor 
that did the same thing as this bundled program. In 3.x, we replaced this cache 
file with generated Java code implementing a ServiceLoader-loaded class that is 
almost exactly the same thing as what the cache file did before (serialize and 
deserialize a list of PluginEntry instances through some format supported by 
core Java).

So now we come to your question about a factories file. That’s an interesting 
idea, though it’s extremely similar to what we’re doing here with ServiceLoader 
(though the split between META-INF/services/ files and module-info.java 
provides/uses entries is confusing). Moving away from ServiceLoader to a custom 
mechanism might be more flexible, though I’m not very knowledgeable about how 
Java modules work with them.

If you examine the output of a generated plugin service file from the 
annotation processor, you may see that it’s something that you can easily write 
by hand, but it’s tedious as it’s replicating information from the plugin 
annotations into PluginEntry instances.

—
Matt Sicker

> On Oct 22, 2023, at 14:35, Volkan Yazıcı  wrote:
> 
> Good point Matt! Annotation processing has always been an unpleasant magic
> to work with from a developer perspective for the reasons you shared. What
> are our alternatives? Can't we offer a `META-INF/log4j.factories`
> functionality similar to Spring Boot? If so,
> 
>   1. What are its cons/pros for users?
>   2. What are its implementation requirements (for the Log4j crew) roughly?
> 
> 
> On Fri, Oct 20, 2023 at 7:40 PM Matt Sicker  wrote:
> 
>> Snippet from the JDK 21 announcement email that pertains to us. While the
>> annotation processor was initially developed to try to be implicit and
>> magic, as we’ve seen both in our build and in IDEs, annotation processing
>> is typically either disabled by default (in IDEs) or sometimes requires
>> manual setup anyways (like in our build). It sounds like starting in Java
>> 22, implicit annotation processing will no longer be available. We’ll need
>> to ensure our docs about this mention that you need to enable annotation
>> processing in your build, too.
>> 
>>> Begin forwarded message:
>>> 
>>> From: David Delabassee 
>>> Subject: JDK 21 Is Now GA, a New VS Code Extension, and an Annotation
>> Processing Heads-up
>>> Date: October 20, 2023 at 4:39:07 AM CDT
>>> To: "dev@logging.apache.org" 
>>> Reply-To: dev@logging.apache.org
>>> List-Id: 
>>> 
>>> ## Heads-Up - JDK 22: Implicit Annotation Processing Behavior Change
>>> 
>>> As discussed in the July 2023 Quality Outreach update [8], starting in
>> JDK 21 javac emits a note if _implicit_ annotation processing is being
>> used, that is, if one or more annotation processors are found and run from
>> the class path when no explicit annotation processing configuration options
>> are used.
>>> 
>>> The note is reported since, quoting from the note text: "A future
>> release of javac may disable annotation processing unless at least one
>> processor is specified by name (-processor), or a search path is specified
>> (--processor-path, --processor-module-path), or annotation processing is
>> enabled explicitly (-proc:only, -proc:full)."
>>> 
>>> That future version of javac has arrived in JDK 22 b19+ with JDK-8306819
>> ("Consider disabling the compiler's default active annotation processing").
>> In the situation where a note was emitted in JDK 21, in JDK 22 no note is
>> emitted, and annotation processors are *not* run. To resto

Fwd: JDK 21 Is Now GA, a New VS Code Extension, and an Annotation Processing Heads-up

[Matt Sicker]

Snippet from the JDK 21 announcement email that pertains to us. While the 
annotation processor was initially developed to try to be implicit and magic, 
as we’ve seen both in our build and in IDEs, annotation processing is typically 
either disabled by default (in IDEs) or sometimes requires manual setup anyways 
(like in our build). It sounds like starting in Java 22, implicit annotation 
processing will no longer be available. We’ll need to ensure our docs about 
this mention that you need to enable annotation processing in your build, too.

> Begin forwarded message:
> 
> From: David Delabassee 
> Subject: JDK 21 Is Now GA, a New VS Code Extension, and an Annotation 
> Processing Heads-up
> Date: October 20, 2023 at 4:39:07 AM CDT
> To: "dev@logging.apache.org" 
> Reply-To: dev@logging.apache.org
> List-Id: 
> 
> ## Heads-Up - JDK 22: Implicit Annotation Processing Behavior Change
> 
> As discussed in the July 2023 Quality Outreach update [8], starting in JDK 21 
> javac emits a note if _implicit_ annotation processing is being used, that 
> is, if one or more annotation processors are found and run from the class 
> path when no explicit annotation processing configuration options are used.
> 
> The note is reported since, quoting from the note text: "A future release of 
> javac may disable annotation processing unless at least one processor is 
> specified by name (-processor), or a search path is specified 
> (--processor-path, --processor-module-path), or annotation processing is 
> enabled explicitly (-proc:only, -proc:full)."
> 
> That future version of javac has arrived in JDK 22 b19+ with JDK-8306819 
> ("Consider disabling the compiler's default active annotation processing"). 
> In the situation where a note was emitted in JDK 21, in JDK 22 no note is 
> emitted, and annotation processors are *not* run. To restore the previous 
> behavior with respect to running annotation processors, add the '-proc:full' 
> javac option.
> 
> Feedback on the annotation processing policy change can be sent to 
> compiler-dev [9].
> 
> [8] https://mail.openjdk.org/pipermail/quality-discuss/2023-July/001122.html
> [9] https://mail.openjdk.org/mailman/listinfo/compiler-dev

Re: Staging sites and repo convention

[Matt Sicker]

If this is only for staging URLs and doesn’t break the production URLs, this 
sounds reasonable.

And the git worktree stuff is new to me!

> On Oct 19, 2023, at 3:03 AM, Piotr P. Karwasz  wrote:
> 
> Hi,
> 
> Since now we have a fast release process It might happen (and it
> already did) that the voting periods for releases will not be
> disjoint.
> 
> That is why I would like to introduce a convention on the procedure to
> stage websites and Nexus repositories.
> 
> For websites I would propose:
> 
> 1. Every Git code repository uses a different staging domain name.
> E.g. `logging-log4j2` would set:
> 
> staging:
>  profile: log4j2
> 
> which will result in a https://logging-log4j2.staged.apache.org URI.
> For the `logging-log4j-site` website repo this will also entail that
> it will have multiple staging branches.
> 2. The `asf-staging` should not be protected. Before staging a website
> the Release Manager would perform:
> 
> git reset --hard origin/asf-site
> git push -f
> 
> hence ensuring that moving changes from the staging branch to
> `asf-site` will be usually a fast-forward and a simple cherry-pick
> `origin/asf-site..asf-staging` at worst.
> 
> For the staging Nexus repo I would propose using a comment to close
> the repo in the format:
> 
> `` version `` RC1
> 
> For example Volkan used "`logging-parent` version `10.2.0` RC`" on the
> 1204 repo and we can easily guess what that repo contains. ;-)
> 
> Piotr
> 
> PS: Maybe we could drop the `*-site` Git repositories except
> `logging-site` and move their content to an `asf-site/asf-staging`
> branch of the corresponding code repo.

Re: Failing tests (was: [VOTE] Release Apache Log4j 2.21.0)

[Matt Sicker]

I’ve seen that test fail in CI before as well. I wonder if there’s any firewall 
issues that can interfere with it, or if it relies on some potentially unusable 
state like a common port number.

> On Oct 16, 2023, at 8:06 AM, Ralph Goers  wrote:
> 
> I enabled that profile. Log4j-core-test failed after just over 23 minutes.
> 
> [INFO] 
> [INFO] Results:
> [INFO] 
> [ERROR] Failures: 
> [ERROR]   
> TlsSyslogAppenderTest>SyslogAppenderTest.testUDPAppender:80->SyslogAppenderTestBase.sendAndCheckLegacyBsdMessage:75->SyslogAppenderTestBase.checkTheNumberOfSentAndReceivedMessages:113
>  The number of received messages should be equal with the number of sent 
> messages expected:<1> but was:<0>
> [INFO] 
> [ERROR] Tests run: 2782, Failures: 1, Errors: 0, Skipped: 38
> 
> Ralph
> 
>> On Oct 15, 2023, at 12:32 PM, Piotr P. Karwasz  
>> wrote:
>> 
>> Hi Ralph,
>> 
>> On Sun, 15 Oct 2023 at 00:49, Ralph Goers  wrote:
>>> My first attempt at building from the source failed with
>>> 
>>> [ERROR]   
>>> SyslogAppenderCustomLayoutTest>SyslogAppenderTest.testUDPAppender:80->SyslogAppenderTestBase.sendAndCheckLegacyBsdMessage:75->SyslogAppenderTestBase.checkTheNumberOfSentAndReceivedMessages:113
>>>  The number of received messages should be equal with the number of sent 
>>> messages expected:<1> but was:<0>
>>> 
>>> My second attempt failed with
>>> 
>>> [ERROR] Failures:
>>> [ERROR]   SystemClockTest.testAfterWaitStillLessThan2Millis:43 diff too 
>>> large: 12 ==> expected:  but was: 
>>> 
>>> On the third try the build succeeded.
>>> 
>>> On the positive side, log4j-core-tests only takes a little over 4 minutes 
>>> now on my new MacBook Pro and the entire build completed in 13 minutes.
>> 
>> I am afraid we will continuously need to improve the parallelisation
>> of tests, but I am assuming you are not complaining? Scalability is
>> never linear, but I would expect that a `-Psequential-tests` run would
>> take something like 20-25 minutes on your computer?
>> 
>> Piotr
> 

Re: JDBC optimizations in 3.x

[Matt Sicker]

> On Oct 16, 2023, at 6:38 AM, Piotr P. Karwasz  wrote:
> 
> 2. There are two ways to map event data to columns: ColumnConfig and
> ColumnMapping. It is unclear to me which way is the recommended one.
> Since we can perform breaking changes in 3.x, could we remove one of
> these methods?

I added ColumnMapping in 2.8 originally for the Cassandra appender, but that 
was later integrated into the other database appenders. I suggest we stick with 
ColumnMapping as the more flexible approach.

Re: Status of log4j-audit

[Matt Sicker]

Log4j Audit has multiple components:

* Audit API for extending log4j-api with some additional audit logging APIs
* Tool for managing your audit event schemata and such (the web app thing)
* Tool for generating structured log classes from the event schemata

Thus, in typical use, you can (and should) specify what version of log4j-core 
you’re using for your application. As for maintenance, the web UI could use an 
overhaul (though it would likely involve a rewrite into something more popular 
like React), but the overall library is fairly compact. I do have an 
outstanding Jira issue for Log4j to work on a more fluent API for structured 
logging, and that sort of feature can be informed or inspired by log4j-audit.

As for the log4j-server question, I’d expect that if we adopt Flume into the 
PMC, then Flume would supersede the server examples since we’d have something 
far more mature and production-ready for such a use case.

> On Oct 10, 2023, at 1:33 PM, Christian Grobmeier  wrote:
> 
> Hello,
> 
> We have been talking about log4j-audit (same thread as with log4j-server).
> 
> I have checked today after seeing Piotr's message, and even after reading the 
> readme, I am still trying to figure out the purpose of this product. That 
> aside, I am concerned the last change was four years ago. -audit is depending 
> to Log4j 2.10, which is affected by log4shell.
> 
> I checked on the releases, and I see only RCs here:
> https://github.com/apache/logging-log4j-audit/tags
> But two releases here:
> https://logging.apache.org/log4j-audit/latest/download.html
> 
> What message are we sending?
> 
> As I understand it we are currently promoting software that contains 
> log4shell without any word of warning or any development plan on the horizon.
> 
> Do we have any development cycles left to fix at least the security issues, 
> with the Flume project probably merging into this project?
> 
> I am not asking for the "will power", but the "real power": if it is not 
> realistic to maintain this project, we should add warning labels, consider 
> EOL, and/or actively search for contributors.
> 
> I am willing to support a bit, but only if I understand the use of -audit :)
> 
> Kind regards,
> Christian

Re: [VOTE] Release Apache Log4j Scala 13.0.0

[Matt Sicker]

+1

I also agree that the Scala 3 reproducible build issue is minor.

> On Oct 10, 2023, at 6:21 AM, Volkan Yazıcı  wrote:
> 
> This is a vote to release the Apache Log4j Scala 13.0.0.
> 
> Website: https://logging.staged.apache.org/log4j/scala
> GitHub: https://github.com/apache/logging-log4j-scala
> Commit: 980f4ed0ba53f93d1514df65ddafb6f97396a975
> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j-scala
> Nexus: 
> https://repository.apache.org/content/repositories/orgapachelogging-1200
> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
> 
> Please download, test, and cast your votes on this mailing list.
> 
> [ ] +1, release the artifacts
> [ ] -1, don't release, because...
> 
> This vote is open for 72 hours and will pass unless getting a
> net negative vote count. All votes are welcome and we encourage
> everyone to test the release, but only the Logging Services PMC
> votes are officially counted. At least 3 +1 votes and more
> positive than negative votes are required.
> 
> === Reproducibility
> 
> The `log4j-api-scala_3.jar` artifact targeting Scala 3 in this release
> is not reproducible due to a bug in the Scala 3 compiler[1]. (The
> other 4 artifacts are reproducible!) Though I don't find this a
> release blocker since we can verify that the difference breaking the
> reproducibility is just a member ordering problem, which doesn't have
> any practical effect from a user's point of view.
> 
> [1] https://github.com/lampepfl/dotty/issues/18248
> 
>  What about CI checks?
> 
> Note that CI-based reproducibility checks pass, since the issue
> appears to be only observable when sources are built on hosts with
> different specs.
> 
>  How can I check the reproducibility myself?
> 
> export 
> NEXUS_REPO=https://repository.apache.org/content/repositories/orgapachelogging-1200
> ./mvnw verify artifact:compare -Dreference.repo=$NEXUS_REPO
> 
>  How can I view the difference myself?
> 
> ./mvnw package
> mkdir t && cd $_
> wget 
> https://repository.apache.org/content/repositories/orgapachelogging-1200/org/apache/logging/log4j/log4j-api-scala_3/13.0.0/log4j-api-scala_3-13.0.0.jar
> unzip log4j-api-scala_3-13.0.0.jar -d reference
> unzip ../log4j-api-scala_3/target/log4j-api-scala_3-13.0.0.jar -d local
> diff -r reference local
> # `diff` will point out that only `LoggingContext$.class` doesn't match
> javap 'reference/org/apache/logging/log4j/scala/LoggingContext$.class'
>> reference.LoggingContext
> javap 'local/org/apache/logging/log4j/scala/LoggingContext$.class' >
> local.LoggingContext
> diff -u reference.LoggingContext local.LoggingContext
> 
>  What is the difference?
> 
> Decompiling and comparing `LoggingContext$.class` files produce the
> following output:
> 
> --- reference.LoggingContext 2023-10-10 12:58:35.188544183 +0200
> +++ local.LoggingContext 2023-10-10 12:58:41.152551071 +0200
> @@ -114,30 +114,30 @@
>   public scala.collection.LazyZip2 lazyZip(scala.collection.Iterable);
>   public scala.collection.IterableFactory iterableFactory();
>   public scala.Function1 compose(scala.Function1);
> -  public double apply$mcDI$sp(int);
> -  public double apply$mcDJ$sp(long);
> -  public double apply$mcDF$sp(float);
> -  public double apply$mcDD$sp(double);
> +  public int apply$mcII$sp(int);
> +  public int apply$mcIJ$sp(long);
> +  public int apply$mcIF$sp(float);
> +  public int apply$mcID$sp(double);
>   public boolean apply$mcZI$sp(int);
>   public boolean apply$mcZJ$sp(long);
>   public boolean apply$mcZF$sp(float);
>   public boolean apply$mcZD$sp(double);
> -  public long apply$mcJI$sp(int);
> -  public long apply$mcJJ$sp(long);
> -  public long apply$mcJF$sp(float);
> -  public long apply$mcJD$sp(double);
> -  public void apply$mcVI$sp(int);
> -  public void apply$mcVJ$sp(long);
> -  public void apply$mcVF$sp(float);
> -  public void apply$mcVD$sp(double);
>   public float apply$mcFI$sp(int);
>   public float apply$mcFJ$sp(long);
>   public float apply$mcFF$sp(float);
>   public float apply$mcFD$sp(double);
> -  public int apply$mcII$sp(int);
> -  public int apply$mcIJ$sp(long);
> -  public int apply$mcIF$sp(float);
> -  public int apply$mcID$sp(double);
> +  public void apply$mcVI$sp(int);
> +  public void apply$mcVJ$sp(long);
> +  public void apply$mcVF$sp(float);
> +  public void apply$mcVD$sp(double);
> +  public double apply$mcDI$sp(int);
> +  public double apply$mcDJ$sp(long);
> +  public double apply$mcDF$sp(float);
> +  public double apply$mcDD$sp(double);
> +  public long apply$mcJI$sp(int);
> +  public long apply$mcJJ$sp(long);
> +  public long apply$mcJF$sp(float);
> +  public long apply$mcJD$sp(double);
>   public scala.Option unapply(java.lang.Object);
>   public scala.PartialFunction elementWise();
>   public scala.PartialFunction orElse(scala.PartialFunction);
> 
>  Why is it a simple ordering problem?
> 
> Since `grep -E 'public (.+) apply\$' | sort` executed against both
> `reference.LoggingContext` and `local.LoggingContext` 

Re: Breaking changes in Log4j API 3.x

[Matt Sicker]

Use PropertyEnvironment; it’s the public API. The only methods not exposed by 
it are internal ones like reload which are used by some tests.

> On Oct 9, 2023, at 6:37 PM, Ralph Goers  wrote:
> 
> 
> 
>> On Oct 9, 2023, at 2:14 PM, Piotr P. Karwasz  wrote:
>> 
>> On Mon, 9 Oct 2023 at 20:57, Ralph Goers  wrote:
>>> We cannot put users in a position where they cannot upgrade until all their 
>>> dependencies do.
>> 
>> I agree, at work I still had a lot of libraries that depended on
>> Commons Lang 2, although
>> 
>>> Note that Spring Boot builds with Log4j 2.x. It needs access to 
>>> PerformanceSensitive, PropertiesUtil, and PropertySource.
>> 
>> I would really prefer not to export `PropertiesUtil`, but at least we
>> can migrate many other internal classes.
> 
> Unfortunately, the way PropertiesUtil was implemented makes it impossible to 
> make private. Lots of things need access to properties.
> 
> Ralph
> 

Re: [log4j] LOG4J2-3669 Scala API license

[Matt Sicker]

I’m not sure which parts of the code are inspired by the other library. I 
mostly worked on the build files :)

> On Oct 4, 2023, at 2:02 AM, Volkan Yazıcı  wrote:
> 
> PJ states in LOG4J2-3669 
> that the NOTICE file should contain reference to Typesafe/Lightbend for
> "inspired code". Could somebody (Matt?) help there, please?

Re: New for 3.0: Ergonomic Default Settings

[Matt Sicker]

Oh right, JTL isn’t in core, but it could be!

And if we change the default layout to continue using a pattern layout, how 
about some colors and such like how Spring Boot defaults?

And yes, multiple default configurations was another idea I had, but I wasn’t 
sure how to phrase that.

> On Oct 3, 2023, at 11:57 AM, Ralph Goers  wrote:
> 
> Are you just talking about changing the current default configuration? Or are 
> you envisioning having more than one somewhow? 
> 
> If this is just about changing the current default configuration then I have 
> some concerns:
> 1. It always has to work. 
> 2. It cannot rely on optional components. JsonTemplateLayout is not in core 
> so it cannot be included in the default. However, we could detect if it is 
> available and use it if it is.
> 
> Changing from PatternLayout to JsonTemplateLayout that only makes sense if 
> you have tools to interpret the JSON. Looking at the raw JSON is painful. 
> I’ve done it.
> 
> Finally, my hope is that no one is using the default configuration. I can’t 
> really think of any application that should be using it. However, we could 
> create multiple defaults tailored to specific application types.
> 
> Ralph
> 
>> On Oct 3, 2023, at 9:35 AM, Matt Sicker  wrote:
>> 
>> I’ve had this idea for many years now, and as we get closer to 3.0, it seems 
>> like it’s time to consider the details. We can change the default 
>> configuration in 3.0 without being as surprising as any other version. I 
>> think we could use this as an opportunity to demonstrate some best practices 
>> and recommendations. For example, we could switch from PatternLayout to 
>> JsonTemplateLayout by default to help prevent common vulnerabilities related 
>> to log message forging (like putting a newline or similar and faking the log 
>> output in the following line). Then there’s an option related to direct 
>> console writing that we have disabled by default despite being benign. There 
>> are probably other settings I haven’t considered at the moment.
>> 
>> So what do you think? Any suggestions for the default configuration?
> 

Re: [VOTE] Release Apache Log4j Kotlin API 1.3.0 (RC2)

[Matt Sicker]

+1

Same comment about the `mvnw` script not being executable. The README could 
perhaps be updated with build instructions in the future.

> On Oct 3, 2023, at 4:13 AM, Volkan Yazıcı  wrote:
> 
> This is a vote to release the Apache Log4j Kotlin API 1.3.0 (RC2).
> 
> Website: https://logging.staged.apache.org/log4j/kotlin
> GitHub: https://github.com/apache/logging-log4j-kotlin
> Commit: 56197c6809fef293832539481b9fdbea0fc3d323
> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j-kotlin
> Nexus: 
> https://repository.apache.org/content/repositories/orgapachelogging-1192
> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
> 
> Please download, test, and cast your votes on this mailing list.
> 
> [ ] +1, release the artifacts
> [ ] -1, don't release, because...
> 
> This vote is open for 72 hours and will pass unless getting a
> net negative vote count. All votes are welcome and we encourage
> everyone to test the release, but only the Logging Services PMC
> votes are officially counted. At least 3 +1 votes and more
> positive than negative votes are required.
> 
> === Release Notes
> 
> This minor release bumps the Kotlin baseline to 1.6.21 and contains
> various small improvements.
> 
>  Added
> 
> * Added an extension property for storing a cached logger (#29)
> * Added facade APIs for manipulating the context map and stack (#30)
> * Added missing `catching` and `throwing` API methods in
>  `KotlinLogger` (#32)
> * Added JPMS support and used `org.apache.logging.log4j.api.kotlin`
>  for the module name
> 
>  Changed
> 
> * Updated Log4j dependency to `2.20.0`
> * Bumped `logging-parent` version to `10.1.1` and overhauled the
>  entire project infrastructure to take advantage of its goodies (#37)
> * Renamed OSGi `Bundle-SymbolicName` from `org.apache.logging.log4j.kotlin`
>  to `org.apache.logging.log4j.api.kotlin`
> * Migrated tests to JUnit 5
> * Bumped Kotlin and Kotlin Extensions baseline to `1.6.21`
>  and `1.6.4` respectively
> * Skipped deploying `log4j-api-kotlin-benchmark` and
>  `log4j-api-kotlin-sample` modules
> 
>  Removed
> 
> * Stopped exporting KDoc to HTML

Re: [VOTE] Release Apache Log4j Tools 0.5.0 (RC2)

[Matt Sicker]

+1

I should note that the included `mvnw` script is not executable in the zip.

> On Oct 3, 2023, at 3:29 AM, Volkan Yazıcı  wrote:
> 
> This is a vote to release the Apache Log4j Tools 0.5.0 (RC2).
> 
> Website: https://logging.staged.apache.org/log4j/tools
> GitHub: https://github.com/apache/logging-log4j-tools
> Commit: 540cbd7dfa97cf26dc83717f738d26ee9a6b6e22
> Distribution: https://dist.apache.org/repos/dist/dev/logging/log4j-tools
> Nexus: 
> https://repository.apache.org/content/repositories/orgapachelogging-1189
> Signing key: 0x077e8893a6dcc33dd4a4d5b256e73ba9a0b592d0
> 
> Please download, test, and cast your votes on this mailing list.
> 
> [ ] +1, release the artifacts
> [ ] -1, don't release, because...
> 
> This vote is open for 72 hours and will pass unless getting a
> net negative vote count. All votes are welcome and we encourage
> everyone to test the release, but only the Logging Services PMC
> votes are officially counted. At least 3 +1 votes and more
> positive than negative votes are required.
> 
> === Release Notes
> 
> This minor release contains various bug fixes and improvements.
> 
>  Added
> 
> * Added OSGi and JPMS support
> * Started publishing the project website
> 
>  Changed
> 
> * Made `author` element optional and bumped the XML schema version to
> `0.1.2` (#81)
> * Make `log4j-changelog-maven-plugin` thread-safe (#80)
> * Update `org.apache.logging:logging-parent` to version `10.1.1`  (#82)
> * Update `org.junit.jupiter:junit-jupiter-engine` to version `5.10.0`

New for 3.0: Ergonomic Default Settings

[Matt Sicker]

I’ve had this idea for many years now, and as we get closer to 3.0, it seems 
like it’s time to consider the details. We can change the default configuration 
in 3.0 without being as surprising as any other version. I think we could use 
this as an opportunity to demonstrate some best practices and recommendations. 
For example, we could switch from PatternLayout to JsonTemplateLayout by 
default to help prevent common vulnerabilities related to log message forging 
(like putting a newline or similar and faking the log output in the following 
line). Then there’s an option related to direct console writing that we have 
disabled by default despite being benign. There are probably other settings I 
haven’t considered at the moment.

So what do you think? Any suggestions for the default configuration?

Re: [chainsaw] What is necessary for a 2.2 release?

[Matt Sicker]

Jackson makes for a good library here because it also supports several data 
formats (YAML, XML, HOCON, and all sorts of others, both textual and binary) 
and is fairly extensible for hooking any custom serialization or 
deserialization logic you need. Given the desire to avoid Java serialization, 
it is perfectly reasonable to avoid XStream as that’s basically Java 
serialization using XML with all the pitfalls that entails (I dealt with this 
fairly extensively back in the Jenkins project which used an old fork of 
XStream for managing config files and state).

I haven’t used Commons Configuration before, but it seems fairly appropriate 
for this sort of thing as well.

> On Oct 2, 2023, at 1:43 PM, Christian Grobmeier  wrote:
> 
> On Mon, Oct 2, 2023, at 16:15, Robert Middleton wrote:
>> At least two reasons I can think of:
>> 1. Xstream doesn’t work on all java versions as you saw
>> 2. Simplify by using the commons config instead of rolling our own.
>> 
>> Each tab should now have just one config file, the idea being that you can
>> now share config files between people.  Before each tab had at least
>> two(maybe three) files.  One was the xml config, one was a java serialized
>> map.  Removing the java serialization is important.
> 
> OK. Do you think something based on Jackson would be good? It's JSON though.
> 
> From an example:
> 
> ObjectMapper objectMapper = new ObjectMapper();
> Car car = new Car("yellow", "renault");
> objectMapper.writeValue(new File("target/car.json"), car);
> 
> We could wrap this in some kind of class and make it accessible per "tab" (or 
> whatever).
> 
> 
>> 
>> -Robert Middleton
>> 
>> On Mon, Oct 2, 2023 at 6:12 AM Christian Grobmeier 
>> wrote:
>> 
>>> 
>>> On Mon, Oct 2, 2023, at 02:55, Robert Middleton wrote:
 Some(most?) of the settings should be saved:
 
>>> https://github.com/apache/logging-chainsaw/blob/5ccb3c8e55dffd4361c549c3bcdac3f3675f79e5/src/main/java/org/apache/log4j/chainsaw/prefs/SettingsManager.java#L191
 
 The stuff that is commented out should just be the old saving code that
 used XStream to save the data out.
>>> 
>>> You made it using this commit (thank you, its easy to track):
>>> 
>>> https://github.com/apache/logging-chainsaw/commit/75bedf98665188eef4d13e4bfbb4b0dae456f65e
>>> 
>>> One question: why did we remove Xstream? it seem there was an update late
>>> 2022 to address some security.
>>> Should we just re-enable it or was there other reasons too?
>>> 
>>> 
>>> 
>>> 
 
 -Robert Middleton
 
 On Sun, Oct 1, 2023 at 3:39 PM Christian Grobmeier >>> 
 wrote:
 
> 
> On Sun, Oct 1, 2023, at 21:28, Scott Deboy wrote:
>> The ability to route events to tabs is a core feature in the code -
>> that's how Chainsaw log messages end up in a Chainsaw-specific tab -
>> but the ability to control that routing via a 'routing expression' was
>> nuked from app-wide preferences - another thing we should bring back.
>> 
>> It looks like we lost a lot of prefs, both panel-level and app-wide
> prefs.
> 
> Yes, I think all prefs are somehow gone. At least everything that is
> writes to a file seems to be commented.
> I didn't remove those things yet, as they seemed to big and I didn't
> understand well how they'd work or how I would test (I lack the
>>> knowledge
> of how the UI should operate but only see what is there now)
> 
> 
>> 
>> Scott
>> 
>> On 10/1/23, Robert Middleton  wrote:
>>> I would say the saving/loading of settings is probably the main
>>> thing to
>>> fix - if I remember correctly, it kinda works at the moment.  Part of
> the
>>> issue with what it did before was that the settings were scattered
>>> among
>>> several different files with no apparent rhyme or reason, and
>>> converting
>>> them to one file I'm not sure if everything works.
>>> 
>>> The one feature that I'm pretty sure doesn't exist is the ability to
> have
>>> multiple log messages go to one tab, but I don't think that is
>>> critical
> for
>>> release.  In order to properly support that I think requires a bit
>>> more
>>> planning on both the UI side(so you can know how things are routed)
>>> and
> on
>>> the back-end side(to do the actual routing).
>>> 
>>> -Robert Middleton
>>> 
>>> On Sun, Oct 1, 2023 at 3:14 PM Christian Grobmeier <
> grobme...@apache.org>
>>> wrote:
>>> 
 On Sun, Oct 1, 2023, at 20:59, Scott Deboy wrote:
> It's great to see the contribution, thanks Christian!
> 
> I pulled down latest master and it looks like there are some UI
> glitches we should fix - for example, resizing the logger tree
>>> pane
> doesn't render correctly.
> 
> As I mentioned before, I assume there are a bunch of features we
>>> lost
> when we moved from log4j1 - some may not be critical, but I think