Re: [log4j] The shape of Log4j

On Tue, Jan 30, 2018 at 1:29 PM, Gary Gregory wrote: > > On Tue, Jan 30, 2018 at 1:05 PM, Matt Sicker wrote: > >> It looks like you can use jdeps for this. >> > > The code to support the packages option is commented out :-( even in > current snapshots:

Re: [log4j] The shape of Log4j

On Tue, Jan 30, 2018 at 1:05 PM, Matt Sicker wrote: > It looks like you can use jdeps for this. > The code to support the packages option is commented out :-( even in current snapshots:

Re: [log4j] The shape of Log4j

It looks like you can use jdeps for this. On 29 January 2018 at 16:11, Gary Gregory wrote: > On Mon, Jan 29, 2018 at 2:39 PM, Matt Sicker wrote: > > > On 29 January 2018 at 13:10, Gary Gregory > wrote: > > > > > > Speaking of

Re: [log4j] The shape of Log4j

On Mon, Jan 29, 2018 at 2:39 PM, Matt Sicker wrote: > On 29 January 2018 at 13:10, Gary Gregory wrote: > > > > Speaking of java.sql (and javax.sql): How can we get the Maven build to > > FAIL if log4-core Java codes uses java[x].sql code? > > > > I'm

Re: [log4j] The shape of Log4j

On 29 January 2018 at 13:10, Gary Gregory wrote: > > Speaking of java.sql (and javax.sql): How can we get the Maven build to > FAIL if log4-core Java codes uses java[x].sql code? > I'm not sure if there are any Maven plugins for enforcing this yet, but I'd imagine it'd be

Re: [log4j] The shape of Log4j

On 29 January 2018 at 15:04, Gary Gregory wrote: > > Those two are done in git master. > Thanks! > How would we deal with dependencies on Apache Commons Compress for example? > Just slice and dice the code to minimize the code depending on CC and put > that in a new

Re: [log4j] The shape of Log4j

On Sat, Jan 27, 2018 at 3:07 PM, Matt Sicker wrote: > Moving the Kafka and ZeroMQ appenders to their own modules sounds like a > good start! Appenders that require third party plugins seem to be a great > candidate for splitting into their own modules. > Those two are done in

Re: [log4j] The shape of Log4j

On Mon, Jan 29, 2018 at 7:20 AM, Gary Gregory wrote: > > > On Sun, Jan 28, 2018 at 9:35 PM, Matt Sicker wrote: > >> That's rather limiting. Here's what we're already using: >> >> * java.compiler: annotation processing (could potentially be split, but >>

Re: [log4j] The shape of Log4j

JNDI is used in log4j-web for being able to locate a LoggingContext or configuration (forget which), and there's a JNDI lookup in log4j-core. Might be other places, but those are the main ones I can remember. Essentially, anything that relies on javax.* other than for some select crypto and

Re: [log4j] The shape of Log4j

On Sun, Jan 28, 2018 at 9:35 PM, Matt Sicker wrote: > That's rather limiting. Here's what we're already using: > > * java.compiler: annotation processing (could potentially be split, but > this situation is already confusing enough for users) > * java.management: JMX > *

Re: [log4j] The shape of Log4j

On Sun, Jan 28, 2018 at 9:54 PM, Ralph Goers wrote: > I didn’t realize that java.base was that limited. Are these all required > at run time or only build? For example, the annotation processor is not > used when log4j is running so I would suspect it is required to

Re: [log4j] The shape of Log4j

I didn’t realize that java.base was that limited. Are these all required at run time or only build? For example, the annotation processor is not used when log4j is running so I would suspect it is required to be present in the build and I would think it would have to be present when the

Re: [log4j] The shape of Log4j

That's rather limiting. Here's what we're already using: * java.compiler: annotation processing (could potentially be split, but this situation is already confusing enough for users) * java.management: JMX * java.naming: JNDI * java.scripting: javascript/groovy/etc plugins * java.xml: XML

Re: [log4j] The shape of Log4j

Also, in Java 9 it must only require the java.base module. Ralph > On Jan 27, 2018, at 6:49 PM, Matt Sicker wrote: > > On 27 January 2018 at 16:18, Ralph Goers wrote: > >> The requirement is that log4j-core have no required dependencies. I should

Re: [log4j] The shape of Log4j

On 27 January 2018 at 16:18, Ralph Goers wrote: > The requirement is that log4j-core have no required dependencies. I should > have as few optional dependencies as possible. > That sounds perfectly reasonable. LMAX and Jackson are good examples. -- Matt Sicker

Re: [log4j] The shape of Log4j

> On Jan 27, 2018, at 9:39 AM, Gary Gregory wrote: > > Should a goal be that log4j-core have no external dependencies? The requirement is that log4j-core have no required dependencies. I should have as few optional dependencies as possible. Ralph

Re: [log4j] The shape of Log4j

Moving the Kafka and ZeroMQ appenders to their own modules sounds like a good start! Appenders that require third party plugins seem to be a great candidate for splitting into their own modules. On 27 January 2018 at 16:06, Gary Gregory wrote: > As a first cut I was

Re: [log4j] The shape of Log4j

As a first cut I was thinking of moving out Kafka and ZeroMQ. The other stuff is interesting... Gary On Jan 27, 2018 3:01 PM, "Matt Sicker" wrote: > On 27 January 2018 at 10:39, Gary Gregory wrote: > > > Should a goal be that log4j-core have no

Re: [log4j] The shape of Log4j

On 27 January 2018 at 10:39, Gary Gregory wrote: > Should a goal be that log4j-core have no external dependencies? > This would be difficult without shading in LMAX Disruptor. Either that, or we may need some log4j-core API tweaks to allow the AsyncLogger code to live in

Re: [log4j] The shape of Log4j

Should a goal be that log4j-core have no external dependencies? Gary On Sat, Jan 27, 2018 at 12:57 AM, Remko Popma wrote: > Perhaps https://issues.apache.org/jira/browse/LOG4J2-2226 (Log4j2 > modularization) can also function as the epic for moving towards the plugin >

Re: [log4j] The shape of Log4j

Perhaps https://issues.apache.org/jira/browse/LOG4J2-2226 (Log4j2 modularization) can also function as the epic for moving towards the plugin API that Matt mentioned. (Shameless plug) Every java main() method deserves http://picocli.info > On Jan 25, 2018, at 9:45, Matt Sicker

Re: [log4j] The shape of Log4j

On Thu, Jan 25, 2018 at 9:05 AM, Mikael Ståldal wrote: > Yes, but it is unfortunate that the worry about the main repo size make us > hesitate splitting out stuff from log4j-core into own modules. And, with perfect timing:

Re: [log4j] The shape of Log4j

Yes, but it is unfortunate that the worry about the main repo size make us hesitate splitting out stuff from log4j-core into own modules. On 2018-01-24 22:37, Ralph Goers wrote: They are not really orthogonal. 1. Log4j core contains some stuff that isn’t really “core” functionality., such

Re: [log4j] The shape of Log4j

On Wed, Jan 24, 2018 at 10:05 PM, Ralph Goers wrote: > Thanks for that Remko. > > We are about due for another Log4j 2 release. I won’t be able to get a > release of log4j-server and then move the plugins to the plugins project in > a short enough time to warrant

Re: [log4j] The shape of Log4j

Thanks for that Remko. We are about due for another Log4j 2 release. I won’t be able to get a release of log4j-server and then move the plugins to the plugins project in a short enough time to warrant holding off a Log4j 2 release. So I am going to say Gary should go ahead and add the mongo3

Re: [log4j] The shape of Log4j

+1 The added complexity of managing the various repo builds and compatibility outweighs the build-time related cost of adding this code to core, easily. Just consider the time required to RM these other repos. There is no savings here to be found. Scott On Jan 24, 2018 6:20 PM, "Remko Popma"

Re: [log4j] The shape of Log4j

If you're interested in reading about how Google uses their monorepo, they have a paper you can read through: https://research.google.com/pubs/pub45424.html It's not clear to me reading the paper if they do this for all of their projects(including public repos on github) or just in-house code; it

Re: [log4j] The shape of Log4j

I’m still not convinced that moving the items that Ralph enumerated out of core will make a significant difference in the time it takes to do a release (I believe other measures will be more effective), but fine. If Ralph wants this as a precondition to continue to perform the Log4j2 releases

Re: [log4j] The shape of Log4j

I'm going to sum up my view here: if we split up repos to include plugins that depend on log4j-core, then we first need to define and release a stable plugin API. If we can't guarantee BC for plugins, then there's no realistic way to both split out additional plugins and have them be officially

Re: [log4j] The shape of Log4j

> On Jan 24, 2018, at 3:02 PM, Gary Gregory wrote: > > On Wed, Jan 24, 2018 at 2:33 PM, Ralph Goers > > wrote: > >> >> >>> On Jan 24, 2018, at 1:56 PM, Gary Gregory >> wrote:

Re: [log4j] The shape of Log4j

On Wed, Jan 24, 2018 at 2:33 PM, Ralph Goers wrote: > > > > On Jan 24, 2018, at 1:56 PM, Gary Gregory > wrote: > > > > On Wed, Jan 24, 2018 at 1:18 PM, Ralph Goers > > > wrote: > > > >> A vote can’t override a

Re: [log4j] The shape of Log4j

On Wed, Jan 24, 2018 at 2:33 PM, Ralph Goers wrote: > > > > On Jan 24, 2018, at 1:56 PM, Gary Gregory > wrote: > > > > On Wed, Jan 24, 2018 at 1:18 PM, Ralph Goers > > > wrote: > > > >> A vote can’t override a

Re: [log4j] The shape of Log4j

They are not really orthogonal. 1. Log4j core contains some stuff that isn’t really “core” functionality., such as Kafka or JMS. These cause the core tests to take longer than necessary. 2. The size of the repo directly effects how long a build takes to run, as does the number of components in

Re: [log4j] The shape of Log4j

> On Jan 24, 2018, at 1:56 PM, Gary Gregory wrote: > > On Wed, Jan 24, 2018 at 1:18 PM, Ralph Goers > wrote: > >> A vote can’t override a veto of a code commit. >> > > I'm trying to get things to move along toward some kind of workable >

Re: [log4j] The shape of Log4j

Yes, and it's unfortunate that they get conflated. On 2018-01-21 16:55, Gary Gregory wrote: I can see two main orthogonal issues: - The size of the git repo. - The size of the log4j-core module.

Re: [log4j] The shape of Log4j

Yes, neither Git nor Maven is optimal for mono-repos since you can only tag/branch the whole repo, and only have one version for the whole build. Google have a big mono-repo, but they do not use Git, nor Maven. I think we should release and version everything in one repo together, otherwise

Re: [log4j] The shape of Log4j

On Wed, Jan 24, 2018 at 2:15 PM, Mikael Ståldal wrote: > On 2018-01-22 23:37, Matt Sicker wrote: > >> On 22 January 2018 at 16:29, Ralph Goers >> wrote: >> >>> >>> If it was up to me I would move the following Appenders: Cassandra, >>> Flume, >>>

Re: [log4j] The shape of Log4j

On 2018-01-22 23:37, Matt Sicker wrote: On 22 January 2018 at 16:29, Ralph Goers wrote: If it was up to me I would move the following Appenders: Cassandra, Flume, JDBC, JMS, JPA, HTTP, Kafka, NoSQL, SMTP, ZeroMQ/JeroMQ, CouchDB, MongoDB. In addition to those I

Re: [log4j] The shape of Log4j

On Wed, Jan 24, 2018 at 1:27 PM, Ralph Goers wrote: > You left off two of the other options: > 1. A repo for related plugins. > 2. A repo for each plugin. > > I am open to any option where these plugins are not included in the main > repo. It sounds like the only

Re: [log4j] The shape of Log4j

On Wed, Jan 24, 2018 at 1:18 PM, Ralph Goers wrote: > A vote can’t override a veto of a code commit. > I'm trying to get things to move along toward some kind of workable solution we can all live with. Why are you talking about procedure? It just feels obstructionist

Re: [log4j] The shape of Log4j

You left off two of the other options: 1. A repo for related plugins. 2. A repo for each plugin. I am open to any option where these plugins are not included in the main repo. It sounds like the only option you are open to is including them in the main repo. Ralph > On Jan 24, 2018, at 1:00

Re: [log4j] The shape of Log4j

A vote can’t override a veto of a code commit. Ralph > On Jan 24, 2018, at 1:00 PM, Gary Gregory wrote: > > I really feel like we're not getting anywhere here :-( > > I have a log4j-mongodb3 appender ready to commit and nowhere to put it. The > empty

Re: [log4j] The shape of Log4j

I really feel like we're not getting anywhere here :-( I have a log4j-mongodb3 appender ready to commit and nowhere to put it. The empty repo/kitchen-sink/dumping-grounds log4j-plugins is not acceptable to me. Ralph does not want it in the main repo. I say it's fine. Now what? Should I call a

Re: [log4j] The shape of Log4j

> On Jan 23, 2018, at 6:51 AM, Gary Gregory wrote: > > On Jan 23, 2018 5:09 AM, "Mikael Ståldal" wrote: > > There is no NoSQL appender. There are Cassandra, CouchDB and MongoDB > appenders. I don't think we should bundle them together. > > > Note

Re: [log4j] The shape of Log4j

On Jan 23, 2018 5:09 AM, "Mikael Ståldal" wrote: There is no NoSQL appender. There are Cassandra, CouchDB and MongoDB appenders. I don't think we should bundle them together. Note that these are already separate modules/jar produced out of the main repo. Gary On

Re: [log4j] The shape of Log4j

There is no NoSQL appender. There are Cassandra, CouchDB and MongoDB appenders. I don't think we should bundle them together. On 2018-01-22 23:29, Ralph Goers wrote: If it was up to me I would move the following Appenders: Cassandra, Flume, JDBC, JMS, JPA, HTTP, Kafka, NoSQL, SMTP,

Re: [log4j] The shape of Log4j

> On Jan 23, 2018, at 7:29, Ralph Goers wrote: > > IMO the main repo should contain the stuff 80% (or more) of our user’s use > and the stuff that is common across all plugins. So the file appenders and > console appenders all belong, probably most of the existing

Re: [log4j] The shape of Log4j

On Sun, Jan 21, 2018 at 4:55 PM, Gary Gregory wrote: > - Put everything back into one repo (Chainsaw too?) In general, yes. However, Chainsaw would be one step too far. From my point of view, Chainsaw is a consumer, and not a part of log4j-core. At the most, split a

Re: [log4j] The shape of Log4j

> On Jan 22, 2018, at 5:10 PM, Gary Gregory wrote: > > On Mon, Jan 22, 2018 at 4:45 PM, Ralph Goers > wrote: >> > "When you have a chart like Maven does users can easily deal with figuring > out what versions they want to use." > > That

Re: [log4j] The shape of Log4j

You are making a mountain out of a mole hill. If a plugin released for 2.8.2 doesn’t work with 2.11.0 we are doing something wrong. Users shouldn’t need to inspect anything. The problem with your premise is assuming these components require some specific version of log4j-core. Ralph > On

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 4:45 PM, Ralph Goers wrote: > > > > On Jan 22, 2018, at 4:26 PM, Gary Gregory > wrote: > > > > On Mon, Jan 22, 2018 at 4:22 PM, Ralph Goers > > > wrote: >

Re: [log4j] The shape of Log4j

My recollection is vague, tied to pom machinations supporting a similar separation for the 'log4j companions': outside of core log4j - chainsaw, component, extras, receivers and zeroconf' Some of the issues I recall having to deal with when we had the generic 'component' module: - Some classes

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 4:22 PM, Ralph Goers wrote: > > > > On Jan 22, 2018, at 4:02 PM, Gary Gregory > wrote: > > > > On Mon, Jan 22, 2018 at 3:54 PM, Ralph Goers > > > wrote: > > > >> > >> > >>> On Jan 22, 2018,

Re: [log4j] The shape of Log4j

> On Jan 22, 2018, at 4:02 PM, Gary Gregory wrote: > > On Mon, Jan 22, 2018 at 3:54 PM, Ralph Goers > wrote: > >> >> >>> On Jan 22, 2018, at 3:44 PM, Gary Gregory >> wrote: >>> >>> On Mon, Jan 22, 2018 at 3:25

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 3:54 PM, Ralph Goers wrote: > > > > On Jan 22, 2018, at 3:44 PM, Gary Gregory > wrote: > > > > On Mon, Jan 22, 2018 at 3:25 PM, Matt Sicker wrote: > > > >> The profiles idea would help a lot for local

Re: [log4j] The shape of Log4j

> On Jan 22, 2018, at 3:44 PM, Gary Gregory wrote: > > On Mon, Jan 22, 2018 at 3:25 PM, Matt Sicker wrote: > >> The profiles idea would help a lot for local development and making it >> easier to run the relevant unit tests when modifying code. This

Re: [log4j] The shape of Log4j

On 22 January 2018 at 16:44, Gary Gregory wrote: > Ah... but you COULD make it help in the release just as well but putting > what you are releasing into the main profile! That's the beauty of it. > Sure, you'd tag the whole thing as one. Presumably, you'd think the RM >

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 3:25 PM, Matt Sicker wrote: > The profiles idea would help a lot for local development and making it > easier to run the relevant unit tests when modifying code. This wouldn't > really help the release process, though, as we'd either need to release >

Re: [log4j] The shape of Log4j

On 22 January 2018 at 16:29, Ralph Goers wrote: > IMO the main repo should contain the stuff 80% (or more) of our user’s use > and the stuff that is common across all plugins. So the file appenders and > console appenders all belong, probably most of the existing

Re: [log4j] The shape of Log4j

> On Jan 22, 2018, at 3:23 PM, Gary Gregory wrote: > > On Mon, Jan 22, 2018 at 2:58 PM, Matt Sicker wrote: > >> Using Maven profiles could help, though when tagging a release, if we >> didn't release every module within, then it'd get out of sync

Re: [log4j] The shape of Log4j

> On Jan 22, 2018, at 2:21 PM, Gary Gregory wrote: > > On Mon, Jan 22, 2018 at 12:58 PM, Matt Sicker wrote: > >> On 22 January 2018 at 13:48, Ralph Goers >> wrote: >> >>> I don’t see why this work would require 3.0 as we

Re: [log4j] The shape of Log4j

The profiles idea would help a lot for local development and making it easier to run the relevant unit tests when modifying code. This wouldn't really help the release process, though, as we'd either need to release everything in the repo at once, or we'd end up with unreleased modules being

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 2:58 PM, Matt Sicker wrote: > Using Maven profiles could help, though when tagging a release, if we > didn't release every module within, then it'd get out of sync anyways. Then > it would still require building every single component in each release, >

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 2:58 PM, Matt Sicker wrote: > Using Maven profiles could help, though when tagging a release, if we > didn't release every module within, then it'd get out of sync anyways. Then > it would still require building every single component in each release, >

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 3:03 PM, Scott Deboy wrote: > Correcting my prior comment: the nightmare I was referring to was the > ugliness we did and un-did related to the never-released receivers and > 'component' companions. Definitely feels like deja vu. > >

Re: [log4j] The shape of Log4j

Correcting my prior comment: the nightmare I was referring to was the ugliness we did and un-did related to the never-released receivers and 'component' companions. Definitely feels like deja vu. https://github.com/apache/log4j-component/blob/trunk/pom.xml apache-log4j-companions-parent etc etc

Re: [log4j] The shape of Log4j

Using Maven profiles could help, though when tagging a release, if we didn't release every module within, then it'd get out of sync anyways. Then it would still require building every single component in each release, even when said components haven't changed at all. On 22 January 2018 at 15:50,

Re: [log4j] The shape of Log4j

Coming back around to the idea of One Repo to Rule Them All*TM* ... what if we used Maven profiles to separate out the "main" build from "other" builds? All that is needed to move modules from one build to another is just to edit the profile! Gary On Mon, Jan 22, 2018 at 12:48 PM, Ralph Goers

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 2:25 PM, Gary Gregory wrote: > > > On Mon, Jan 22, 2018 at 2:21 PM, Gary Gregory > wrote: > >> >> >> On Mon, Jan 22, 2018 at 12:58 PM, Matt Sicker wrote: >> >>> On 22 January 2018 at 13:48, Ralph Goers

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 2:21 PM, Gary Gregory wrote: > > > On Mon, Jan 22, 2018 at 12:58 PM, Matt Sicker wrote: > >> On 22 January 2018 at 13:48, Ralph Goers >> wrote: >> >> > I don’t see why this work would require 3.0 as

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 12:58 PM, Matt Sicker wrote: > On 22 January 2018 at 13:48, Ralph Goers > wrote: > > > I don’t see why this work would require 3.0 as we aren’t planning on > > breaking any contracts to do this. > > > > I'm referring to

Re: [log4j] The shape of Log4j

I'm curious: What is threshold where it would be OK to add stuff to the main repo? Gary On Mon, Jan 22, 2018 at 12:48 PM, Ralph Goers wrote: > I don’t see why this work would require 3.0 as we aren’t planning on > breaking any contracts to do this. > > As I’ve said,

Re: [log4j] The shape of Log4j

On 22 January 2018 at 13:48, Ralph Goers wrote: > I don’t see why this work would require 3.0 as we aren’t planning on > breaking any contracts to do this. > I'm referring to something more in line with past proposals of having a sort of log4j-core-api or similar

Re: [log4j] The shape of Log4j

On Mon, Jan 22, 2018 at 12:36 PM, Matt Sicker wrote: > This whole conversation just reminds me of all the times in the past I > suggested full modularization of the core API/SPI. I think such an effort > would make sense in a 3.0 release. > > In 2.x, perhaps what we can do is

Re: [log4j] The shape of Log4j

I don’t see why this work would require 3.0 as we aren’t planning on breaking any contracts to do this. As I’ve said, I am not tied to each plugin having its own repo. I am OK with these options; a) each plugin has its own repo and site and is released independently, b) we use the plugins repo

Re: [log4j] The shape of Log4j

On Sun, Jan 21, 2018 at 5:51 PM, Remko Popma wrote: > Log4j2 is not like Maven. Maven (I assume) has a plugin API. Log4j2 > modules depend on the internals of log4j-core. > > I agree with Gary that not being able to verify that a core change doesn’t > break any modules

Re: [log4j] The shape of Log4j

This whole conversation just reminds me of all the times in the past I suggested full modularization of the core API/SPI. I think such an effort would make sense in a 3.0 release. In 2.x, perhaps what we can do is define a stable plugin API based on what we already have. Since we're still

Re: [log4j] The shape of Log4j

On Sun, Jan 21, 2018 at 4:14 PM, Ralph Goers wrote: > I am very much against the idea of a single repo. Yes, you can have > multiple projects in the repo but I am not sure how that can be sanely > released. I much prefer the model that Maven has taken. They are now

Re: [log4j] The shape of Log4j

On Sun, Jan 21, 2018 at 5:51 PM, Remko Popma wrote: > Log4j2 is not like Maven. Maven (I assume) has a plugin API. Log4j2 > modules depend on the internals of log4j-core. > > I agree with Gary that not being able to verify that a core change doesn’t > break any modules

Re: [log4j] The shape of Log4j

FWIW, Some of this is borne out by looking at the stats at https://repository.apache.org/#central-stat . The good news is that the downloads are 3 times what they were a year ago. But you can see that the usage of these extra components is 1000

Re: [log4j] The shape of Log4j

Remko, I am all for speeding up the build. I have been complaining about this for well over a year and it keeps getting worse with every release. The compile of log4j-core takes less than 20 seconds on the machine provided by my employer. On the same machine the tests take 11 minutes. On this

Re: [log4j] The shape of Log4j

I am very much against the idea of a single repo. Yes, you can have multiple projects in the repo but I am not sure how that can be sanely released. I much prefer the model that Maven has taken. They are now using gitbox [1] which seems to allow GitHub to be the primary repo. Every Maven plugin