I have started to extract KafkaAppender to its own module, please review Git branch LOG4J2-1889. This can be a template on how to do it for other plugins.
On Tue, Apr 25, 2017 at 5:14 PM, Gary Gregory <[email protected]> wrote: > To be honest here, it feels to me like we are splitting hair. Maybe we > could proceed with m12n and then do an SPI. > > It also feels odd to call it SPI if it has a boatload of "stuff" in it like > our config code. > > Maybe we leave -core as in and pull out JRE-based appenders and layouts > into a -basic module. > > Gary > > On Apr 25, 2017 8:03 AM, "Matt Sicker" <[email protected]> wrote: > > > No, it would define the interfaces (and abstract classes so that we don't > > need to introduce Appender2, Appender3, etc., without having default > > methods available) along with the core configuration and plugin system. > > Essentially all the parts of log4j-core that we want to create a > > well-defined service provider interface for so that not only can users > > define custom plugins more safely, but also as a way for us to eat our > own > > dog food and make sure it works well enough like that. It could in theory > > allow for a log4j-core 3.0 in the future while still running with > log4j-api > > 2.x and log4j-spi 2.x. > > > > On 25 April 2017 at 09:39, Gary Gregory <[email protected]> wrote: > > > > > Would SPI only contain interfaces? > > > > > > Gary > > > > > > On Apr 25, 2017 7:36 AM, "Mikael Ståldal" <[email protected]> > > > wrote: > > > > > > > The goal of log4j-spi is that plugins can depend on log4j-spi (and > > > > log4j-api), not on log4j-core. > > > > > > > > On Tue, Apr 25, 2017 at 4:33 PM, Gary Gregory < > [email protected]> > > > > wrote: > > > > > > > > > If my app depends on the BOM module, do I get everything so I can > > > > > assembly:single them into a zip with my app? > > > > > > > > > > Gary > > > > > > > > > > On Apr 25, 2017 7:09 AM, "Matt Sicker" <[email protected]> wrote: > > > > > > > > > > > I also agree that modules should really stick to required > > > dependencies. > > > > > > > > > > > > As for only requiring log4j-core, the idea here is that you could > > > still > > > > > > just do: > > > > > > > > > > > > compile 'org.apache.logging.log4j:log4j-core:2.+' > > > > > > > > > > > > or whatever the equivalent is for your build system, and you'll > > still > > > > get > > > > > > log4j-api and anything else required. You'd only need log4j-api > for > > > > > > logging, log4j-spi for writing custom plugins, or log4j-core to > get > > > the > > > > > > standard plugins. > > > > > > > > > > > > On 25 April 2017 at 09:07, Mikael Ståldal < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > I think we should aim for all modules having only required > > > > > dependencies. > > > > > > > > > > > > > > We should not add any new optional dependencies, and all new > > > modules > > > > > > should > > > > > > > only have required dependencies. > > > > > > > > > > > > > > Maybe we won't be able to get rid of all optional dependencies > in > > > > > > > log4j-core right away, though. Maybe we will release 2.9 with a > > few > > > > > > > optional dependencies left. > > > > > > > > > > > > > > On Tue, Apr 25, 2017 at 4:03 PM, Gary Gregory < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > We should have consensus on the big picture here... are we > all > > Ok > > > > > with > > > > > > > the > > > > > > > > idea of all modules only having _required_ dependencies? > > > > > > > > > > > > > > > > Gary > > > > > > > > > > > > > > > > On Apr 25, 2017 6:57 AM, "Remko Popma" < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > Nice analysis Mikael! > > > > > > > > > > > > > > > > > > I'm a bit fuzzy on log4j-spi, what was that for again? The > > list > > > > > says > > > > > > > > "core > > > > > > > > > will depend on spi" but I think it's worth making an > effort > > to > > > > > > ensure > > > > > > > > that > > > > > > > > > basic (file) logging behavior only needs core+API... Why > does > > > spi > > > > > > need > > > > > > > to > > > > > > > > > be separated from core? > > > > > > > > > > > > > > > > > > My first thought about jdbc was that since jdbc doesn't > > require > > > > > > > external > > > > > > > > > dependencies we should probably leave it in core. I guess > it > > > > > depends > > > > > > on > > > > > > > > > what we're trying to achieve with (or how far we want to > > take) > > > > the > > > > > > > > > modularization: do we want to reduce core to its absolute > > > minimum > > > > > or > > > > > > > are > > > > > > > > we > > > > > > > > > aiming to split off external dependencies? > > > > > > > > > > > > > > > > > > Looking at the list I can see how many of these make sense > > and > > > at > > > > > the > > > > > > > > same > > > > > > > > > time I'm thinking, that's a lot of modules! :-) > > > > > > > > > > > > > > > > > > Remko > > > > > > > > > > > > > > > > > > > > > > > > > > > (Shameless plug) Every main() method deserves > > > > http://picocli.info > > > > > > > > > > > > > > > > > > > On Apr 25, 2017, at 18:51, Mikael Ståldal < > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > I guess that log4-core will become: > > > > > > > > > > > > > > > > > > > > - log4j-core (will depend on log4j-spi) > > > > > > > > > > - log4j-spi > > > > > > > > > > - log4j-csv > > > > > > > > > > - log4j-xml (XmlLayout) > > > > > > > > > > - log4j-json (JsonLayout) > > > > > > > > > > - log4j-yaml (YamlLayout) > > > > > > > > > > - log4j-kafka > > > > > > > > > > - log4j-smtp > > > > > > > > > > - log4j-jms > > > > > > > > > > - log4j-jdbc (or can this be kept in log4j-core?) > > > > > > > > > > - log4j-jpa > > > > > > > > > > - log4j-zeromq > > > > > > > > > > - log4j-server (already done, not yet released) > > > > > > > > > > - log4j-tools (command line tools) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then we should also split log4j-nosql: > > > > > > > > > > > > > > > > > > > > - log4j-cassandra > > > > > > > > > > - log4j-couchdb > > > > > > > > > > - log4j-mongodb > > > > > > > > > > - log4j-lucene (new, under development) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Mon, Apr 24, 2017 at 7:43 PM, Remko Popma < > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >> How many new modules are we talking about, concretely? > > > > > > > > > >> > > > > > > > > > >> Matt mentioned the StackOverflow questions about > > transitive > > > > > > > > dependencies > > > > > > > > > >> etc, but I imagine splitting log4j-core into 5 or more > new > > > > > modules > > > > > > > > will > > > > > > > > > >> also cause confusion... It won't be trivial for users to > > > > figure > > > > > > out > > > > > > > > > which > > > > > > > > > >> of the many modules they do or don't need. The coarse > > > > > granularity > > > > > > of > > > > > > > > the > > > > > > > > > >> current modules is a good thing for users. > > > > > > > > > >> > > > > > > > > > >> What problem are we trying to solve? And how can we > solve > > it > > > > > with > > > > > > > the > > > > > > > > > least > > > > > > > > > >> disruption to our users? > > > > > > > > > >> > > > > > > > > > >> Would it be an idea, for example, to provide separate > jars > > > for > > > > > the > > > > > > > > > separate > > > > > > > > > >> modules, but in addition create a combined jar > > > > (log4j-core-all) > > > > > > that > > > > > > > > > >> contains all the classes in log4j-core as well as the > > > classes > > > > in > > > > > > the > > > > > > > > new > > > > > > > > > >> modules we split out from core? > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >>> On Tue, Apr 25, 2017 at 2:00 AM, Matt Sicker < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > >>> > > > > > > > > > >>> I agree with Ralph here. I'm sure we'll figure out > rather > > > > > quickly > > > > > > > > which > > > > > > > > > >>> modules are easy to put into rarely updated > repositories. > > > > > > > > > >>> > > > > > > > > > >>> On 24 April 2017 at 11:39, Ralph Goers < > > > > > > [email protected] > > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > >>> > > > > > > > > > >>>> I would prefer a hybrid approach. First things should > > be > > > > > moved > > > > > > to > > > > > > > > > >>>> separate modules. Then, if they don’t seem to be > > modified > > > > > > > frequently > > > > > > > > > >> they > > > > > > > > > >>>> can be moved to a separate repo. For example, I think > it > > > > would > > > > > > be > > > > > > > OK > > > > > > > > > >> for > > > > > > > > > >>>> the Flume Appender to be in a separate repo. It hasn’t > > > > changed > > > > > > in > > > > > > > > > >> quite a > > > > > > > > > >>>> while and I can’t remember the last time it was > modified > > > due > > > > > to > > > > > > > > > changes > > > > > > > > > >>> in > > > > > > > > > >>>> Log4j it has and while continue to change with changes > > > made > > > > in > > > > > > > Flume > > > > > > > > > >>>> releases. I imagine we have quite a few components > that > > > are > > > > > > > > similar. > > > > > > > > > >>>> > > > > > > > > > >>>> Ralph > > > > > > > > > >>>> > > > > > > > > > >>>>> On Apr 24, 2017, at 8:39 AM, Gary Gregory < > > > > > > > [email protected]> > > > > > > > > > >>>> wrote: > > > > > > > > > >>>>> > > > > > > > > > >>>>> On Apr 24, 2017 2:38 AM, "Mikael Ståldal" < > > > > > > > > [email protected] > > > > > > > > > >>> > > > > > > > > > >>>> wrote: > > > > > > > > > >>>>> > > > > > > > > > >>>>> I fully agree with Matt's both proposals. > > > > > > > > > >>>>> > > > > > > > > > >>>>> I'm skeptic to creating more repositories (than we > > > already > > > > > > have) > > > > > > > > > >>> though. > > > > > > > > > >>>> I > > > > > > > > > >>>>> think that we should start by splitting out modules > > from > > > > > > > log4j-core > > > > > > > > > >> and > > > > > > > > > >>>>> keep those modules in the main repository with > > > synchronized > > > > > > > > > >> versioning > > > > > > > > > >>>> and > > > > > > > > > >>>>> releases, at least for the 2.9 release. We can always > > > move > > > > > > those > > > > > > > > > >>> modules > > > > > > > > > >>>> to > > > > > > > > > >>>>> other repositories later if we want to. > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> I do not like more repos either. Since we have > already > > > gone > > > > > > down > > > > > > > > the > > > > > > > > > >>> more > > > > > > > > > >>>>> modules road, I say we keep going. > > > > > > > > > >>>>> > > > > > > > > > >>>>> Gary > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> It is a lot of administrative work to create a new > > > > repository > > > > > > (as > > > > > > > > we > > > > > > > > > >>> have > > > > > > > > > >>>>> seen for log4j-scala), I don't want us to do all that > > > work > > > > > over > > > > > > > and > > > > > > > > > >>> over > > > > > > > > > >>>>> again unless really necessary. > > > > > > > > > >>>>> > > > > > > > > > >>>>> We have a JIRA ticket for this: > > > > > > > > > >>>>> https://issues.apache.org/jira/browse/LOG4J2-1650 > > > > > > > > > >>>>> > > > > > > > > > >>>>> I have already started by breaking out log4j-server: > > > > > > > > > >>>>> https://issues.apache.org/jira/browse/LOG4J2-1851 > > > > > > > > > >>>>> > > > > > > > > > >>>>> I think the next step is to break out plugins > (layouts > > > and > > > > > > > > appenders) > > > > > > > > > >>>> with > > > > > > > > > >>>>> optional 3rd party dependencies into their own > modules. > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> On Sun, Apr 23, 2017 at 7:45 PM, Matt Sicker < > > > > > [email protected] > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > >>>>> > > > > > > > > > >>>>>> I think I brought this topic up like 3 years ago > when > > I > > > > was > > > > > > > > working > > > > > > > > > >> on > > > > > > > > > >>>>>> initial OSGi support, but now that we have 3 more > > years > > > > > worth > > > > > > of > > > > > > > > > >> code > > > > > > > > > >>>>>> additions and optional features, I think this might > > be a > > > > > more > > > > > > > > > >>>> appropriate > > > > > > > > > >>>>>> time to discuss it again in light of experience. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Building log4j-core itself already takes a long > time, > > > and > > > > > many > > > > > > > > > >> plugins > > > > > > > > > >>>>>> aren't updated very often at all. In the past, > > requiring > > > > > users > > > > > > > to > > > > > > > > > >>> simply > > > > > > > > > >>>>>> add log4j-core plus any transitive dependencies to > use > > > > > > optional > > > > > > > > > >>> features > > > > > > > > > >>>>>> seemed to work well enough, but I still think > that's a > > > > > > confusing > > > > > > > > > >>>>>> distribution mechanism as demonstrated by the > numerous > > > bug > > > > > > > reports > > > > > > > > > >> and > > > > > > > > > >>>>>> Stack Overflow posts regarding missing transitive > > > > > dependencies > > > > > > > for > > > > > > > > > >>>> various > > > > > > > > > >>>>>> features. I spent some time experimenting with Log4j > > > Boot > > > > a > > > > > > > little > > > > > > > > > >>> while > > > > > > > > > >>>>>> ago to help alleviate this problem, but this may be > > > > > > unnecessary > > > > > > > if > > > > > > > > > >> we > > > > > > > > > >>>> can > > > > > > > > > >>>>>> agree to modularize log4j-core itself. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> I have two different proposals, both of which can be > > > used > > > > at > > > > > > the > > > > > > > > > >> same > > > > > > > > > >>>>> time. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> 1. Split out everything from log4j-core that > requires > > > 3rd > > > > > > party > > > > > > > > > >>>>>> dependencies (except for AsyncLogger, though perhaps > > we > > > > > could > > > > > > > > > >> consider > > > > > > > > > >>>>>> shading and renaming those classes like some other > low > > > > level > > > > > > > > > >> libraries > > > > > > > > > >>>> do > > > > > > > > > >>>>>> with JCTools). Ideally, I'd like to see each module > > have > > > > > > > required > > > > > > > > > >>>>>> dependencies instead of optional ones, so that if, > for > > > > > > > instance, I > > > > > > > > > >>>> include > > > > > > > > > >>>>>> a "log4j-config-yaml" dependency, I know that Log4j > > will > > > > > > support > > > > > > > > > >> YAML > > > > > > > > > >>>>>> configuration without having to specify the > individual > > > > > Jackson > > > > > > > > > >>>>>> dependencies. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> 2. Split out from log4j-core a sort of log4j-spi > > module > > > > > which > > > > > > > > > >> defines > > > > > > > > > >>>>>> interfaces, abstract classes, and annotations for > > > plugins > > > > > that > > > > > > > > would > > > > > > > > > >>> be > > > > > > > > > >>>>>> promoted to the same level of backwards > compatibility > > > > > > guarantees > > > > > > > > as > > > > > > > > > >>>>>> log4j-api. This would aid in cementing what we > really > > > wish > > > > > to > > > > > > > > > >> maintain > > > > > > > > > >>>>>> compatibility with in the backend while allowing > other > > > > > modules > > > > > > > to > > > > > > > > > >> have > > > > > > > > > >>>>> less > > > > > > > > > >>>>>> strict guarantees. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> With proposal #1, I'd think that we could more > easily > > > > start > > > > > > > moving > > > > > > > > > >>>> modules > > > > > > > > > >>>>>> into separate repositories and release trains. > Without > > > #2, > > > > > > > though, > > > > > > > > > >>> this > > > > > > > > > >>>>>> makes version support more annoying to handle, but > > > that's > > > > > what > > > > > > > > we'll > > > > > > > > > >>>> face > > > > > > > > > >>>>>> regardless as we separate more repositories. If we > go > > > this > > > > > > > route, > > > > > > > > > >> then > > > > > > > > > >>>>>> there will be no need for a Log4j Boot subproject. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> What do you all think? > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> -- > > > > > > > > > >>>>>> Matt Sicker <[email protected]> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> -- > > > > > > > > > >>>>> [image: MagineTV] > > > > > > > > > >>>>> > > > > > > > > > >>>>> *Mikael Ståldal* > > > > > > > > > >>>>> Senior software developer > > > > > > > > > >>>>> > > > > > > > > > >>>>> *Magine TV* > > > > > > > > > >>>>> [email protected] > > > > > > > > > >>>>> Grev Turegatan 3 | 114 46 Stockholm, Sweden | > > > > > > www.magine.com > > > > > > > > > >>>>> > > > > > > > > > >>>>> Privileged and/or Confidential Information may be > > > contained > > > > > in > > > > > > > this > > > > > > > > > >>>>> message. If you are not the addressee indicated in > this > > > > > message > > > > > > > > > >>>>> (or responsible for delivery of the message to such a > > > > > person), > > > > > > > you > > > > > > > > > >> may > > > > > > > > > >>>> not > > > > > > > > > >>>>> copy or deliver this message to anyone. In such case, > > > > > > > > > >>>>> you should destroy this message and kindly notify the > > > > sender > > > > > by > > > > > > > > reply > > > > > > > > > >>>>> email. > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> -- > > > > > > > > > >>> Matt Sicker <[email protected]> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > [image: MagineTV] > > > > > > > > > > > > > > > > > > > > *Mikael Ståldal* > > > > > > > > > > Senior software developer > > > > > > > > > > > > > > > > > > > > *Magine TV* > > > > > > > > > > [email protected] > > > > > > > > > > Grev Turegatan 3 | 114 46 Stockholm, Sweden | > > > > www.magine.com > > > > > > > > > > > > > > > > > > > > Privileged and/or Confidential Information may be > contained > > > in > > > > > this > > > > > > > > > > message. If you are not the addressee indicated in this > > > message > > > > > > > > > > (or responsible for delivery of the message to such a > > > person), > > > > > you > > > > > > > may > > > > > > > > > not > > > > > > > > > > copy or deliver this message to anyone. In such case, > > > > > > > > > > you should destroy this message and kindly notify the > > sender > > > by > > > > > > reply > > > > > > > > > > email. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > [image: MagineTV] > > > > > > > > > > > > > > *Mikael Ståldal* > > > > > > > Senior software developer > > > > > > > > > > > > > > *Magine TV* > > > > > > > [email protected] > > > > > > > Grev Turegatan 3 | 114 46 Stockholm, Sweden | > www.magine.com > > > > > > > > > > > > > > Privileged and/or Confidential Information may be contained in > > this > > > > > > > message. If you are not the addressee indicated in this message > > > > > > > (or responsible for delivery of the message to such a person), > > you > > > > may > > > > > > not > > > > > > > copy or deliver this message to anyone. In such case, > > > > > > > you should destroy this message and kindly notify the sender by > > > reply > > > > > > > email. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Matt Sicker <[email protected]> > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > [image: MagineTV] > > > > > > > > *Mikael Ståldal* > > > > Senior software developer > > > > > > > > *Magine TV* > > > > [email protected] > > > > Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com > > > > > > > > Privileged and/or Confidential Information may be contained in this > > > > message. If you are not the addressee indicated in this message > > > > (or responsible for delivery of the message to such a person), you > may > > > not > > > > copy or deliver this message to anyone. In such case, > > > > you should destroy this message and kindly notify the sender by reply > > > > email. > > > > > > > > > > > > > > > -- > > Matt Sicker <[email protected]> > > > -- [image: MagineTV] *Mikael Ståldal* Senior software developer *Magine TV* [email protected] Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com Privileged and/or Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such a person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email.
