Le ven. 8 avr. 2022 à 19:28, Matt Sicker <boa...@gmail.com> a écrit :
> One iteration of the plugin system was based on Weld and OpenWebBeans, > but the CDI API is far too verbose for a use case where I didn't have > a need to integrate with a decade worth of legacy Java EE APIs (no > JNDI, no JPA, no EJB, no Resource(s) annotations, none of the > DI-alternatives leading up to CDI). > Hmm, not sure what you ref to but cdi standalone is quite light and does not depend on anything else than atinject. > For changing annotations at runtime, the idea I had here was to be > able to either register annotations in the API so that you can opt-in > to which annotations to use, or we could check the name of the > annotation class and use anything properly named (similar to how > Lombok treats nullability annotations). So in either case, I'd follow > an approach like that in order to support all of javax, jakarta, and > shaded versions of the same annotations. > So you take cdi path somehow, this is one of my point. Whatever plugins you need, you'll need IoC feature on one side and whatever system you have you want it integrated with your app IoC on another side so you dont want another mecanism IMHO. > Yes, the current DI system from Log4j supports field, constructor, and > method injection. I did not add interceptors and decorators as we > didn't have a need for them (we've found it easier to create plugins > for this purpose instead, though I can see these concepts being > potentially useful in other apps). > Yet, look a cli, crest thought it was not needed but it came up quickly as an important need, will be the same IMHO. > Most DI systems I've ever used or tried using are optimized for the > use case where the configurer is also the same person compiling and > packaging the application in the first place. Not CDI ;). This works fairly well > for custom applications, but it gets extremely complicated when you > try to expose some of this configurability to the end user of the > application (or library). I love to point at the Spinnaker project for > this as that's what I've been working on as a day job for about a year > now, and we have a few attempts at making this experience suck less, > but it ended up with a barely maintained additional microservice whose > job is to translate one input Spring config file into like 10+ config > files, but this only works with features that explicitly add > themselves to the configuration system (less common when working on > Spinnaker directly as you incrementally update your Spring config > files). While there's a fairly new plugin system available there, a > ton of existing functionality is still integrated directly in > Spinnaker which is an example of where you'd have far more code at > runtime on the classpath than you actually desire. Log4j is another > similar project with plenty of optional plugins baked in the core, but > that's being split up more successfully than Spinnaker has been at > doing similarly. > > From a CLI plugin perspective, my thoughts here would be to make it > simpler to make an extensible CLI tool like kubectl. Understood but still. Crest+cdi is not far. Perhaps something > like Quarkus would be useful here, or it would be useful as a DI > backend (I did mention Avaje Inject which has a similar code > generation feature to Quarkus but without the non-DI features). > Quarkus being an half-baked cdi a bit fatter i guess cdi is still an option...and to be honest spring and guice reach the same goal technically as of today, they are just more or less neat technically and more or less fat in terms of stack. So overall I can see the temptation but not sure what it brings technically. That said, once again, if you have a community to get started with this is not a blocker ;). > On Fri, Apr 8, 2022 at 12:06 PM Romain Manni-Bucau > <rmannibu...@gmail.com> wrote: > > > > Le ven. 8 avr. 2022 à 18:50, Matt Sicker <boa...@gmail.com> a écrit : > > > > > I suspect at this point that most of the remaining slowness in startup > > > on Log4j is related to code that _doesn't_ use plugins. There are some > > > strategies that configure on startup in log4j-api based on system > > > properties and service loaders which are provided for improved > > > steady-state performance (or for garbage-free logging) but add > > > complexity to startup. > > > > > > > The light IoC and scanner also are not really fast for a generic > embeddable > > plugin system. > > > > > > > > > > From a DI point of view, I followed the standard @Inject style API > > > that projects such as Guice already use, though I included my own > > > copies of the javax annotations since we weren't relying on additional > > > dependencies there. In a generic plugins framework, this could be > > > abstracted further to support any general DI framework. > > > > > > > So it means your solution can: > > > > 1. be modified at startup (ie I can change the annotation without any > > bytecode rewriting/agent/classfiletransformer)? > > 2. remove some plugin at startup writing a simple extension outside the > > plugin code (and not just remove it after it is added) > > 3. it supports @Inject everywhere (constructor, fields, setters) > > 4. it it supports javax and jakarta (don't blame me ;)) > > 5. it supports interceptors and decorators > > etc... > > > > This is why a bridge solution is rarely a good solution and that as soon > as > > you have an IoC you have a built-in plugin solution so never need another > > one. > > > > > > > > > > While the point of the plugin system is to be a much simpler approach > > > than something as complex as OSGi, it does seem like a valid concern > > > that it could become a sort of OSGi clone over time. With that in > > > mind, it becomes interesting to think what such a plugin system built > > > on top of OSGi would look like, though I haven't really explored that > > > possibility much (it was something I suggested long ago in the Jenkins > > > project as a way to improve and standardize their plugin system, > > > though there seems to be too much tech debt there to successfully do > > > so). > > > > > > > I was not even at OSGi, just plain CDI/Spring case when I said it > wouldn't > > be that helping ot write apps, even graalvm ones. > > > > > > > > > > A potential design goal that could avoid OSGi-ifying the library would > > > be to aim to be a superfast, lightweight plugin system that can work > > > well with libraries, CLI apps (particularly interesting when using > > > GraalVM for AOT compilation or when using jlink/jpackage/etc. to > > > create a standalone application), and server software. If the DI > > > aspect is sufficiently pluggable, then that would allow for using > > > different DI backends appropriate to the target environment (e.g., > > > using Spring in a long-running server app, or Avaje for a CLI app with > > > its code generation feature, or even the built-in DI impl for a > > > minimal-dependency library). > > > > > > > CDI fits that description AFAIK (I'm using openwebbeans and tomitribe > crest > > quite often for CLI implementations but guess quarkus can be an option > too > > if you don't care to be fatter) > > > > > > > > > > On Fri, Apr 8, 2022 at 2:38 AM Romain Manni-Bucau < > rmannibu...@gmail.com> > > > wrote: > > > > > > > > Guess that theorically there is room for a new generic plugin system > but > > > I > > > > would mention a few points: > > > > > > > > 1. log4j one is not yet a good *generic* one for multiple reasons > but the > > > > biggest blocker for me is that it is slow (slower than a plain IoC > as of > > > > today, even dropping some legacy parts) - there is a ticket about it > but > > > we > > > > are still slower than a plain CDI container even if it is way better > than > > > > 2.x on 3.x branch > > > > 2. Most naturally you want your plugin system to integrate your IoC > > > > (spring, CDI, guice, ...) - to get injections, same bean pattern than > > > your > > > > IoC/app, extensions, .... If we go with log4j system then we must do > > > > bridges which are quite useless in practise so it is an abstraction > you > > > > will fight against instead of leveraging. > > > > 3. If there is a community around this project then it fits apache > so it > > > > can go but I think it belongs better to incubator and not commons > > > > > > > > Hope it makes sense > > > > > > > > Romain Manni-Bucau > > > > @rmannibucau <https://twitter.com/rmannibucau> | Blog > > > > <https://rmannibucau.metawerx.net/> | Old Blog > > > > <http://rmannibucau.wordpress.com> | Github < > > > https://github.com/rmannibucau> | > > > > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book > > > > < > > > > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > > > > > > > > > > > > > > Le ven. 8 avr. 2022 à 09:23, Peter Verhas <pe...@verhas.com> a > écrit : > > > > > > > > > Thanks Ralph for the detailed explanation. I appreciate it and now > I > > > see > > > > > the points. > > > > > > > > > > I have never experienced an application where class loading time > was an > > > > > issue, and I understand that it can really be. > > > > > I have never experienced a setup where there were a lot of "plugin" > > > classes > > > > > on the classpath or on the modulepath that were never loaded by the > > > > > application. > > > > > > > > > > We still are not on the same page about module systems. I am > happily > > > using > > > > > it in all of my new projects and my experience is mixed. I accept > the > > > > > struggle that it takes to make it properly, like opening the > packaged > > > via > > > > > command line options for testing purposes. On the other hand the > > > structure > > > > > and the encapsulation is one step better than without JPMS. > > > > > > > > > > >But since you don’t want to look at he Log4j plugin system > > > > > > > > > > I did have a look at it to some level, as I also mentioned it in > my > > > > > previous mail. I am not an expert in that field as you are and I > never > > > > > will, and that is what I do not want to be. > > > > > > > > > > I accept that there is room for a commons-plugin project. The > project > > > has > > > > > to address these questions to be on the right track (not to me, to > the > > > > > project itself): > > > > > > > > > > - How will it be a "plugin" project and not another dependency > > > injection > > > > > framework? > > > > > - What will distinguish it from module systems, like OSGi and what > will > > > > > stop it from becoming another OSGi by the years as new features get > > > added > > > > > to the library. > > > > > - What applications using plugins are the examples for different > > > solutions? > > > > > (Log4j is a good example to show that there is a need, you also > > > explained > > > > > patiently why it is not a simple ServiceLoader, but it is only one > way > > > to > > > > > solve it. Other applications may approach the issue differently. > Maven, > > > > > Attlassian products, other build tools, JUnit 5 and so on.) > > > > > - Based on the gathered knowledge on the previous point, what is > the > > > high > > > > > level architecture of a plugin system the library will support and > what > > > > > services will it provide? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 7, 2022 at 7:57 PM Ralph Goers < > ralph.go...@dslextreme.com > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Apr 7, 2022, at 2:52 AM, Peter Verhas <pe...@verhas.com> > wrote: > > > > > > > > > > > > > > > > > > > > >> I would suggest that before responding to this email that > > > > > > >> you go look at how log4j-plugins is implemented in the master > > > branch. > > > > > > >> > > > > > > > > > > > > > > Sure, if you propose to create a commons-plugin library that > is an > > > > > > extract > > > > > > > of the plugin handling of log4j so that log4j next releases do > not > > > need > > > > > > to > > > > > > > keep the functionality inside log4j but rather as a dependency > on > > > this > > > > > > new > > > > > > > library. On the other hand, if the goal is to create something > > > that is > > > > > > > useful generally then we should look at other solutions as > well. > > > > > > > > > > > > It remains to be seen whether Log4j would replace log4j-plugins > with > > > > > > Commons-plugins. The point of looking at Log4j was to understand > the > > > > > > problem it is solving and how it solves it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> From this it is clear that you completely misunderstand what > the > > > > > plugin > > > > > > >> system is doing. > > > > > > >> > > > > > > > > > > > > > > That may really be very much true. > > > > > > > > > > > > > > > > > > > > >> First, the plugin system DOES use ServiceLoader. In fact, > Log4j > > > uses > > > > > > >> ServiceLoader > > > > > > >> in at least 7 different places. For one, it uses > ServiceLoader to > > > > > locate > > > > > > >> the Log4j > > > > > > >> implementation similar to how SLF4J does (although Log4j > > > implemented > > > > > it > > > > > > >> prior to SLF4J). > > > > > > >> > > > > > > >> Consider that you have 150 or so plugins, of which maybe only > 10 > > > will > > > > > > get > > > > > > >> used. But > > > > > > >> you don’t know beforehand which 10. ServiceLoader would have > to > > > load > > > > > all > > > > > > >> 150 classes. > > > > > > > > > > > > > > > > > > > > > Not. It does not need to create instances of all of them. > This is > > > > > what I > > > > > > > explained, and this is the reason why I explained it. JPMS or > not, > > > the > > > > > > new > > > > > > > ServiceLoader used by Java 9 and later will call the static > factory > > > > > > method > > > > > > > of the class if that exists. Then the plugin can decide if it > wants > > > > > > itself > > > > > > > to be instantiated or not. Somewhere there should be some logic > > > that > > > > > > > filters 10 from the 150. If that is in a central place > implementing > > > > > some > > > > > > > industry-best-practice configuration based algorithm, or > > > outsources the > > > > > > > decision to the plugins themselves should not make a big > > > difference in > > > > > > > performance. > > > > > > > > > > > > > > On the second though, instantiating 150 lightweight classes > must > > > not > > > > > be > > > > > > a > > > > > > > big burden. The ServiceProvider JavaDoc documentation suggest > that > > > > > > services > > > > > > > should be implemented as proxies or factories instantiating > their > > > real > > > > > > > working class lazily in case the instantiation is costly. > > > > > > > > > > > > Class loading in general is slow. Perhaps not to you but we have > > > > > customers > > > > > > who > > > > > > complain about all the things we do during startup and locating > and > > > > > > instantiating > > > > > > plugins is one of them. When we first switched to use the > > > ServiceLoader > > > > > we > > > > > > attempted to load the plugin classes simply by having the plugin > > > > > > definition include > > > > > > the Class.Performance was very bad. We now only include the class > > > name > > > > > and > > > > > > only access the Class when absolutely necessary. > > > > > > > > > > > > Again, if you think about what the Plugin Manager is doing, it is > > > using > > > > > > ServiceLoader > > > > > > to find all Plugin Service instances. Each of these contains the > > > > > > definitions of all the > > > > > > plugins. The Plugin Manager then uses these to create the actual > > > plugins. > > > > > > So in > > > > > > essence, the services are used as part of the factory. Note that > > > > > decisions > > > > > > about > > > > > > whether to load a plugin or not do not have to be made when the > > > plugin > > > > > > service > > > > > > s accessed. That is deferred until the Plugin actually needs to > be > > > wired. > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is indeed very slow. Yes we have benchmarked it. But the > > > Plugin > > > > > > >> system IS a > > > > > > >> Java Service. So instead, you need to only load 1 class for > every > > > jar > > > > > > that > > > > > > >> contains plugins. > > > > > > >> The result is that you only need to load 4 or 5 classes. This > is > > > > > > >> considerably faster. > > > > > > >> > > > > > > > > > > > > > > Yes, this is the approach where you insert another layer above > the > > > > > > service > > > > > > > loader that does part of the job that the service loader can > also > > > do > > > > > in a > > > > > > > different way. Essentially you let the plugin libraries, each > > > > > containing > > > > > > > several plugin classes to implement their own service loaders. > > > > > > > > > > > > Sort of. With Log4j’s plugin system Plugins are not aware of > > > > > ServiceLoader > > > > > > at all. Other than that, it behaves the way you are describing. > > > > > > > > > > > > > > > > > > > > > > > > > > >> In addition, services loaded by ServiceLoader are singletons. > > > > > > > > > > > > > > > > > > > > > No, they are not. Every ServiceLoader instance will create new > > > > > instances, > > > > > > > and even a single ServiceLoader can be asked calling the > `reload()` > > > > > > method > > > > > > > to evict the instances from its cache and create new instances. > > > > > > > > > > > > Umm. Even calling reload() doesn’t change it from being a > singleton. > > > You > > > > > > can > > > > > > only have a single instance at a time. > > > > > > > > > > > > Yes, I suppose if you use multiple ServiceLoader instances then > you > > > can I > > > > > > nstantiate multiple instances of the service. However, as you > noted > > > that > > > > > > wouldn’t > > > > > > be the best practice since the ServiceLoader really should be > > > > > > instantiating a > > > > > > factory that create as many instances as you want. > > > > > > > > > > > > > > > > > > > > Perhaps developing a commons-plugin system is a good point to > > > require > > > > > > > application developers and plugin developers to use the module > > > system. > > > > > > > > > > > > Please no. I would never encourage anyone to use it. It has > caused > > > > > nothing > > > > > > but pain. Yes, that is opinion, but opinion based on experience. > > > > > > > > > > > > > > > > > > > > > > > > > > >> Again, I would suggest you look at the log4j-plugins > > > implementation > > > > > > before > > > > > > >> making sweeping statements such as this. > > > > > > >> > > > > > > > > > > > > > > I am not an expert about log4j and the plugin system of > it,which > > > you > > > > > are. > > > > > > > From this distance it seems that log4j uses factories for the > > > plugins > > > > > as > > > > > > > services or builders in case there is a need for some > > > configuration. In > > > > > > > case of the builders the configuration parameters are injected > > > that way > > > > > > > this plugin system implements a simple DI framework like > Plexus, > > > Pico, > > > > > > > Guava, Spring instead of using one. With the factory approach > it > > > does > > > > > the > > > > > > > same as the JDK ServiceLoader calling the `provider()` method. > > > > > > > > > > > > > > Again, if you find that answering the questions is difficult > there > > > may > > > > > be > > > > > > > several reasons. One thing is for sure: I am not trolling you. > I am > > > > > > asking > > > > > > > these questions because I honestly believe that any new library > > > should > > > > > > > address the things that are not addressed by other systems and > > > > > libraries, > > > > > > > and should provide a better way to do things, otherwise they > will > > > not > > > > > be > > > > > > > widely used. > > > > > > > > > > > > > > > > > > Frankly, I don’t think there is more to discuss. You asked why > not > > > just > > > > > > use > > > > > > ServiceLoader. But you have pretty much conceded that vanilla > > > > > > ServiceLoader > > > > > > by itself is not a good plugin system. You have agreed that using > > > > > > ServiceLoader > > > > > > to load factories is pretty much necessary. But since you don’t > want > > > to > > > > > > look at > > > > > > he Log4j plugin system to see how it makes creating plugins easy > it > > > is > > > > > > hard to > > > > > > discuss how it achieves the goals beyond just using > ServiceLoader to > > > load > > > > > > factories. > > > > > > > > > > > > Ralph > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Peter Verhas > > > > > pe...@verhas.com > > > > > t: +41791542095 > > > > > skype: verhas > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
