Le ven. 8 avr. 2022 à 18:50, Matt Sicker <boa...@gmail.com> a écrit :
> I suspect at this point that most of the remaining slowness in startup > on Log4j is related to code that _doesn't_ use plugins. There are some > strategies that configure on startup in log4j-api based on system > properties and service loaders which are provided for improved > steady-state performance (or for garbage-free logging) but add > complexity to startup. > The light IoC and scanner also are not really fast for a generic embeddable plugin system. > > From a DI point of view, I followed the standard @Inject style API > that projects such as Guice already use, though I included my own > copies of the javax annotations since we weren't relying on additional > dependencies there. In a generic plugins framework, this could be > abstracted further to support any general DI framework. > So it means your solution can: 1. be modified at startup (ie I can change the annotation without any bytecode rewriting/agent/classfiletransformer)? 2. remove some plugin at startup writing a simple extension outside the plugin code (and not just remove it after it is added) 3. it supports @Inject everywhere (constructor, fields, setters) 4. it it supports javax and jakarta (don't blame me ;)) 5. it supports interceptors and decorators etc... This is why a bridge solution is rarely a good solution and that as soon as you have an IoC you have a built-in plugin solution so never need another one. > > While the point of the plugin system is to be a much simpler approach > than something as complex as OSGi, it does seem like a valid concern > that it could become a sort of OSGi clone over time. With that in > mind, it becomes interesting to think what such a plugin system built > on top of OSGi would look like, though I haven't really explored that > possibility much (it was something I suggested long ago in the Jenkins > project as a way to improve and standardize their plugin system, > though there seems to be too much tech debt there to successfully do > so). > I was not even at OSGi, just plain CDI/Spring case when I said it wouldn't be that helping ot write apps, even graalvm ones. > > A potential design goal that could avoid OSGi-ifying the library would > be to aim to be a superfast, lightweight plugin system that can work > well with libraries, CLI apps (particularly interesting when using > GraalVM for AOT compilation or when using jlink/jpackage/etc. to > create a standalone application), and server software. If the DI > aspect is sufficiently pluggable, then that would allow for using > different DI backends appropriate to the target environment (e.g., > using Spring in a long-running server app, or Avaje for a CLI app with > its code generation feature, or even the built-in DI impl for a > minimal-dependency library). > CDI fits that description AFAIK (I'm using openwebbeans and tomitribe crest quite often for CLI implementations but guess quarkus can be an option too if you don't care to be fatter) > > On Fri, Apr 8, 2022 at 2:38 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > > > > Guess that theorically there is room for a new generic plugin system but > I > > would mention a few points: > > > > 1. log4j one is not yet a good *generic* one for multiple reasons but the > > biggest blocker for me is that it is slow (slower than a plain IoC as of > > today, even dropping some legacy parts) - there is a ticket about it but > we > > are still slower than a plain CDI container even if it is way better than > > 2.x on 3.x branch > > 2. Most naturally you want your plugin system to integrate your IoC > > (spring, CDI, guice, ...) - to get injections, same bean pattern than > your > > IoC/app, extensions, .... If we go with log4j system then we must do > > bridges which are quite useless in practise so it is an abstraction you > > will fight against instead of leveraging. > > 3. If there is a community around this project then it fits apache so it > > can go but I think it belongs better to incubator and not commons > > > > Hope it makes sense > > > > Romain Manni-Bucau > > @rmannibucau <https://twitter.com/rmannibucau> | Blog > > <https://rmannibucau.metawerx.net/> | Old Blog > > <http://rmannibucau.wordpress.com> | Github < > https://github.com/rmannibucau> | > > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book > > < > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > > > > > > Le ven. 8 avr. 2022 à 09:23, Peter Verhas <pe...@verhas.com> a écrit : > > > > > Thanks Ralph for the detailed explanation. I appreciate it and now I > see > > > the points. > > > > > > I have never experienced an application where class loading time was an > > > issue, and I understand that it can really be. > > > I have never experienced a setup where there were a lot of "plugin" > classes > > > on the classpath or on the modulepath that were never loaded by the > > > application. > > > > > > We still are not on the same page about module systems. I am happily > using > > > it in all of my new projects and my experience is mixed. I accept the > > > struggle that it takes to make it properly, like opening the packaged > via > > > command line options for testing purposes. On the other hand the > structure > > > and the encapsulation is one step better than without JPMS. > > > > > > >But since you don’t want to look at he Log4j plugin system > > > > > > I did have a look at it to some level, as I also mentioned it in my > > > previous mail. I am not an expert in that field as you are and I never > > > will, and that is what I do not want to be. > > > > > > I accept that there is room for a commons-plugin project. The project > has > > > to address these questions to be on the right track (not to me, to the > > > project itself): > > > > > > - How will it be a "plugin" project and not another dependency > injection > > > framework? > > > - What will distinguish it from module systems, like OSGi and what will > > > stop it from becoming another OSGi by the years as new features get > added > > > to the library. > > > - What applications using plugins are the examples for different > solutions? > > > (Log4j is a good example to show that there is a need, you also > explained > > > patiently why it is not a simple ServiceLoader, but it is only one way > to > > > solve it. Other applications may approach the issue differently. Maven, > > > Attlassian products, other build tools, JUnit 5 and so on.) > > > - Based on the gathered knowledge on the previous point, what is the > high > > > level architecture of a plugin system the library will support and what > > > services will it provide? > > > > > > > > > > > > > > > > > > On Thu, Apr 7, 2022 at 7:57 PM Ralph Goers <ralph.go...@dslextreme.com > > > > > wrote: > > > > > > > > > > > > > > > > On Apr 7, 2022, at 2:52 AM, Peter Verhas <pe...@verhas.com> wrote: > > > > > > > > > > > > > > >> I would suggest that before responding to this email that > > > > >> you go look at how log4j-plugins is implemented in the master > branch. > > > > >> > > > > > > > > > > Sure, if you propose to create a commons-plugin library that is an > > > > extract > > > > > of the plugin handling of log4j so that log4j next releases do not > need > > > > to > > > > > keep the functionality inside log4j but rather as a dependency on > this > > > > new > > > > > library. On the other hand, if the goal is to create something > that is > > > > > useful generally then we should look at other solutions as well. > > > > > > > > It remains to be seen whether Log4j would replace log4j-plugins with > > > > Commons-plugins. The point of looking at Log4j was to understand the > > > > problem it is solving and how it solves it. > > > > > > > > > > > > > > > > > > > > > > >> From this it is clear that you completely misunderstand what the > > > plugin > > > > >> system is doing. > > > > >> > > > > > > > > > > That may really be very much true. > > > > > > > > > > > > > > >> First, the plugin system DOES use ServiceLoader. In fact, Log4j > uses > > > > >> ServiceLoader > > > > >> in at least 7 different places. For one, it uses ServiceLoader to > > > locate > > > > >> the Log4j > > > > >> implementation similar to how SLF4J does (although Log4j > implemented > > > it > > > > >> prior to SLF4J). > > > > >> > > > > >> Consider that you have 150 or so plugins, of which maybe only 10 > will > > > > get > > > > >> used. But > > > > >> you don’t know beforehand which 10. ServiceLoader would have to > load > > > all > > > > >> 150 classes. > > > > > > > > > > > > > > > Not. It does not need to create instances of all of them. This is > > > what I > > > > > explained, and this is the reason why I explained it. JPMS or not, > the > > > > new > > > > > ServiceLoader used by Java 9 and later will call the static factory > > > > method > > > > > of the class if that exists. Then the plugin can decide if it wants > > > > itself > > > > > to be instantiated or not. Somewhere there should be some logic > that > > > > > filters 10 from the 150. If that is in a central place implementing > > > some > > > > > industry-best-practice configuration based algorithm, or > outsources the > > > > > decision to the plugins themselves should not make a big > difference in > > > > > performance. > > > > > > > > > > On the second though, instantiating 150 lightweight classes must > not > > > be > > > > a > > > > > big burden. The ServiceProvider JavaDoc documentation suggest that > > > > services > > > > > should be implemented as proxies or factories instantiating their > real > > > > > working class lazily in case the instantiation is costly. > > > > > > > > Class loading in general is slow. Perhaps not to you but we have > > > customers > > > > who > > > > complain about all the things we do during startup and locating and > > > > instantiating > > > > plugins is one of them. When we first switched to use the > ServiceLoader > > > we > > > > attempted to load the plugin classes simply by having the plugin > > > > definition include > > > > the Class.Performance was very bad. We now only include the class > name > > > and > > > > only access the Class when absolutely necessary. > > > > > > > > Again, if you think about what the Plugin Manager is doing, it is > using > > > > ServiceLoader > > > > to find all Plugin Service instances. Each of these contains the > > > > definitions of all the > > > > plugins. The Plugin Manager then uses these to create the actual > plugins. > > > > So in > > > > essence, the services are used as part of the factory. Note that > > > decisions > > > > about > > > > whether to load a plugin or not do not have to be made when the > plugin > > > > service > > > > s accessed. That is deferred until the Plugin actually needs to be > wired. > > > > > > > > > > > > > > > > > > >> This is indeed very slow. Yes we have benchmarked it. But the > Plugin > > > > >> system IS a > > > > >> Java Service. So instead, you need to only load 1 class for every > jar > > > > that > > > > >> contains plugins. > > > > >> The result is that you only need to load 4 or 5 classes. This is > > > > >> considerably faster. > > > > >> > > > > > > > > > > Yes, this is the approach where you insert another layer above the > > > > service > > > > > loader that does part of the job that the service loader can also > do > > > in a > > > > > different way. Essentially you let the plugin libraries, each > > > containing > > > > > several plugin classes to implement their own service loaders. > > > > > > > > Sort of. With Log4j’s plugin system Plugins are not aware of > > > ServiceLoader > > > > at all. Other than that, it behaves the way you are describing. > > > > > > > > > > > > > > > > > > >> In addition, services loaded by ServiceLoader are singletons. > > > > > > > > > > > > > > > No, they are not. Every ServiceLoader instance will create new > > > instances, > > > > > and even a single ServiceLoader can be asked calling the `reload()` > > > > method > > > > > to evict the instances from its cache and create new instances. > > > > > > > > Umm. Even calling reload() doesn’t change it from being a singleton. > You > > > > can > > > > only have a single instance at a time. > > > > > > > > Yes, I suppose if you use multiple ServiceLoader instances then you > can I > > > > nstantiate multiple instances of the service. However, as you noted > that > > > > wouldn’t > > > > be the best practice since the ServiceLoader really should be > > > > instantiating a > > > > factory that create as many instances as you want. > > > > > > > > > > > > > > Perhaps developing a commons-plugin system is a good point to > require > > > > > application developers and plugin developers to use the module > system. > > > > > > > > Please no. I would never encourage anyone to use it. It has caused > > > nothing > > > > but pain. Yes, that is opinion, but opinion based on experience. > > > > > > > > > > > > > > > > > > >> Again, I would suggest you look at the log4j-plugins > implementation > > > > before > > > > >> making sweeping statements such as this. > > > > >> > > > > > > > > > > I am not an expert about log4j and the plugin system of it,which > you > > > are. > > > > > From this distance it seems that log4j uses factories for the > plugins > > > as > > > > > services or builders in case there is a need for some > configuration. In > > > > > case of the builders the configuration parameters are injected > that way > > > > > this plugin system implements a simple DI framework like Plexus, > Pico, > > > > > Guava, Spring instead of using one. With the factory approach it > does > > > the > > > > > same as the JDK ServiceLoader calling the `provider()` method. > > > > > > > > > > Again, if you find that answering the questions is difficult there > may > > > be > > > > > several reasons. One thing is for sure: I am not trolling you. I am > > > > asking > > > > > these questions because I honestly believe that any new library > should > > > > > address the things that are not addressed by other systems and > > > libraries, > > > > > and should provide a better way to do things, otherwise they will > not > > > be > > > > > widely used. > > > > > > > > > > > > Frankly, I don’t think there is more to discuss. You asked why not > just > > > > use > > > > ServiceLoader. But you have pretty much conceded that vanilla > > > > ServiceLoader > > > > by itself is not a good plugin system. You have agreed that using > > > > ServiceLoader > > > > to load factories is pretty much necessary. But since you don’t want > to > > > > look at > > > > he Log4j plugin system to see how it makes creating plugins easy it > is > > > > hard to > > > > discuss how it achieves the goals beyond just using ServiceLoader to > load > > > > factories. > > > > > > > > Ralph > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > > > > > > > > > -- > > > Peter Verhas > > > pe...@verhas.com > > > t: +41791542095 > > > skype: verhas > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
