Guess that theorically there is room for a new generic plugin system but I would mention a few points:
1. log4j one is not yet a good *generic* one for multiple reasons but the biggest blocker for me is that it is slow (slower than a plain IoC as of today, even dropping some legacy parts) - there is a ticket about it but we are still slower than a plain CDI container even if it is way better than 2.x on 3.x branch 2. Most naturally you want your plugin system to integrate your IoC (spring, CDI, guice, ...) - to get injections, same bean pattern than your IoC/app, extensions, .... If we go with log4j system then we must do bridges which are quite useless in practise so it is an abstraction you will fight against instead of leveraging. 3. If there is a community around this project then it fits apache so it can go but I think it belongs better to incubator and not commons Hope it makes sense Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> Le ven. 8 avr. 2022 à 09:23, Peter Verhas <pe...@verhas.com> a écrit : > Thanks Ralph for the detailed explanation. I appreciate it and now I see > the points. > > I have never experienced an application where class loading time was an > issue, and I understand that it can really be. > I have never experienced a setup where there were a lot of "plugin" classes > on the classpath or on the modulepath that were never loaded by the > application. > > We still are not on the same page about module systems. I am happily using > it in all of my new projects and my experience is mixed. I accept the > struggle that it takes to make it properly, like opening the packaged via > command line options for testing purposes. On the other hand the structure > and the encapsulation is one step better than without JPMS. > > >But since you don’t want to look at he Log4j plugin system > > I did have a look at it to some level, as I also mentioned it in my > previous mail. I am not an expert in that field as you are and I never > will, and that is what I do not want to be. > > I accept that there is room for a commons-plugin project. The project has > to address these questions to be on the right track (not to me, to the > project itself): > > - How will it be a "plugin" project and not another dependency injection > framework? > - What will distinguish it from module systems, like OSGi and what will > stop it from becoming another OSGi by the years as new features get added > to the library. > - What applications using plugins are the examples for different solutions? > (Log4j is a good example to show that there is a need, you also explained > patiently why it is not a simple ServiceLoader, but it is only one way to > solve it. Other applications may approach the issue differently. Maven, > Attlassian products, other build tools, JUnit 5 and so on.) > - Based on the gathered knowledge on the previous point, what is the high > level architecture of a plugin system the library will support and what > services will it provide? > > > > > > On Thu, Apr 7, 2022 at 7:57 PM Ralph Goers <ralph.go...@dslextreme.com> > wrote: > > > > > > > > On Apr 7, 2022, at 2:52 AM, Peter Verhas <pe...@verhas.com> wrote: > > > > > > > > >> I would suggest that before responding to this email that > > >> you go look at how log4j-plugins is implemented in the master branch. > > >> > > > > > > Sure, if you propose to create a commons-plugin library that is an > > extract > > > of the plugin handling of log4j so that log4j next releases do not need > > to > > > keep the functionality inside log4j but rather as a dependency on this > > new > > > library. On the other hand, if the goal is to create something that is > > > useful generally then we should look at other solutions as well. > > > > It remains to be seen whether Log4j would replace log4j-plugins with > > Commons-plugins. The point of looking at Log4j was to understand the > > problem it is solving and how it solves it. > > > > > > > > > > > > >> From this it is clear that you completely misunderstand what the > plugin > > >> system is doing. > > >> > > > > > > That may really be very much true. > > > > > > > > >> First, the plugin system DOES use ServiceLoader. In fact, Log4j uses > > >> ServiceLoader > > >> in at least 7 different places. For one, it uses ServiceLoader to > locate > > >> the Log4j > > >> implementation similar to how SLF4J does (although Log4j implemented > it > > >> prior to SLF4J). > > >> > > >> Consider that you have 150 or so plugins, of which maybe only 10 will > > get > > >> used. But > > >> you don’t know beforehand which 10. ServiceLoader would have to load > all > > >> 150 classes. > > > > > > > > > Not. It does not need to create instances of all of them. This is > what I > > > explained, and this is the reason why I explained it. JPMS or not, the > > new > > > ServiceLoader used by Java 9 and later will call the static factory > > method > > > of the class if that exists. Then the plugin can decide if it wants > > itself > > > to be instantiated or not. Somewhere there should be some logic that > > > filters 10 from the 150. If that is in a central place implementing > some > > > industry-best-practice configuration based algorithm, or outsources the > > > decision to the plugins themselves should not make a big difference in > > > performance. > > > > > > On the second though, instantiating 150 lightweight classes must not > be > > a > > > big burden. The ServiceProvider JavaDoc documentation suggest that > > services > > > should be implemented as proxies or factories instantiating their real > > > working class lazily in case the instantiation is costly. > > > > Class loading in general is slow. Perhaps not to you but we have > customers > > who > > complain about all the things we do during startup and locating and > > instantiating > > plugins is one of them. When we first switched to use the ServiceLoader > we > > attempted to load the plugin classes simply by having the plugin > > definition include > > the Class.Performance was very bad. We now only include the class name > and > > only access the Class when absolutely necessary. > > > > Again, if you think about what the Plugin Manager is doing, it is using > > ServiceLoader > > to find all Plugin Service instances. Each of these contains the > > definitions of all the > > plugins. The Plugin Manager then uses these to create the actual plugins. > > So in > > essence, the services are used as part of the factory. Note that > decisions > > about > > whether to load a plugin or not do not have to be made when the plugin > > service > > s accessed. That is deferred until the Plugin actually needs to be wired. > > > > > > > > > > >> This is indeed very slow. Yes we have benchmarked it. But the Plugin > > >> system IS a > > >> Java Service. So instead, you need to only load 1 class for every jar > > that > > >> contains plugins. > > >> The result is that you only need to load 4 or 5 classes. This is > > >> considerably faster. > > >> > > > > > > Yes, this is the approach where you insert another layer above the > > service > > > loader that does part of the job that the service loader can also do > in a > > > different way. Essentially you let the plugin libraries, each > containing > > > several plugin classes to implement their own service loaders. > > > > Sort of. With Log4j’s plugin system Plugins are not aware of > ServiceLoader > > at all. Other than that, it behaves the way you are describing. > > > > > > > > > > >> In addition, services loaded by ServiceLoader are singletons. > > > > > > > > > No, they are not. Every ServiceLoader instance will create new > instances, > > > and even a single ServiceLoader can be asked calling the `reload()` > > method > > > to evict the instances from its cache and create new instances. > > > > Umm. Even calling reload() doesn’t change it from being a singleton. You > > can > > only have a single instance at a time. > > > > Yes, I suppose if you use multiple ServiceLoader instances then you can I > > nstantiate multiple instances of the service. However, as you noted that > > wouldn’t > > be the best practice since the ServiceLoader really should be > > instantiating a > > factory that create as many instances as you want. > > > > > > > > Perhaps developing a commons-plugin system is a good point to require > > > application developers and plugin developers to use the module system. > > > > Please no. I would never encourage anyone to use it. It has caused > nothing > > but pain. Yes, that is opinion, but opinion based on experience. > > > > > > > > > > >> Again, I would suggest you look at the log4j-plugins implementation > > before > > >> making sweeping statements such as this. > > >> > > > > > > I am not an expert about log4j and the plugin system of it,which you > are. > > > From this distance it seems that log4j uses factories for the plugins > as > > > services or builders in case there is a need for some configuration. In > > > case of the builders the configuration parameters are injected that way > > > this plugin system implements a simple DI framework like Plexus, Pico, > > > Guava, Spring instead of using one. With the factory approach it does > the > > > same as the JDK ServiceLoader calling the `provider()` method. > > > > > > Again, if you find that answering the questions is difficult there may > be > > > several reasons. One thing is for sure: I am not trolling you. I am > > asking > > > these questions because I honestly believe that any new library should > > > address the things that are not addressed by other systems and > libraries, > > > and should provide a better way to do things, otherwise they will not > be > > > widely used. > > > > > > Frankly, I don’t think there is more to discuss. You asked why not just > > use > > ServiceLoader. But you have pretty much conceded that vanilla > > ServiceLoader > > by itself is not a good plugin system. You have agreed that using > > ServiceLoader > > to load factories is pretty much necessary. But since you don’t want to > > look at > > he Log4j plugin system to see how it makes creating plugins easy it is > > hard to > > discuss how it achieves the goals beyond just using ServiceLoader to load > > factories. > > > > Ralph > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > -- > Peter Verhas > pe...@verhas.com > t: +41791542095 > skype: verhas >
