Thanks Ralph for the detailed explanation. I appreciate it and now I see the points.
I have never experienced an application where class loading time was an issue, and I understand that it can really be. I have never experienced a setup where there were a lot of "plugin" classes on the classpath or on the modulepath that were never loaded by the application. We still are not on the same page about module systems. I am happily using it in all of my new projects and my experience is mixed. I accept the struggle that it takes to make it properly, like opening the packaged via command line options for testing purposes. On the other hand the structure and the encapsulation is one step better than without JPMS. >But since you don’t want to look at he Log4j plugin system I did have a look at it to some level, as I also mentioned it in my previous mail. I am not an expert in that field as you are and I never will, and that is what I do not want to be. I accept that there is room for a commons-plugin project. The project has to address these questions to be on the right track (not to me, to the project itself): - How will it be a "plugin" project and not another dependency injection framework? - What will distinguish it from module systems, like OSGi and what will stop it from becoming another OSGi by the years as new features get added to the library. - What applications using plugins are the examples for different solutions? (Log4j is a good example to show that there is a need, you also explained patiently why it is not a simple ServiceLoader, but it is only one way to solve it. Other applications may approach the issue differently. Maven, Attlassian products, other build tools, JUnit 5 and so on.) - Based on the gathered knowledge on the previous point, what is the high level architecture of a plugin system the library will support and what services will it provide? On Thu, Apr 7, 2022 at 7:57 PM Ralph Goers <ralph.go...@dslextreme.com> wrote: > > > > On Apr 7, 2022, at 2:52 AM, Peter Verhas <pe...@verhas.com> wrote: > > > > > >> I would suggest that before responding to this email that > >> you go look at how log4j-plugins is implemented in the master branch. > >> > > > > Sure, if you propose to create a commons-plugin library that is an > extract > > of the plugin handling of log4j so that log4j next releases do not need > to > > keep the functionality inside log4j but rather as a dependency on this > new > > library. On the other hand, if the goal is to create something that is > > useful generally then we should look at other solutions as well. > > It remains to be seen whether Log4j would replace log4j-plugins with > Commons-plugins. The point of looking at Log4j was to understand the > problem it is solving and how it solves it. > > > > > > > >> From this it is clear that you completely misunderstand what the plugin > >> system is doing. > >> > > > > That may really be very much true. > > > > > >> First, the plugin system DOES use ServiceLoader. In fact, Log4j uses > >> ServiceLoader > >> in at least 7 different places. For one, it uses ServiceLoader to locate > >> the Log4j > >> implementation similar to how SLF4J does (although Log4j implemented it > >> prior to SLF4J). > >> > >> Consider that you have 150 or so plugins, of which maybe only 10 will > get > >> used. But > >> you don’t know beforehand which 10. ServiceLoader would have to load all > >> 150 classes. > > > > > > Not. It does not need to create instances of all of them. This is what I > > explained, and this is the reason why I explained it. JPMS or not, the > new > > ServiceLoader used by Java 9 and later will call the static factory > method > > of the class if that exists. Then the plugin can decide if it wants > itself > > to be instantiated or not. Somewhere there should be some logic that > > filters 10 from the 150. If that is in a central place implementing some > > industry-best-practice configuration based algorithm, or outsources the > > decision to the plugins themselves should not make a big difference in > > performance. > > > > On the second though, instantiating 150 lightweight classes must not be > a > > big burden. The ServiceProvider JavaDoc documentation suggest that > services > > should be implemented as proxies or factories instantiating their real > > working class lazily in case the instantiation is costly. > > Class loading in general is slow. Perhaps not to you but we have customers > who > complain about all the things we do during startup and locating and > instantiating > plugins is one of them. When we first switched to use the ServiceLoader we > attempted to load the plugin classes simply by having the plugin > definition include > the Class.Performance was very bad. We now only include the class name and > only access the Class when absolutely necessary. > > Again, if you think about what the Plugin Manager is doing, it is using > ServiceLoader > to find all Plugin Service instances. Each of these contains the > definitions of all the > plugins. The Plugin Manager then uses these to create the actual plugins. > So in > essence, the services are used as part of the factory. Note that decisions > about > whether to load a plugin or not do not have to be made when the plugin > service > s accessed. That is deferred until the Plugin actually needs to be wired. > > > > > > >> This is indeed very slow. Yes we have benchmarked it. But the Plugin > >> system IS a > >> Java Service. So instead, you need to only load 1 class for every jar > that > >> contains plugins. > >> The result is that you only need to load 4 or 5 classes. This is > >> considerably faster. > >> > > > > Yes, this is the approach where you insert another layer above the > service > > loader that does part of the job that the service loader can also do in a > > different way. Essentially you let the plugin libraries, each containing > > several plugin classes to implement their own service loaders. > > Sort of. With Log4j’s plugin system Plugins are not aware of ServiceLoader > at all. Other than that, it behaves the way you are describing. > > > > > > >> In addition, services loaded by ServiceLoader are singletons. > > > > > > No, they are not. Every ServiceLoader instance will create new instances, > > and even a single ServiceLoader can be asked calling the `reload()` > method > > to evict the instances from its cache and create new instances. > > Umm. Even calling reload() doesn’t change it from being a singleton. You > can > only have a single instance at a time. > > Yes, I suppose if you use multiple ServiceLoader instances then you can I > nstantiate multiple instances of the service. However, as you noted that > wouldn’t > be the best practice since the ServiceLoader really should be > instantiating a > factory that create as many instances as you want. > > > > > Perhaps developing a commons-plugin system is a good point to require > > application developers and plugin developers to use the module system. > > Please no. I would never encourage anyone to use it. It has caused nothing > but pain. Yes, that is opinion, but opinion based on experience. > > > > > > >> Again, I would suggest you look at the log4j-plugins implementation > before > >> making sweeping statements such as this. > >> > > > > I am not an expert about log4j and the plugin system of it,which you are. > > From this distance it seems that log4j uses factories for the plugins as > > services or builders in case there is a need for some configuration. In > > case of the builders the configuration parameters are injected that way > > this plugin system implements a simple DI framework like Plexus, Pico, > > Guava, Spring instead of using one. With the factory approach it does the > > same as the JDK ServiceLoader calling the `provider()` method. > > > > Again, if you find that answering the questions is difficult there may be > > several reasons. One thing is for sure: I am not trolling you. I am > asking > > these questions because I honestly believe that any new library should > > address the things that are not addressed by other systems and libraries, > > and should provide a better way to do things, otherwise they will not be > > widely used. > > > Frankly, I don’t think there is more to discuss. You asked why not just > use > ServiceLoader. But you have pretty much conceded that vanilla > ServiceLoader > by itself is not a good plugin system. You have agreed that using > ServiceLoader > to load factories is pretty much necessary. But since you don’t want to > look at > he Log4j plugin system to see how it makes creating plugins easy it is > hard to > discuss how it achieves the goals beyond just using ServiceLoader to load > factories. > > Ralph > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > -- Peter Verhas pe...@verhas.com t: +41791542095 skype: verhas
