Narcasserun opened a new issue #6589: URL: https://github.com/apache/dolphinscheduler/issues/6589
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement. ### Description Hello everyone The subject of this email is to initiate a discussion on DS plugin loading and packaging Design. DS plugin related load adjustment is very important,so I will use Chinese and English to describe. ### 1.aims - The core purpose of DS plugin is to decouple the upper core logic of DS from the specific implementation. Some design concepts help DS become a very easy to expand project. Recently, I have been doing secondary development based on ds2.0, but I found that the packaged directory of each DS plug-in has the same dependent jar, and the transformation made by Presto on SPI loading results in nearly 1g compressed packages after packaging. I think DS community doesn't want to see this. Because I participated in the development of the company's internal computing and metadata middleware and adopted the plug-in design concept, but the maintainability became very poor because each plug-in managed the common dependency independently. - We hope that DS can not only improve its scalability, but also improve its maintainability through class loading sharing and directory sharing. DS plug-ins only need to maintain the implementation of plug-ins, which will be very friendly to new developers who want to participate in the DS community. ### 2. The following is my analysis of the advantages and disadvantages of DS loading plug-ins? The transformation of ds2.0 has come to an end, and its scalability has been greatly improved. The design of plugin is realized by using java SPI, and the upper core logic of DS is decoupled from the specific implementation. SPI implements classes through cyclic loading. Obviously, it will load all classes together, whether used or not, which causes a certain waste of resources. DS / Presto defines the same rule * * plugins should not have access to the system (application) class loader * *. Although this avoids a certain waste of resources, the disadvantage is that there will be duplicate jars in each plug-in packaging directory, resulting in an increase of nearly 50% from about 130m to about 200m, as shown in Figure 1 and Figure 2. Although Trino installation package has been optimized in structure, its disadvantages still exist. As shown in Figure 3 and Figure 4, there is still nearly 600m after compression. ### 3. What benefits and disadvantages can class loading sharing and directory sharing bring to ds2.0? - Class loading sharing: plugins should access to the system (application) class loader. Presto / Trino makes such rules because there are too many plug-ins and plug-ins and applications are in the same JVM instance. Multiple loaders are shared through thread context and will be loaded circularly. Unlike DS, it has four JVM instances. Take the alarm module for example. In fact, each alarm plug-in depends almost the same, and the dependence is relatively simple. It is completely different from the Trino / Presto plug-in, which has a strong dependence on the environment. After class loading and sharing, only the corresponding plug-ins need to be packaged into the plug-in directory in each plug-in directory to realize the separation of plug-in dependency and plug-in implementation. In the plugin directory, you only need to maintain the jar implemented by the plug-in. - Directory sharing: the implementation of directory sharing is simply to package the jar packages common to plug-ins. It is necessary to load multiple directories when the plug-ins are loaded, or do directory mapping for the jars used by the plug-ins when installing. The disadvantage is that the general directory of plug-ins is not easy to maintain, and the complexity of packaging commands and class loading logic will increase, and the readability of code will decrease. I personally think this is one of the reasons why Trino / Presto developers don't do it. Here's the picture Figure 1: Figure 2:  Figure 3:  Figure 4: ![Uploading image.png…]() ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
