I agree with Jing that the current doc is quite preliminary, and needs to be turned into a FLIP.
I'll be very interested in this FLIP, and looking forward to it. We've suffered from reshading/relocating heavy connector dependencies for a long time. Besides, we've implemented a plugin mechanism to load hive udf in our batch SQL scenario internally, it saves us a lot of effort to handle the dependency conflicts. The most challenging part would be the (de)serialization of those classes loaded through plugins according to our experience. (I noticed you have already considered this) Jing Ge <j...@ververica.com.invalid> 于2023年7月21日周五 06:44写道： > Hi Zhiqiang, > > Thanks for your proposal. The idea is very interesting! Since almost all > connectors are or will be externalized, the pluggable design you suggested > could help reduce the development effort. > > As you mentioned, the attached doc contains only your preliminary idea. I > would suggest that you could turn it into a FLIP with more details and do > the followings: > > 1. Describe a real conflict case that will benefit from the pluggable > design > 2. Describe where you want to modify the code, or provide a POC branch > 3. Guideline to tell users how to use it, i.e. where (the plugins dir) > should the connector jar be located, how does it look like with an example, > etc. > > WDYT? > > Best regards, > Jing > > > On Fri, Jul 14, 2023 at 3:00 PM zhiqiang li <lizhiqiang....@gmail.com> > wrote: > > > Hi devs, > > > > I have observed that in [1], connectors and formats are pluggable, > > allowing user code to be easily integrated. The advantages of having > > pluggable connectors are evident, as it helps avoid conflicts between > > different versions of jar packages. If classloader isolation is not used, > > shading becomes necessary to resolve conflicts, resulting in a > significant > > waste of development time. I understand that implementing this change may > > require numerous API modifications, so I would like to discuss in this > > email. > > > > > Plugins cannot access classes from other plugins or from Flink that > have > > not been specifically whitelisted. > > > This strict isolation allows plugins to contain conflicting versions of > > the same library without the need to relocate classes or to converge to > > common versions. > > > Currently, file systems and metric reporters are pluggable but, in the > > future, connectors, formats, and even user code should also be pluggable. > > > > [2] It is my preliminary idea. > > > > [1] > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/ > > [2] > > > https://docs.google.com/document/d/1XP2fBpcntK0YIdQ_Ax7JV2MhNdebvkFxSiNJRp6WQ24/edit?usp=sharing > > > > > > Best, > > Zhiqiang > > > > > -- Best, Benchao Li
