Thank you Piotr for bringing this discussion to the mailing list! As it was not explicitly mentioned in the first email, I wanted to add that there is also already an open PR[1] with my implementation of the basic plugin mechanism for FileSystem. Looking forward to some feedback from the community.
[1] https://github.com/apache/flink/pull/8038 <https://github.com/apache/flink/pull/8038> Best, Stefan > On 10. Apr 2019, at 17:08, zhijiang <wangzhijiang...@aliyun.com.INVALID> > wrote: > > Thanks Piotr for proposing this new feature. > > The solution for class loader issue is really helpful in production, and we > ofen encountered this pain point before. > It might bring more possibilities based on this pluggable mechanism. Hope to > see the progress soon. :) > > Best, > Zhijiang > ------------------------------------------------------------------ > From:Jeff Zhang <zjf...@gmail.com> > Send Time:2019年4月10日(星期三) 22:01 > To:dev <dev@flink.apache.org> > Subject:Re: Introducing Flink's Plugin mechanism > > Thank Piotr for driving this plugin mechanism. Pluggability is pretty > important for the ecosystem of flink. > > Piotr Nowojski <pi...@ververica.com> 于2019年4月10日周三 下午5:48写道： > >> Hi Flink developers, >> >> I would like to introduce a new plugin loading mechanism that we are >> working on right now [1]. The idea is quite simple: isolate services in >> separate independent class loaders, so that classes and dependencies do not >> leak between them and/or Flink runtime itself. Currently we have quite some >> problems with dependency convergence in multiple places. Some of them we >> are solving by shading (built in file systems, metrics), some we are >> forcing users to deal with them (custom file systems/metrics) and others we >> do not solve (connectors - we do not support using different Kafka versions >> in the same job/SQL). With proper plugins, that are loaded in independent >> class loaders, those issues could be solved in a generic way. >> >> Current scope of implementation targets only file systems, without a >> centralised Plugin architecture and with Plugins that are only “statically” >> initialised at the TaskManager and JobManager start up. More or less we are >> just replacing the way how FileSystem’s implementations are discovered & >> loaded. >> >> In the future this idea could be extended to different modules, like >> metric reporters, connectors, functions/data types (especially in SQL), >> state backends, internal storage or other future efforts. Some of those >> would be easier than others: the metric reporters would require some >> smaller refactor, while connectors would require some bigger API design >> discussions, which I would like to avoid at the moment. Nevertheless I >> wanted to reach out with this idea so if some other potential use cases pop >> up in the future, more people will be aware. >> >> Piotr Nowojski >> >> >> [1] https://issues.apache.org/jira/browse/FLINK-11952 < >> https://issues.apache.org/jira/browse/FLINK-11952> > > > > -- > Best Regards > > Jeff Zhang >
