[
https://issues.apache.org/jira/browse/FLINK-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047720#comment-17047720
]
Piotr Nowojski commented on FLINK-15786:
----------------------------------------
[~maguowei] I think {{FlinkKafkaPartitioner}} is a valid concern and I haven't
considered it, as until now the idea of using plugins for DataStream connectors
was just a high level idea in my head and nobody (that I know of) was doing any
proper research in that direction. The general problem is how to customise a
plugin by injecting a custom code into it.
Could you explain a bit more what do you mean by {{XClassloader}}?
For one thing, as long as the interactions between custom class and the plugin
are trivial, I think there are no problems with loading the custom class from
any class loader (including user). Where by trivial I mean the interface of
that custom class refers just to standard java classes or Flink SPI (anything
that's annotated as Public, PublicEvolving or Experimental). As Flink SPI and
standard java classes are always loaded from parent class loader.
In other words {{FlinkKafkaPartitioner}}, and all classes/objects that it's
using in it's signature (argument/return types) must be treated as part of the
Flink SPI. In that case, {{FlinkKafkaPartitioner}} instance (created in any
class loader), can be safely passed to any plugin. There is no need for user
class implementing {{FlinkKafkaPartitioner}} to have access to any plugin
specific class. Just it's implementation and Kafka connector both depend on the
Flink SPI.
Another issue is how to actually create and pass an instance of
{{FlinkKafkaPartitioner}}. But that I think also could be solved in some ways
(instantiate on the client side, serialise it and put it somehow into
{{SourceDescriptor}} and during the deserialisation load it from the user's
class loader context?).
> Load connector code with separate classloader
> ---------------------------------------------
>
> Key: FLINK-15786
> URL: https://issues.apache.org/jira/browse/FLINK-15786
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: Guowei Ma
> Priority: Major
> Labels: usability
>
> Currently, connector code can be seen as part of user code. Usually, users
> only need to add the corresponding connector as a dependency and package it
> in the user jar. This is convenient enough.
> However, connectors usually need to interact with external systems and often
> introduce heavy dependencies, there is a high possibility of a class conflict
> of different connectors or the user code of the same job. For example, every
> one or two weeks, we will receive issue reports relevant with connector class
> conflict from our users. The problem can get worse when users want to analyze
> data from different sources and write output to different sinks.
> Using separate classloader to load the different connector code could resolve
> the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
