[
https://issues.apache.org/jira/browse/FLINK-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049216#comment-17049216
]
Piotr Nowojski commented on FLINK-15786:
----------------------------------------
> There might be two options:
1. The problem with that, is that we would design a plugin to a plugin. It
might be the cleanest solution, but might be complicated to explain and I don't
know what would be the side effects.
2. I think easy to do, but user would have to implement custom
{{FlinkKafkaPartioner}} in some separate project & compile it to a jar
independently of his job?
3. Another option, to just pull {{FlinkKafkaPartioner}} class into the Flink's
SPI. Interface would be defined in {{flink-core}}. It would be polluting the
general SPI with some plugin's class, but benefit would flat single level
plugins structure and easy to extend by the user?
> And we keep creating the source operator at the client.
But what you described for {{XClassLoader}}, doesn't solve the problem of for
example using two incompatible connectors (like different Kafka versions) if we
keep loading them as they are loaded now at the client? For that we need some
form of builder/factory with the SourceDescriptor as I described above.
> The order of searching for a class is:
>
> 1. jar in the user classpath
> 2. jar in the connectors jars
> 3. jar in the system classpath
How would you know which class to look in 1. or 2.? If that's fixed order, user
could brake the connector by including some dependency?
> Load connector code with separate classloader
> ---------------------------------------------
>
> Key: FLINK-15786
> URL: https://issues.apache.org/jira/browse/FLINK-15786
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: Guowei Ma
> Priority: Major
> Labels: usability
>
> Currently, connector code can be seen as part of user code. Usually, users
> only need to add the corresponding connector as a dependency and package it
> in the user jar. This is convenient enough.
> However, connectors usually need to interact with external systems and often
> introduce heavy dependencies, there is a high possibility of a class conflict
> of different connectors or the user code of the same job. For example, every
> one or two weeks, we will receive issue reports relevant with connector class
> conflict from our users. The problem can get worse when users want to analyze
> data from different sources and write output to different sinks.
> Using separate classloader to load the different connector code could resolve
> the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
