[
https://issues.apache.org/jira/browse/FLINK-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049222#comment-17049222
]
Arvid Heise commented on FLINK-15786:
-------------------------------------
First of all, I'd very much like to also make connectors pluggable. [~maguowei]
raised a ton of concerns that I haven't thought through, but it still feels
solvable.
>From the user perspective, I'd like to use the connector as is. I wouldn't
>mind changing the scope in such a way that it wouldn't be bundled in my fat
>jar (I'd probably even like it, since my fat jar gets smaller).
So I'm currently not convinced of the generic configuration approach.
That leaves us with tinkering with the classloading as outlined in
[~maguowei]'s last comment.
As a user, I'd like to put my Kafka-connector-X into the plugins folder and it
somehow automatically works with my smaller fat jar.
The `XClassLoader` already does a good job. But we also have the option to swap
1. and 2.
With the current order, a user that (accidentally) bundles kafka-connector in
addition to the plugin, would load his bundled version. That gives him the
freedom to use a different version than in the plugin, but it would also
potentially pollute the perm space in case of multiple jobs per session.
With swapped order, we force a specific version. User code classes would be
completely ignored.
I'm not sure which option is better but I just wanted to leave my trail of
thought here.
> Load connector code with separate classloader
> ---------------------------------------------
>
> Key: FLINK-15786
> URL: https://issues.apache.org/jira/browse/FLINK-15786
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: Guowei Ma
> Priority: Major
> Labels: usability
>
> Currently, connector code can be seen as part of user code. Usually, users
> only need to add the corresponding connector as a dependency and package it
> in the user jar. This is convenient enough.
> However, connectors usually need to interact with external systems and often
> introduce heavy dependencies, there is a high possibility of a class conflict
> of different connectors or the user code of the same job. For example, every
> one or two weeks, we will receive issue reports relevant with connector class
> conflict from our users. The problem can get worse when users want to analyze
> data from different sources and write output to different sinks.
> Using separate classloader to load the different connector code could resolve
> the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
