[
https://issues.apache.org/jira/browse/FLINK-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17018086#comment-17018086
]
Wenlong Lyu commented on FLINK-15635:
-------------------------------------
hi, [~twalthr], I would like to share some more classloading issues to solve in
long run:
# A sql platform product needs to provide as many connectors as possible for
the convenience of the users. It is not good to load them all in a single job,
which would waste a lot of resources and take long to deploy the job.
# There can be different connectors for a kind of storage in different
versions in the platform, which can have api or class conflicts, we can not
just load them together in a job, and shading the class can solve some of the
conflicts but not all.
# Catalog is the storage of table and function definition, but currently,
Catalog does not care about the related classloading when we need to get a
table from it.
And my some intuitive thought:
# Something like plugin in flink-core needs to be introduced to support
dynamic class loading for user defined connectors.
# Catalog can be responsible for providing jars for tables it provides. By
default, we can put connector jars in /opt/connectors/XXX in flink distribution
and catalog can search jars needed from the directories according to the
factories matched result.
# table env should be able to adding jars from catalog to job graph when
submitting a job to a flink cluster.
What do you think?
> Allow passing a ClassLoader to EnvironmentSettings
> --------------------------------------------------
>
> Key: FLINK-15635
> URL: https://issues.apache.org/jira/browse/FLINK-15635
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API, Table SQL / Planner
> Reporter: Timo Walther
> Priority: Major
>
> We had a couple of class loading issues in the past because people forgot to
> use the right classloader in {{flink-table}}. The SQL Client executor code
> hacks a classloader into the planner process by using {{wrapClassLoader}}
> that sets the threads context classloader.
> Instead we should allow passing a class loader to environment settings. This
> class loader can be passed to the planner and can be stored in table
> environment, table config, etc. to have a consistent class loading behavior.
> Having this in place should replace the need for
> {{Thread.currentThread().getContextClassLoader()}} in the entire
> {{flink-table}} module.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
