[
https://issues.apache.org/jira/browse/FLINK-24897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446258#comment-17446258
]
Biao Geng edited comment on FLINK-24897 at 2/10/22, 2:35 AM:
-------------------------------------------------------------
Hi [~trohrmann] , would you mind giving any comments on if we have the promise
that jars under {{usrlib}} will always be loaded by user classloader?
Besides, after above discussion with Yang, the current solution is:
1. {{usrlib}} will be shipped automatically if it exists.
2. If we add {{usrlib}} in ship files again, we will throw exception.
3. {{usrlib}} will work for both per job and application mode.
4. Jars in {{usrlib}} will be loaded by user classloader only when
UserJarInclusion is DISABLED.
In other cases, AppClassLoader will be used.
5. {{usrlib}} directory should not exist in {{yarn.provided.lib.dirs}} or
{{{}yarn.ship-files{}}}. If users want to ship {{{}usrlib{}}}, they should just
create {{usrlib}} under FLINK_HOME dir.
6. key of CLASSPATH_INCLUDE_USER_JAR will be renamed from
\{{yarn.per-job-cluster.include-user-jar}} to
\{{yarn.classpath.include-user-jar}} for better naming.
Thanks.
was (Author: bgeng777):
Hi [~trohrmann] , would you mind giving any comments on if we have the promise
that jars under {{usrlib}} will always be loaded by user classloader?
Besides, after above discussion with Yang, the current solution is:
1. {{usrlib}} will be shipped automatically if it exists.
2. If we add {{usrlib}} in ship files again, we will throw exception.
3. {{usrlib}} will work for both per job and application mode.
4. Jars in {{usrlib}} will be loaded by user classloader only when
UserJarInclusion is DISABLED.
In other cases, AppClassLoader will be used.
5. {{usrlib}} directory should not exist in {{yarn.provided.lib.dirs}} or
{{yarn.ship-files}}. If users want to ship {{usrlib}}, they should just create
{{usrlib}} under FLINK_HOME dir.
Thanks.
> Make usrlib work for YARN application and per-job
> -------------------------------------------------
>
> Key: FLINK-24897
> URL: https://issues.apache.org/jira/browse/FLINK-24897
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Reporter: Biao Geng
> Assignee: Biao Geng
> Priority: Major
> Labels: pull-request-available
>
> Hi there,
> I am working to utilize application mode to submit flink jobs to YARN cluster
> but I find that currently there is no easy way to ship my user-defined
> jars(e.g. some custom connectors or udf jars that would be shared by some
> jobs) and ask the FlinkUserCodeClassLoader to load classes in these jars.
> I checked some relevant jiras, like FLINK-21289. In k8s mode, there is a
> solution that users can use `usrlib` directory to store their user-defined
> jars and these jars would be loaded by FlinkUserCodeClassLoader when the job
> is executed on JM/TM.
> But on YARN mode, `usrlib` does not work as that:
> In this method(org.apache.flink.yarn.YarnClusterDescriptor#addShipFiles), if
> I want to use `yarn.ship-files` to ship `usrlib` from my flink client(in my
> local machine) to remote cluster, I must not set UserJarInclusion to
> DISABLED due to the checkArgument(). However, if I do not set that option to
> DISABLED, the user jars to be shipped will be added into systemClassPaths. As
> a result, classes in those user jars will be loaded by AppClassLoader.
> But if I do not ship these jars, there is no convenient way to utilize these
> jars in my flink run command. Currently, all I can do seems to use `-C`
> option, which means I have to upload my jars to some shared store first and
> then use these remote paths. It is not so perfect as we have already make it
> possible to ship jars or files directly and we also introduce `usrlib` in
> application mode on YARN. It would be more user-friendly if we can allow
> shipping `usrlib` from local to remote cluster while using
> FlinkUserCodeClassLoader to load classes in the jars in `usrlib`.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
