[
https://issues.apache.org/jira/browse/FLINK-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969388#comment-16969388
]
Victor Wong edited comment on FLINK-14626 at 11/7/19 4:20 PM:
--------------------------------------------------------------
[~chesnay], thanks for your quick response. As an engineer providing
company-wide stream processing capabilities for other teams, I have seen quite
a few errors related to this issue. It's really prone to including Hadoop
dependencies in the uber jar, especially when your job deals with Hadoop
systems, e.g. HBase.
But I agree that it _would change existing behavior and could result in subtle
bugs,_ so I will close this issue and find another way to solve the related
problems.
was (Author: victor-wong):
[~chesnay], thanks for your quick response. As an engineer providing
company-wide stream processing capabilities for other teams, I have seen quite
a few errors related to this issue. It's really prone to including Hadoop
dependencies in the uber jar, especially when your job deals with Hadoop
systems, e.g. HBase.
But I agree that it _would change existing behavior and could result in subtle
bugs,_ so I will close this PR and find another way to solve the related
problems.
> User jar packaged with hadoop dependencies may cause class conflit with
> hadoop jars on yarn
> -------------------------------------------------------------------------------------------
>
> Key: FLINK-14626
> URL: https://issues.apache.org/jira/browse/FLINK-14626
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.9.1
> Reporter: Victor Wong
> Priority: Major
>
> Currently, the yarn application classpath is placed behind Flink classpath
> (including user jars), which will cause conflict if the user jar accidentally
> included hadoop dependencies.
> {code:java}
> // org.apache.flink.yarn.Utils#setupYarnClassPath
> public static void setupYarnClassPath(Configuration conf, Map<String, String>
> appMasterEnv) {
> addToEnvironment(
> appMasterEnv,
> Environment.CLASSPATH.name(),
> appMasterEnv.get(ENV_FLINK_CLASSPATH));
> String[] applicationClassPathEntries = conf.getStrings(
> YarnConfiguration.YARN_APPLICATION_CLASSPATH,
> YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH);
> for (String c : applicationClassPathEntries) {
> addToEnvironment(appMasterEnv, Environment.CLASSPATH.name(), c.trim());
> }
> }
> {code}
> Maybe we should place the user jars behind yarn application classpath when
> `org.apache.flink.yarn.configuration.YarnConfigOptions.UserJarInclusion` is
> set to LAST, like this "flink-xxx.jar:hadoop-xxx.jar:user-xxx.jar".
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
