[
https://issues.apache.org/jira/browse/FLINK-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959717#comment-16959717
]
Yang Wang edited comment on FLINK-14382 at 10/25/19 12:38 PM:
--------------------------------------------------------------
[~pnowojski] [~trohrmann]
> About the environment
What i mean is to set `ENV_FLINK_PLUGINS_DIR` to `$PWD/plugins` for job manager
in `YarnClusterDescriptor`. Then it will be parsed by
`GlobalConfiguration#loadConfiguration` in `YarnJobClusterEntrypoint` and set
to configuration. So we could use it from flink configuration just like
`configuration.getString(ConfigConstants.ENV_FLINK_PLUGINS_DIR)` later.
Are you suggesting to set `ConfigConstants.ENV_FLINK_PLUGINS_DIR` directly to a
relative path(./plugins) to flink configuration in client?
> About the test
Do you mean to add an end to end test to cover the plugins function? Or just
check the correct classpath and flink configuration have been set.
was (Author: fly_in_gis):
[~pnowojski]
> About the environment
What i mean is to set `ENV_FLINK_PLUGINS_DIR` to `$PWD/plugins` for job manager
in `YarnClusterDescriptor`. Then it will be parsed by
`GlobalConfiguration#loadConfiguration` in `YarnJobClusterEntrypoint` and set
to configuration. So we could use it from flink configuration just like
`configuration.getString(ConfigConstants.ENV_FLINK_PLUGINS_DIR)` later.
Are you suggesting to set `ConfigConstants.ENV_FLINK_PLUGINS_DIR` directly to a
relative path(./plugins) to flink configuration in client?
> About the test
Do you mean to add an end to end test to cover the plugins function? Or just
check the correct classpath and flink configuration have been set.
> Incorrect handling of FLINK_PLUGINS_DIR on Yarn
> -----------------------------------------------
>
> Key: FLINK-14382
> URL: https://issues.apache.org/jira/browse/FLINK-14382
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.9.0, 1.10.0
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Blocker
> Fix For: 1.10.0, 1.9.2
>
>
> *(This ticket is a blocker for 1.10 release while critical for 1.9.x)*
> When creating and starting up the yarn containers there are two issues with
> how the {{FLINK_PLUGINS_DIR}} is being handled.
> # Content of the {{plugins}} directory is currently added to the class path,
> braking the encapsulation of the plugins from one another
> # {{FLINK_PLUGINS_DIR}} is passed to the container as an absolute path as
> seen by the client. Because of that TaskManager or JobManager can not use it.
> Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}}
> method (which calls relevant {{addEnvironmentFoldersToShipFiles}} and
> {{uploadAndRegisterFiles}} methods)
> (original description)
> If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to
> [flink
> configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
> according to the environment.
> In yarn mode, the local path will be set in flink-conf.yaml and used by
> jobmanager and taskmanager. We will find the warning log like below.
> {code:java}
> 2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
> - Environment variable [FLINK_PLUGINS_DIR] is set to
> [/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
> but the directory doesn't exist
> {code}
> It was in introduced by FLINK-12143.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
