[
https://issues.apache.org/jira/browse/FLINK-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-14382:
-----------------------------------
Description:
When creating and starting up the yarn containers there are two issues with how
the {{FLINK_PLUGINS_DIR}} is being handled.
# Content of the {{plugins}} directory is currently added to the class path,
braking the encapsulation of the plugins from one another
# {{FLINK_PLUGINS_DIR}} is passed to the container as an absolute path as seen
by the client. Because of that TaskManager or JobManager can not use it.
(original description)
If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to [flink
configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
according to the environment.
In yarn mode, the local path will be set in flink-conf.yaml and used by
jobmanager and taskmanager. We will find the warning log like below.
{code:java}
2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
- Environment variable [FLINK_PLUGINS_DIR] is set to
[/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
but the directory doesn't exist
{code}
It was in introduced by FLINK-12143.
was:
When creating and starting up the yarn containers there are two issues with how
the {{FLINK_PLUGINS_DIR}} is being handled:
# content of the {{plugins}} directory is added to the class path -
invalidating the whole plugins idea, that they should be loaded in separate
class loaders
# {{FLINK_PLUGINS_DIR}} is passed to the TaskManager and JobManager as an
absolute path, which means it's invalid from the perspective of processes
running inside the yarn. After uploading, {{plugins}} directory should be
referred as relative path.
Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}}
method (which calls {{addEnvironmentFoldersToShipFiles}} and
{{uploadAndRegisterFiles}}).
(originally reported description):
If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to [flink
configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
according to the environment.
In yarn mode, the local path will be set in flink-conf.yaml and used by
jobmanager and taskmanager. We will find the warning log like below.
{code:java}
2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
- Environment variable [FLINK_PLUGINS_DIR] is set to
[/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
but the directory doesn't exist
{code}
It was in introduced by FLINK-12143.
> Incorrectly handling of FLINK_PLUGINS_DIR on Yarn
> -------------------------------------------------
>
> Key: FLINK-14382
> URL: https://issues.apache.org/jira/browse/FLINK-14382
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.9.0, 1.10.0
> Reporter: Yang Wang
> Priority: Major
> Fix For: 1.10.0, 1.9.2
>
>
> When creating and starting up the yarn containers there are two issues with
> how the {{FLINK_PLUGINS_DIR}} is being handled.
> # Content of the {{plugins}} directory is currently added to the class path,
> braking the encapsulation of the plugins from one another
> # {{FLINK_PLUGINS_DIR}} is passed to the container as an absolute path as
> seen by the client. Because of that TaskManager or JobManager can not use it.
> (original description)
> If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to
> [flink
> configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
> according to the environment.
> In yarn mode, the local path will be set in flink-conf.yaml and used by
> jobmanager and taskmanager. We will find the warning log like below.
> {code:java}
> 2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
> - Environment variable [FLINK_PLUGINS_DIR] is set to
> [/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
> but the directory doesn't exist
> {code}
> It was in introduced by FLINK-12143.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
