[
https://issues.apache.org/jira/browse/FLINK-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-14382:
-----------------------------------
Description:
*(This ticket is a blocker for 1.10 release while critical for 1.9.x)*
When creating and starting up the yarn containers there are two issues with how
the {{FLINK_PLUGINS_DIR}} is being handled.
# Content of the {{plugins}} directory is currently added to the class path,
braking the encapsulation of the plugins from one another
# {{FLINK_PLUGINS_DIR}} is passed to the container as an absolute path as seen
by the client. Because of that TaskManager or JobManager can not use it.
Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}}
method (which calls relevant {{addEnvironmentFoldersToShipFiles}} and
{{uploadAndRegisterFiles}} methods)
(original description)
If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to [flink
configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
according to the environment.
In yarn mode, the local path will be set in flink-conf.yaml and used by
jobmanager and taskmanager. We will find the warning log like below.
{code:java}
2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
- Environment variable [FLINK_PLUGINS_DIR] is set to
[/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
but the directory doesn't exist
{code}
It was in introduced by FLINK-12143.
was:
*(This ticket is a blocker for 1.10 release while critical for 1.9.2.)*
When creating and starting up the yarn containers there are two issues with how
the {{FLINK_PLUGINS_DIR}} is being handled.
# Content of the {{plugins}} directory is currently added to the class path,
braking the encapsulation of the plugins from one another
# {{FLINK_PLUGINS_DIR}} is passed to the container as an absolute path as seen
by the client. Because of that TaskManager or JobManager can not use it.
Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}}
method (which calls relevant {{addEnvironmentFoldersToShipFiles}} and
{{uploadAndRegisterFiles}} methods)
(original description)
If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to [flink
configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
according to the environment.
In yarn mode, the local path will be set in flink-conf.yaml and used by
jobmanager and taskmanager. We will find the warning log like below.
{code:java}
2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
- Environment variable [FLINK_PLUGINS_DIR] is set to
[/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
but the directory doesn't exist
{code}
It was in introduced by FLINK-12143.
> Incorrect handling of FLINK_PLUGINS_DIR on Yarn
> -----------------------------------------------
>
> Key: FLINK-14382
> URL: https://issues.apache.org/jira/browse/FLINK-14382
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.9.0, 1.10.0
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Blocker
> Fix For: 1.10.0, 1.9.2
>
>
> *(This ticket is a blocker for 1.10 release while critical for 1.9.x)*
> When creating and starting up the yarn containers there are two issues with
> how the {{FLINK_PLUGINS_DIR}} is being handled.
> # Content of the {{plugins}} directory is currently added to the class path,
> braking the encapsulation of the plugins from one another
> # {{FLINK_PLUGINS_DIR}} is passed to the container as an absolute path as
> seen by the client. Because of that TaskManager or JobManager can not use it.
> Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}}
> method (which calls relevant {{addEnvironmentFoldersToShipFiles}} and
> {{uploadAndRegisterFiles}} methods)
> (original description)
> If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to
> [flink
> configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
> according to the environment.
> In yarn mode, the local path will be set in flink-conf.yaml and used by
> jobmanager and taskmanager. We will find the warning log like below.
> {code:java}
> 2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
> - Environment variable [FLINK_PLUGINS_DIR] is set to
> [/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
> but the directory doesn't exist
> {code}
> It was in introduced by FLINK-12143.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
