[
https://issues.apache.org/jira/browse/FLINK-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-14382:
-----------------------------------
Description:
When creating and starting up the yarn containers there are two issues with how
the {{FLINK_PLUGINS_DIR}} is being handled:
# content of the {{plugins}} directory is added to the class path -
invalidating the whole plugins idea, that they should be loaded in separate
class loaders
# {{FLINK_PLUGINS_DIR}} is passed to the TaskManager and JobManager as an
absolute path, which means it's invalid from the perspective of processes
running inside the yarn. After uploading, {{plugins}} directory should be
referred as relative path.
Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}}
method (which calls {{addEnvironmentFoldersToShipFiles}} and
{{uploadAndRegisterFiles}}).
(originally reported description):
If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to [flink
configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
according to the environment.
In yarn mode, the local path will be set in flink-conf.yaml and used by
jobmanager and taskmanager. We will find the warning log like below.
{code:java}
2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
- Environment variable [FLINK_PLUGINS_DIR] is set to
[/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
but the directory doesn't exist
{code}
It was in introduced by FLINK-12143.
was:
When creating and starting up the yarn containers there are two issues with how
the
If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to [flink
configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
according to the environment.
In yarn mode, the local path will be set in flink-conf.yaml and used by
jobmanager and taskmanager. We will find the warning log like below.
{code:java}
2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
- Environment variable [FLINK_PLUGINS_DIR] is set to
[/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
but the directory doesn't exist
{code}
It was in introduced by FLINK-12143.
> Incorrectly handling of FLINK_PLUGINS_DIR on Yarn
> -------------------------------------------------
>
> Key: FLINK-14382
> URL: https://issues.apache.org/jira/browse/FLINK-14382
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.9.0, 1.10.0
> Reporter: Yang Wang
> Priority: Major
> Fix For: 1.10.0, 1.9.2
>
>
> When creating and starting up the yarn containers there are two issues with
> how the {{FLINK_PLUGINS_DIR}} is being handled:
> # content of the {{plugins}} directory is added to the class path -
> invalidating the whole plugins idea, that they should be loaded in separate
> class loaders
> # {{FLINK_PLUGINS_DIR}} is passed to the TaskManager and JobManager as an
> absolute path, which means it's invalid from the perspective of processes
> running inside the yarn. After uploading, {{plugins}} directory should be
> referred as relative path.
> Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}}
> method (which calls {{addEnvironmentFoldersToShipFiles}} and
> {{uploadAndRegisterFiles}}).
> (originally reported description):
> If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to
> [flink
> configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
> according to the environment.
> In yarn mode, the local path will be set in flink-conf.yaml and used by
> jobmanager and taskmanager. We will find the warning log like below.
> {code:java}
> 2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
> - Environment variable [FLINK_PLUGINS_DIR] is set to
> [/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
> but the directory doesn't exist
> {code}
> It was in introduced by FLINK-12143.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
