[
https://issues.apache.org/jira/browse/FLINK-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959663#comment-16959663
]
Piotr Nowojski edited comment on FLINK-14382 at 10/25/19 11:31 AM:
-------------------------------------------------------------------
> I think we just need to ship the plugin directory only and do not add to
>classpath
Yes, exactly.
> Also the environment `ENV_FLINK_PLUGINS_DIR` should be set for job manager
>and task manager container.
It was a conscious decision discussed by me, [~trohrmann] and couple of other
people, to not relay environment variables from the entry points (for example
flink-cli), but read them once and pass them programatically via a config file,
so I would keep it as it is. Environment variables have all the drawbacks of
using global variables plus some more (hard to search usages for example). Also
it's not nice that we have a multiple ways how to configure Flink. In the long
run we will probably want to unify everything and pass all of the configuration
in the same way, and moving away from relaying on environment variables (unless
we have to).
I'm not saying that we can not change the direction, but I'm just writing down
our reasoning behind this.
> Are you working on this issue, [~pnowojski] ? If not, would you mind to
>assign this ticket to me? I have some time to fix it.
Sure :) Done. Keep in mind that this probably needs some good test coverage
(probably both unit test + maybe an end to end test).
was (Author: pnowojski):
> I think we just need to ship the plugin directory only and do not add to
>classpath
Yes, exactly.
> Also the environment `ENV_FLINK_PLUGINS_DIR` should be set for job manager
>and task manager container.
It was a conscious decision discussed by me, [~trohrmann] and couple of other
people, to not relay environment variables from the entry points (for example
flink-cli), but read them once and pass them programatically via a config file,
so I would keep it as it is. Environment variables have all the drawbacks of
using global variables plus some more (hard to search usages for example). Also
it's not nice that we have a multiple ways how to configure Flink. In the long
run we will probably want to unify everything and pass all of the configuration
in the same way, and moving away from relaying on environment variables (unless
we have to).
I'm not saying that we can not change the direction, but I'm just writing down
our reasoning behind this.
> Are you working on this issue, [~pnowojski] ? If not, would you mind to
>assign this ticket to me? I have some time to fix it.
Sure :) Done. Keep in mind that this probably needs some good test coverage
(probably both unit test + maybe an end to end test).
> Incorrect handling of FLINK_PLUGINS_DIR on Yarn
> -----------------------------------------------
>
> Key: FLINK-14382
> URL: https://issues.apache.org/jira/browse/FLINK-14382
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.9.0, 1.10.0
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Critical
> Fix For: 1.10.0, 1.9.2
>
>
> When creating and starting up the yarn containers there are two issues with
> how the {{FLINK_PLUGINS_DIR}} is being handled.
> # Content of the {{plugins}} directory is currently added to the class path,
> braking the encapsulation of the plugins from one another
> # {{FLINK_PLUGINS_DIR}} is passed to the container as an absolute path as
> seen by the client. Because of that TaskManager or JobManager can not use it.
> Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}}
> method (which calls relevant {{addEnvironmentFoldersToShipFiles}} and
> {{uploadAndRegisterFiles}} methods)
> (original description)
> If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to
> [flink
> configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
> according to the environment.
> In yarn mode, the local path will be set in flink-conf.yaml and used by
> jobmanager and taskmanager. We will find the warning log like below.
> {code:java}
> 2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
> - Environment variable [FLINK_PLUGINS_DIR] is set to
> [/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
> but the directory doesn't exist
> {code}
> It was in introduced by FLINK-12143.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
