[
https://issues.apache.org/jira/browse/FLINK-14382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959752#comment-16959752
]
Piotr Nowojski edited comment on FLINK-14382 at 10/25/19 1:08 PM:
------------------------------------------------------------------
> Are you suggesting to set `ConfigConstants.ENV_FLINK_PLUGINS_DIR` directly to
>a relative path(./plugins) to flink configuration in client?
Yes, that's the idea, unless this is for some reason much more difficult or has
some other issues. Or at least create a copy of the client's configuration,
that will be used only inside the yarn containers with the paths correctly
converted to ones that are visible/valid from within the container.
> Do you mean to add an end to end test to cover the plugins function? Or just
>check the correct classpath and flink configuration have been set.
The first one. I think the biggest issue here is that the whole plugins system
was not working as designed in yarn. It was loading the classes, but not via
the plugins mechanism and without the separation. There is already an
end-to-end test coverage of plugins, including yarn (introduced by [this
commit|https://github.com/apache/flink/commit/b1d5bf963e9ce062b2e349f5296a56da6816340e]),
but it's clearly not sufficient - current test is checking whether we can read
a {{--input dummy://localhost/words}} file provided by a {{DummyFS}} configured
as a plugin. And this works, however it doesn't test the essence of the Plugins.
I think probably we need to repeat in end-to-end test the similar logic that is
already implemented by the unit test
{{org.apache.flink.test.plugin.PluginLoaderTest}}.
For example in the end to end test load not only one file from one dummy
FIleSystem, but from two ({{DummyFSFactory}} and {{AnotherDummyFSFactory}}).
Each of this dummy FileSystem should validate that there they can not see any
classes from the other FileSystem ({{DummyFSFactory}} shouldn't be visible when
loading {{AnotherDummyFSFactory}} and vice versa). I guess for this we would
also need to modify {{WordCount}} example to union multiple inputs.
was (Author: pnowojski):
> Are you suggesting to set `ConfigConstants.ENV_FLINK_PLUGINS_DIR` directly to
>a relative path(./plugins) to flink configuration in client?
Yes, that's the idea, unless this is for some reason much more difficult or has
some other issues. Or at least create a copy of the client's configuration,
that will be used only inside the yarn containers with the paths correctly
converted to ones that are visible/valid from within the container.
> Do you mean to add an end to end test to cover the plugins function? Or just
>check the correct classpath and flink configuration have been set.
The first one. I think the biggest issue here is that the whole plugins system
was not working as designed in yarn. It was loading the classes, but not via
the plugins mechanism and without the separation. There is already an
end-to-end test coverage of plugins, including yarn (introduced by [this
commit|[https://github.com/apache/flink/commit/b1d5bf963e9ce062b2e349f5296a56da6816340e]]),
but it's clearly not sufficient - current test is checking whether we can read
a \{{--input dummy://localhost/words}} file provided by a \{{DummyFS}}
configured as a plugin. And this works, however it doesn't test the essence of
the Plugins.
I think probably we need to repeat in end-to-end test the similar logic that is
already implemented by the unit test
{{org.apache.flink.test.plugin.PluginLoaderTest}}.
For example in the end to end test load not only one file from one dummy
FIleSystem, but from two (\{{DummyFSFactory}} and {{AnotherDummyFSFactory}}).
Each of this dummy FileSystem should validate that there they can not see any
classes from the other FileSystem (\{{DummyFSFactory}} shouldn't be visible
when loading {{AnotherDummyFSFactory}} and vice versa). I guess for this we
would also need to modify {{WordCount}} example to union multiple inputs.
> Incorrect handling of FLINK_PLUGINS_DIR on Yarn
> -----------------------------------------------
>
> Key: FLINK-14382
> URL: https://issues.apache.org/jira/browse/FLINK-14382
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.9.0, 1.10.0
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Blocker
> Fix For: 1.10.0, 1.9.2
>
>
> *(This ticket is a blocker for 1.10 release while critical for 1.9.x)*
> When creating and starting up the yarn containers there are two issues with
> how the {{FLINK_PLUGINS_DIR}} is being handled.
> # Content of the {{plugins}} directory is currently added to the class path,
> braking the encapsulation of the plugins from one another
> # {{FLINK_PLUGINS_DIR}} is passed to the container as an absolute path as
> seen by the client. Because of that TaskManager or JobManager can not use it.
> Both bugs are probably contained to {{YarnClusterDescriptor#startAppMaster}}
> method (which calls relevant {{addEnvironmentFoldersToShipFiles}} and
> {{uploadAndRegisterFiles}} methods)
> (original description)
> If we do not set FLINK_PLUGINS_DIR in flink-conf.yaml, it will be set to
> [flink
> configuration|https://github.com/apache/flink/blob/9e6ff81e22d6f5f04abb50ca1aea84fd2542bf9d/flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java#L158]
> according to the environment.
> In yarn mode, the local path will be set in flink-conf.yaml and used by
> jobmanager and taskmanager. We will find the warning log like below.
> {code:java}
> 2019-10-12 19:24:58,165 WARN org.apache.flink.core.plugin.PluginConfig
> - Environment variable [FLINK_PLUGINS_DIR] is set to
> [/Users/wangy/IdeaProjects/apache-flink/flink-dist/target/flink-1.10-SNAPSHOT-bin/flink-1.10-SNAPSHOT/plugins]
> but the directory doesn't exist
> {code}
> It was in introduced by FLINK-12143.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
