Yaroslav Tkachenko created FLINK-23127:
------------------------------------------
Summary: Can't use plugins for GCS filesystem
Key: FLINK-23127
URL: https://issues.apache.org/jira/browse/FLINK-23127
Project: Flink
Issue Type: Bug
Affects Versions: 1.13.0
Reporter: Yaroslav Tkachenko
Attachments: exception-stacktrace.txt
I've been trying to add support for the GCS filesystem. I have a working
example where I add two JARs to the */opt/flink/lib/* folder:
* [GCS Hadoop
connector|https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar]
* *Shaded* Hadoop using
[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar|https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar]
Now I'm trying to follow the advice from [this
page|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/overview/#pluggable-file-systems]
and use Plugins instead. I followed the recommendations from
[here|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/plugins/].
Now I have two JARs in the */opt/flink/plugins/hadoop-gcs/* folder:
* [GCS Hadoop
connector|https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-2.2.1.jar]
* *Non-shaded* [Hadoop using
hadoop-common-2.10.1.jar|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.10.1/hadoop-common-2.10.1.jar]
As I can see, shading is not required for plugins (that's one of the reasons to
use them), so I want to make it work with a simple non-shaded _hadoop-common_.
However, the JobManager fails with an exception (full stacktrace is available
is an attachment):
{quote}Caused by:
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in
the classpath/dependencies.
{quote}
The exception is thrown when _org.apache.hadoop.conf.Configuration_ and
_org.apache.hadoop.fs.FileSystem_ [are not available in the
classpath|https://github.com/apache/flink/blob/f2f2befee76d08b4d9aa592438dc0cf5ebe2ef96/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L1123-L1124],
but they're available in hadoop-common and should have been loaded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
