[
https://issues.apache.org/jira/browse/FLINK-23127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise updated FLINK-23127:
--------------------------------
Component/s: FileSystems
> Can't use plugins for GCS filesystem
> ------------------------------------
>
> Key: FLINK-23127
> URL: https://issues.apache.org/jira/browse/FLINK-23127
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.13.0
> Reporter: Yaroslav Tkachenko
> Priority: Major
> Attachments: exception-stacktrace.txt
>
>
> I've been trying to add support for the GCS filesystem. I have a working
> example where I add two JARs to the */opt/flink/lib/* folder:
> * [GCS Hadoop
> connector|https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar]
> * *Shaded* Hadoop using
> [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar|https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar]
> Now I'm trying to follow the advice from [this
> page|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/overview/#pluggable-file-systems]
> and use Plugins instead. I followed the recommendations from
> [here|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/filesystems/plugins/].
> Now I have two JARs in the */opt/flink/plugins/hadoop-gcs/* folder:
> * [GCS Hadoop
> connector|https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-2.2.1.jar]
> * *Non-shaded* [Hadoop using
> hadoop-common-2.10.1.jar|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.10.1/hadoop-common-2.10.1.jar]
> As I can see, shading is not required for plugins (that's one of the reasons
> to use them), so I want to make it work with a simple non-shaded
> _hadoop-common_.
> However, the JobManager fails with an exception (full stacktrace is available
> is an attachment):
> {quote}Caused by:
> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not
> in the classpath/dependencies.
> {quote}
> The exception is thrown when _org.apache.hadoop.conf.Configuration_ and
> _org.apache.hadoop.fs.FileSystem_ [are not available in the
> classpath|https://github.com/apache/flink/blob/f2f2befee76d08b4d9aa592438dc0cf5ebe2ef96/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L1123-L1124],
> but they're available in hadoop-common and should have been loaded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
