1u0 commented on a change in pull request #9169: [FLINK-12998][docs] Update
documentation for file systems loading as plugins
URL: https://github.com/apache/flink/pull/9169#discussion_r305389810
##########
File path: docs/ops/filesystems/index.md
##########
@@ -35,32 +35,50 @@ File system instances are instantiated once per process
and then cached/pooled,
* This will be replaced by the TOC
{:toc}
-## Built-in File Systems
+## Local File System
-Flink ships with implementations for the following file systems:
+Flink has built-in support for the file system of the local machine, including
any NFS or SAN drives mounted into that local file system.
+It can be used by default without additional configuration. Local files are
referenced with the *file://* URI scheme.
- - **local**: This file system is used when the scheme is *"file://"*, and it
represents the file system of the local machine, including any NFS or SAN
drives mounted into that local file system.
+## Pluggable File Systems
- - **S3**: Flink directly provides file systems to talk to Amazon S3 with two
alternative implementations, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
Both implementations are self-contained with no dependency footprint.
-
- - **MapR FS**: The MapR file system *"maprfs://"* is automatically available
when the MapR libraries are in the classpath.
+Apache Flink project supports the following file systems as plugins:
+
+ - [**Amazon S3**](./s3.html) object storage is supported by two alternative
implementations, `flink-s3-fs-presto` and `flink-s3-fs-hadoop`.
+ Both implementations are self-contained with no dependency footprint.
+
+ - **MapR FS**: The MapR file system *maprfs://* is automatically available
when the MapR libraries are in the classpath.
+ <span class="label label-info">Note</span> You'd need to build from sources
the `flink-mapr-fs` JAR file.
- - **OpenStack Swift FS**: Flink directly provides a file system to talk to
the OpenStack Swift file system, registered under the scheme *"swift://"*.
- The implementation of `flink-swift-fs-hadoop` is based on the [Hadoop
Project](https://hadoop.apache.org/) but is self-contained with no dependency
footprint.
- To use it when using Flink as a library, add the respective maven dependency
(`org.apache.flink:flink-swift-fs-hadoop:{{ site.version }}`
- When starting a Flink application from the Flink binaries, copy or move the
respective jar file from the `opt` folder to the `lib` folder.
+ - **OpenStack Swift FS** is supported by `flink-swift-fs-hadoop` and
registered under the *swift://* URI scheme.
+ The implementation is based on the [Hadoop
Project](https://hadoop.apache.org/) but is self-contained with no dependency
footprint.
+ To use it when using Flink as a library, add the respective maven dependency
(`org.apache.flink:flink-swift-fs-hadoop:{{ site.version }}`).
- - **Azure Blob Storage**:
- Flink directly provides a file system to work with Azure Blob Storage.
- This filesystem is registered under the scheme *"wasb(s)://"*.
- The implementation is self-contained with no dependency footprint.
+ - **[Aliyun Object Storage Service](./oss.html)** is supported by
`flink-oss-fs-hadoop` and registered under the *oss://* URI scheme.
+ The implementation is based on the [Hadoop
Project](https://hadoop.apache.org/) but is self-contained with no dependency
footprint.
+
+ - **[Azure Blob Storage](./azure.html)** is supported by
`flink-azure-fs-hadoop` and registered under the *wasb(s)://* URI schemes.
+ The implementation is based on the [Hadoop
Project](https://hadoop.apache.org/) but is self-contained with no dependency
footprint.
+
+To use a pluggable file systems, copy the corresponding JAR file from the
`opt` directory to a directory under `plugins` directory
+of your Flink distribution before starting Flink, e.g.
+
+{% highlight bash %}
+mkdir ./plugins/s3-fs-hadoop
+cp ./opt/flink-s3-fs-hadoop-{{ site.version }}.jar ./plugins/s3-fs-hadoop/
+{% endhighlight %}
+
+<span class="label label-danger">Attention</span> The plugin mechanism for
file systems was introduced since Flink version `1.9` to
+support dedicated Java class loaders per file system and move away from class
shading mechanism. You can still use the provided file systems
+(or your own implementations) via the old mechanism by copying corresponding
JAR file into `lib` directory.
+
+The non plugin-based loading mechanism is deprecated and will be not supported
in the future.
Review comment:
I don't know what is the plan with MapR.
To totally deprecate `/lib` based FS jars loading, MapR adapter also may
probably need to become a separate plugin.
The main motivation to use plugins is to get rid of shading (and to allow
those components to have their own versions of dependent libraries). Allowing
to use the application class loader to load such components would be
error-prone and not user friendly.
> OK, but "Not being supported" is a bit ambiguous
I've changed/re-phrased this paragraph, although the new version may sound
not concrete too. 😄
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
