1u0 commented on a change in pull request #9169: [FLINK-12998][docs] Update documentation for file systems loading as plugins URL: https://github.com/apache/flink/pull/9169#discussion_r309557749
########## File path: docs/ops/filesystems/index.md ########## @@ -83,17 +105,21 @@ fs.hdfs.hadoopconf: /path/to/etc/hadoop This registers `/path/to/etc/hadoop` as Hadoop's configuration directory and is where Flink will look for the `core-site.xml` and `hdfs-site.xml` files. -## Adding new File System Implementations +## Adding a new pluggable File System implementation -File systems are represented via the `org.apache.flink.core.fs.FileSystem` class, which captures the ways to access and modify files and objects in that file system. -Implementations are discovered by Flink through Java's service abstraction, making it easy to add new file system implementations. +File systems are represented via the `org.apache.flink.core.fs.FileSystem` class, which captures the ways to access and modify files and objects in that file system. To add a new file system: - Add the File System implementation, which is a subclass of `org.apache.flink.core.fs.FileSystem`. - Add a factory that instantiates that file system and declares the scheme under which the FileSystem is registered. This must be a subclass of `org.apache.flink.core.fs.FileSystemFactory`. - - Add a service entry. Create a file `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` which contains the class name of your file system factory class. + - Add a service entry. Create a file `META-INF/services/org.apache.flink.core.fs.FileSystemFactory` which contains the class name of your file system factory class + (see the [Java Service Loader docs](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html) for more details). + +During plugins discovery, the file system factory class will be loaded by a dedicated Java class loader to avoid class conflicts with other plugins and Flink components. +The same class loader should be used during file system instantiation and the file system operation calls. -See the [Java Service Loader docs](https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html) for more details on how service loaders work. +<span class="label label-warning">Warning</span> In practice, it means you should avoid using `Thread.currentThread().getContextClassLoader()` class loader +in your implementation. Review comment: I'd prefer not to document it. Currently the `create()` call has the class loader set only because there are some existing implementation that rely on it. So far there is no proper reason why would FSs need it, they can explicitly use the correct class loader. In the future, it should be possible to remove the additional setup of class loader for `create()` call. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
