Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1524#discussion_r51123709
--- Diff: flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java
---
@@ -159,56 +163,109 @@ public int hashCode() {
/**
* Returns a reference to the {@link FileSystem} instance for accessing
the
* local file system.
- *
+ *
* @return a reference to the {@link FileSystem} instance for accessing
the
- * local file system.
+ * local file system.
*/
public static FileSystem getLocalFileSystem() {
// this should really never fail.
try {
URI localUri = OperatingSystem.isWindows() ? new
URI("file:/") : new URI("file:///");
return get(localUri);
- }
- catch (Exception e) {
+ } catch (Exception e) {
throw new RuntimeException("Cannot create URI for local
file system");
}
}
/**
+ * The default filesystem scheme to be used. This can be specified by
the parameter
+ * <code>fs.default-scheme</code> in <code>flink-conf.yaml</code>. By
default this is
+ * set to <code>file:///</code> (see {@link
ConfigConstants#FILESYSTEM_SCHEME}
+ * and {@link ConfigConstants#DEFAULT_FILESYSTEM_SCHEME}), and uses the
local filesystem.
+ */
+ private static URI defaultScheme;
+
+ /**
+ * <p>Sets the default filesystem scheme based on the user-specified
configuration parameter
+ * <code>fs.default-scheme</code>. By default this is set to
<code>file:///</code>
+ * (see {@link ConfigConstants#FILESYSTEM_SCHEME} and
+ * {@link ConfigConstants#DEFAULT_FILESYSTEM_SCHEME}),
+ * and the local filesystem is used.
+ * <p>
+ * As an example, if set to <code>hdfs://localhost:9000/</code>, then
an HDFS deployment
+ * with the namenode being on the local node and listening to port 9000
is going to be used.
+ * In this case, a file path specified as
<code>/user/USERNAME/in.txt</code>
+ * is going to be transformed into
<code>hdfs://localhost:9000/user/USERNAME/in.txt</code>. By
+ * default this is set to <code>file:///</code> which points to the
local filesystem.
+ * @param config the configuration from where to fetch the parameter.
+ */
+ public static void setDefaultScheme(Configuration config) throws
IOException {
+ synchronized (SYNCHRONIZATION_OBJECT) {
+ if (defaultScheme == null) {
+ String stringifiedUri =
config.getString(ConfigConstants.FILESYSTEM_SCHEME,
+
ConfigConstants.DEFAULT_FILESYSTEM_SCHEME);
+ try {
+ defaultScheme = new URI(stringifiedUri);
+ } catch (URISyntaxException e) {
+ throw new IOException("The URI used to
set the default filesystem " +
+ "scheme ('" + stringifiedUri +
"') is not valid.");
+ }
+ }
+ }
+ }
+
+ /**
+ * <p><b>ATTENTION:</b> this method is only used in tests.</p>
+ * <p>Clears the previously set <code>fs.default-scheme</code>
+ * (see {@link ConfigConstants#FILESYSTEM_SCHEME})</p>
+ */
+ public static void clearDefaultScheme() {
--- End diff --
This should not be public. Users do not read comments and simply call this.
The tests probably need a custom reflection cleaner to reset that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
