[
https://issues.apache.org/jira/browse/FLINK-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295290#comment-16295290
]
ASF GitHub Bot commented on FLINK-8279:
---------------------------------------
Github user EronWright commented on a diff in the pull request:
https://github.com/apache/flink/pull/5176#discussion_r157548338
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java ---
@@ -127,21 +132,31 @@ private static BlobStoreService
createFileSystemBlobStore(Configuration configur
}
/**
- * Creates a local storage directory for a blob service under the given
parent directory.
+ * Creates a local storage directory for a blob service under the
configuration parameter given
+ * by {@link BlobServerOptions#STORAGE_DIRECTORY}. If this is
<tt>null</tt> or empty, we will
+ * fall back to the TaskManager temp directories (given by
+ * {@link ConfigConstants#TASK_MANAGER_TMP_DIR_KEY}; which in turn
falls back to
+ * {@link ConfigConstants#DEFAULT_TASK_MANAGER_TMP_PATH} currently set
to
+ * <tt>java.io.tmpdir</tt>) and choose one among them at random.
*
- * @param basePath
- * base path, i.e. parent directory, of the storage
directory to use (if <tt>null</tt> or
- * empty, the path in <tt>java.io.tmpdir</tt> will be used)
+ * @param config
+ * Flink configuration
*
* @return a new local storage directory
*
* @throws IOException
* thrown if the local file storage cannot be created or
is not usable
*/
- static File initLocalStorageDirectory(String basePath) throws
IOException {
+ static File initLocalStorageDirectory(Configuration config) throws
IOException {
+
+ String basePath =
config.getString(BlobServerOptions.STORAGE_DIRECTORY);
+
File baseDir;
if (StringUtils.isNullOrWhitespaceOnly(basePath)) {
- baseDir = new
File(System.getProperty("java.io.tmpdir"));
+ final String[] tmpDirPaths = config.getString(
+ ConfigConstants.TASK_MANAGER_TMP_DIR_KEY,
+
ConfigConstants.DEFAULT_TASK_MANAGER_TMP_PATH).split(",|" + File.pathSeparator);
--- End diff --
Consider encapsulating this parsing logic into
`TaskManagerServicesConfiguration` or similar.
> Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp
> directories
> ---------------------------------------------------------------------------------
>
> Key: FLINK-8279
> URL: https://issues.apache.org/jira/browse/FLINK-8279
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination, Network
> Affects Versions: 1.4.0
> Reporter: Nico Kruber
> Assignee: Nico Kruber
>
> Currently, the BLOB server and cache processes (temporarily) stash incoming
> files into their local file system in the directory given by the
> {{blob.storage.directory}} configuration property. If this property is not
> set or empty, it will fall back to {{java.io.tmpdir}}.
> Instead, in a Mesos/YARN environment, we could use the temporary directories
> they assigned to the Flink job which are not only the proper folder to use,
> but may also offer some more space.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
