[
https://issues.apache.org/jira/browse/FLINK-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16294992#comment-16294992
]
ASF GitHub Bot commented on FLINK-8279:
---------------------------------------
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/5176
[FLINK-8279][blob] fall back to TaskManager temp directories first
## What is the purpose of the change
Instead of falling back to `java.io.tmpdir` directly, the BLOB server and
cache processes should fall back to the TaskManager temp directories (given by
`ConfigConstants#TASK_MANAGER_TMP_DIR_KEY}` directly, before falling back
to `ConfigConstants#DEFAULT_TASK_MANAGER_TMP_PATH` (set to
`java.io.tmpdir`).
In a Mesos/YARN environment, this means that we will use the designated
temp directories for our jobs instead of the system-wide `java.io.tmpdir`.
These directories may also offer some more space.
## Brief change log
- change `BlobUtils#initLocalStorageDirectory()` storage directory
initialisation to use temporary directories in this order (and fall back to the
next if not available):
- `BlobServerOptions.STORAGE_DIRECTORY`
- `ConfigConstants.TASK_MANAGER_TMP_DIR_KEY` (and randomly select one
among them)
- `ConfigConstants.DEFAULT_TASK_MANAGER_TMP_PATH`
## Verifying this change
This change added tests and can be verified as follows:
- added tests verifying the fallbacks
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **Docs, JavaDocs**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-8279
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5176.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5176
----
commit 08e539c9abf825aa1ca60574cd9e0c5724cee252
Author: Nico Kruber <[email protected]>
Date: 2017-12-18T13:19:02Z
[FLINK-8279][blob] fall back to TaskManager temp directories first
Instead of falling back to java.io.tmpdir directly, the BLOB server and
cache
processes fall back to the TaskManager temp directories (given by
ConfigConstants#TASK_MANAGER_TMP_DIR_KEY) directly, before falling back
to ConfigConstants#DEFAULT_TASK_MANAGER_TMP_PATH (set to java.io.tmpdir).
In a Mesos/YARN environment, this means that we will use the designated temp
directories for our jobs instead of the system-wide java.io.tmpdir. These
directories may also offer some more space.
----
> Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp
> directories
> ---------------------------------------------------------------------------------
>
> Key: FLINK-8279
> URL: https://issues.apache.org/jira/browse/FLINK-8279
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Coordination, Network
> Affects Versions: 1.4.0
> Reporter: Nico Kruber
> Assignee: Nico Kruber
>
> Currently, the BLOB server and cache processes (temporarily) stash incoming
> files into their local file system in the directory given by the
> {{blob.storage.directory}} configuration property. If this property is not
> set or empty, it will fall back to {{java.io.tmpdir}}.
> Instead, in a Mesos/YARN environment, we could use the temporary directories
> they assigned to the Flink job which are not only the proper folder to use,
> but may also offer some more space.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
