GitHub user JTaky opened a pull request:
https://github.com/apache/flink/pull/6284
[FLINK-9762] CoreOptions.TMP_DIRS ('io.tmp.dirs') wrongly managed on Yarn
## What is the purpose of the change
Currently TMP_DIRS (io.tmp.dirs) option has a bug in case of default
behaviour on Yarn platform:
If not value has been set via parameter on Yarn platform the env value
'LOCAL_DIRS' is used on Application Master.
When task manager is started this 'LOCAL_DIRS' is passed to the
TaskManager, so TaskManager do not understand that he needs to have a default
behaviour.
The proposed ~~hack~~ fix is to remove TMP_DIRS value in case if default
behaviour is required before starting TaskManager.
## Brief change log
* Extract method which set _io.tmp.dirs_ configuration value
* Fix configuration pass to TaskManager (**I am not sure what is the best
way to implement this**):
** Add a new flag, saying _io.tmp.dirs_ is value has been overridden
** If _io.tmp.dirs_ has *not* been overridden we remove _io.tmp.dirs_
remove config value, by setting the empty value (should we remove? clone
without a given key?)
* Correction of the documentation
## Verifying this change
This change is already covered by existing tests, such as
MiniClusterResource for Mesos part.
1. I do not see a way to test launching of the TaskManager with a correct
configuration.
2. A unit test documenting specific logic of TMP_DIRS managing is added:
BootstrapToolsTest#testSetTmpDirectoriesConfig
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JTaky/flink tmpDirs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6284.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6284
----
commit d0c0bf7682843947bd1cfdf27775eb75c2864232
Author: Oleksandr Nitavskyi <o.nitavskyi@...>
Date: 2018-07-02T07:42:17Z
[FLINK-9762][yarn] Extract method: tmp dirs setting
commit 99bdeb924a2bf91eab39462ac8253ee79dcb1137
Author: Oleksandr Nitavskyi <o.nitavskyi@...>
Date: 2018-07-02T08:06:56Z
[FLINK-9762][yarn] Fix TMP_DIRS configuration passing
* HACK: Instead of removing set value as empty
commit 2fd5754209508e082e4678e8605a25e3d25f8209
Author: Oleksandr Nitavskyi <o.nitavskyi@...>
Date: 2018-07-09T09:15:14Z
[FLINK-9762] correct documenatation for 'io.tmp.dirs' config
----
---
