GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/5672
[FLINK-8872][flip6] fix yarn detached mode command parsing
## What is the purpose of the change
Running yarn per-job cluster in detached mode via `-yd` currently does not
detach and instead waits for the job to finish ( parameter `-d` works, through).
Example:
```
./bin/flink run -m yarn-cluster -yn 10 -yjm 768 -ytm 3072 -ys 2 -yd -p 20
-c org.apache.flink.streaming.examples.wordcount.WordCount
./examples/streaming/WordCount.jar --input <file>
```
The output somewhat says it is running detached but it will not:
```
2018-03-05 13:41:28,343 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - The Flink YARN
client has been started in detached mode. In order to stop Flink on YARN, use
the following command or a YARN web interface to stop it:
yarn application -kill application_1519984124671_0006
Please also note that the temporary files of the YARN session in the home
directoy will not be removed.
Starting execution of program
```
Please note that this PR also includes #5671 and #5670.
## Brief change log
- move `isDetachedMode()` switch from `ProgramOptions` to
`CustomCommandLine`
- extend some job submission debugging messages with detached mode info
for the tests:
- create test-jar in `flink-clients`
- add test-scope dependency from `flink-yarn` to the test-jar of
`flink-clients`
- add a small job jar file (with the word count from `flink-clients`) for
deployment tests to `flink-yarn`
## Verifying this change
This change added tests and can be verified as follows:
- added verification of `CustomCommandLine#isDetachedMode()` to
`CliFrontendRunTest`
- add `FlinkYarnSessionCliTest#testCorrectSettingOfDetachedMode()`
- add `CliFrontendRunTestWithYarn` test which tests `CliFrontend` together
with `FlinkYarnSessionCli` (this tests what was previously not working - each
individual component was tested but the error did not show in those tests)
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **only internally**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **yes**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable**
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-8872
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5672.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5672
----
commit 77cea62307c49cddc9b0699b36dfcb81d52a2f44
Author: Nico Kruber <nico@...>
Date: 2018-03-09T09:56:31Z
[hotfix][cli][tests] let CliFrontendRunTest extend from TestLogger
commit 49a502f50abf9d351ac0adbb1a78bfbc9f06cd73
Author: Nico Kruber <nico@...>
Date: 2018-03-06T10:43:32Z
[FLINK-8904][cli][tests] always restore the previous sysout when changing
it in the test
commit 417689deed94b00228e3959ba7ccdfc71985d8ac
Author: Nico Kruber <nico@...>
Date: 2018-03-09T10:05:51Z
[FLINK-8905][rest][client] fix RestClusterClient#getMaxSlots() returning 0
commit 3f2c31e7959038f0699cfcdc157ccc3b15a39ee1
Author: Nico Kruber <nico@...>
Date: 2018-03-05T17:24:17Z
[FLINK-8872][flip6] fix yarn detached mode command parsing
The detached flag if given by "-yd" was not passed correctly into the
CliFrontend and resulted in the CLI waiting for submitted jobs to finish
instead
of detaching from the execution.
commit 47a92c5c34c7422f809f5e2424221c8bdb2b8cc8
Author: Nico Kruber <nico@...>
Date: 2018-03-08T10:07:08Z
[FLINK-8906][flip6][tests] also test Flip6DefaultCLI in
org.apache.flink.client.cli tests
commit e4bc32b4960242ea5c3ec762e76fa40d7eeb540b
Author: Nico Kruber <nico@...>
Date: 2018-03-08T11:07:27Z
[FLINK-8872][yarn] add tests for YARN detached mode command line parsing
with CliFrontend
- create a test-jar of flink-clients
- create a jar with test programs inside flink-yarn (copy from
flink-clients)
- create CliFrontendRunTestWithYarn based on CliFrontendRunTest that
verifies
CliFrontend's parsing in conjunction with FlinkYarnSessionCli
-> verify detached mode in this test (can be extended further in the future)
commit 37cf32b42b451e90d4658e203889ae2542ad3a59
Author: Nico Kruber <nico@...>
Date: 2018-03-09T11:11:55Z
[hotfix][typo] fix typo in AbstractYarnClusterDescriptor
----
---
