GitHub user szyszy opened a pull request:
https://github.com/apache/spark/pull/20761
[SPARK-20327][CORE][YARN] Add CLI support for YARN custom resources, like
GPUs
## What changes were proposed in this pull request?
This PR adds CLI support for YARN custom resources, e.g. GPUs and any other
resources YARN defines.
The custom resources are defined with Spark properties, no additional CLI
arguments were introduced.
The properties can be defined in the following form:
**Driver resources, client mode:**
Format: `spark.yarn.am.resource.<resource-name>`
The property name follows the naming convention of YARN AM cores / memory
properties: `spark.yarn.am.memory and spark.yarn.am.cores
`
**Driver resources, cluster mode:**
Format: `spark.yarn.driver.resource.<resource-name>`
The property name follows the naming convention of driver cores / memory
properties: `spark.driver.memory and spark.driver.cores.`
**Executor resources:**
Format: `spark.yarn.executor.resource.<resource-name>`
The property name follows the naming convention of executor cores / memory
properties: `spark.executor.memory / spark.executor.cores`.
For the driver resources (cluster mode) and executor resources properties,
we use the `yarn` prefix here as custom resource types are specific to YARN,
currently.
Please note that a validation logic is added to avoid having requested
resources defined in 2 ways, for example defining the following configs:
```
"--conf", "spark.driver.memory=2G",
"--conf", "spark.yarn.driver.resource.memory=1G"
```
will not start execution and prints an error message:
`spark.driver.memory and spark.yarn.driver.resource.memory configs are both
present, only one of them is allowed at the same time!`
## How was this patch tested?
Unit tests + manual execution.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/szyszy/spark SPARK-20327
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20761.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20761
----
commit 0bb46134ce61eee8658b40dc3f5598dcc83c34fa
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-14T11:36:14Z
SPARK-20327. Introduce custom resource type configs and use new yarn
resource api
commit 9a17a740d7b4700ecd3aebbce3cf4a42241698f7
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-14T11:45:06Z
SPARK-20327. add yarn prefix to new resource type configs
commit e4c63b043d7c13959c65bc16229ff71bd0d714af
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-14T12:59:46Z
SPARK-20327. add yarn prefix to constant
commit 6819f95add04ec9679827c569cd2d4d6ebf747a3
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-14T13:12:13Z
SPARK-20327. add missing call to
ResourceTypeHelper.setResourceInfoFromResourceTypes
commit 825e95782c6df6ce27c2da7384c4cba6f91c9de3
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-14T17:18:54Z
SPARK-20327. use reflection for all yarn api calls (hadoop 2.x
compatibility)
commit b0667f6bc18473289a1065b2337641c9fecc8ee3
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-15T13:07:16Z
SPARK-20327. add clarifying comment to SparkSubmitArguments
commit 99673ddc51fa112f5e13846f7a92b864bc264a49
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-15T14:22:38Z
SPARK-20327. refactor SparkSubmitArguments: validate duplicate resource
type configs
commit d20e94e928ca10e08fe19572cd3d2c2466dca417
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-15T14:44:48Z
SPARK-20327. renamings, comment fixes
commit 6c9d5e086d87916ec505bed56d1b4d883fa698ac
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-20T14:11:19Z
SPARK-20327. fix scalastyle issues
commit d44c4183ab4324eb159516171ee714f5a4c655be
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-22T13:59:17Z
SPARK-20327. added debug log to yarn Client
commit 106bbde7db4efc07a85186e4df905e5af99cb9e5
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-22T14:31:02Z
SPARK-20327. unit test fixes
commit f43d8d9f2ef25fb777bac5830f771e1de656684b
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-22T14:31:30Z
SPARK-20327. improve resource type logging
commit 2117fa005185fe5d80e1366ebbb0978903b92954
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-22T16:09:13Z
SPARK-20327. collect and print resource type error messages instead of
printing the first
commit 0d41ff747b3b6d5d515103c9d2d4396a65b4ecc0
Author: Szilard Nemeth <snemeth@...>
Date: 2018-02-22T16:44:20Z
SPARK-20327. documentation update
commit 48b6baa073559f65d5e1a785c6b2982d34fec54f
Author: Szilard Nemeth <snemeth@...>
Date: 2018-03-06T21:24:32Z
SPARK-20327. test fixes
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
